- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests/build-pkg: change the local git repo path
by Lin Jiaxin 15 Mar '21

15 Mar '21

As the path of repo mirrors under /srv/git changes, introduce upstream_dir to get new path. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 43932865..350d8fdf 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -6,6 +6,7 @@ # - pkg_ver # - config # upstream_url +# upstream_dir # We'll use the below global env vars, but not treat them as parameters to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ @@ -123,9 +124,9 @@ replace_source() do echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { - url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" + url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#commit=${upstream_commit}" [ -n "$git_tag" ] && { - url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#tag=${git_tag}" + url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#tag=${git_tag}" } } } -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/auto_submit: change the upstream_repo path to submit
by Lin Jiaxin 15 Mar '21

15 Mar '21

Reference: 5d6cea1bd531409f0444abf14088fdecd8b08c2d Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index faebb2f..a27075b 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -33,7 +33,7 @@ class AutoSubmit git_repo = newrefs_info['git_repo'] puts "git_repo : #{git_repo}" url = Array(newrefs_info['url'])[0] - submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{git_repo} upstream_url=#{url}"] + submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{newrefs_info['belong']}/#{git_repo} upstream_url=#{url}"] if newrefs_info['pkgbuild_repo'] pkgbuild_repo = get_pkgbuild_repo(newrefs_info['pkgbuild_repo']) -- 2.23.0

2 2

[PATCH compass-ci] optimize doc
by Luan Shengde 15 Mar '21

15 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-tests-container.zh.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/build-lkp-tests-container.zh.md b/doc/manual/build-lkp-tests-container.zh.md index 10a1611..407ddbf 100644 --- a/doc/manual/build-lkp-tests-container.zh.md +++ b/doc/manual/build-lkp-tests-container.zh.md @@ -60,4 +60,4 @@ submit命令： - 了解更多submit命令的使用方法，请参考：[submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… + 了解更多submit命令的使用方法，请参考：[submit用户手册](https://gitee.com/wu_fengguang/compass-c… -- 2.23.0

1 0

[PATCH compass-ci 2/2] service/scheduler: fix the mq connection exception
by Wu Zhende 15 Mar '21

15 Mar '21

[Why] If the connection to the mq service is disconnected, the interface for sending messages is abnormal, affecting the main logic. [How] Start a thread to send mq messages. Reconnect and send every 5 seconds, three attempts. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 10 ++++++++++ src/scheduler/close_job.cr | 2 +- src/scheduler/find_job_boot.cr | 2 +- src/scheduler/update_job_parameter.cr | 2 +- 4 files changed, 13 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f5aba03..9528f54 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -218,4 +218,14 @@ class Sched return nil end + + def mq_pushlish_confirm(queue, msg) + 3.times do + @mq.pushlish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end end diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index ea1dbf5..e014361 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -45,7 +45,7 @@ class Sched "job_state" => "close", "time" => get_time } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..9e83537 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -32,7 +32,7 @@ class Sched "time" => get_time, "job_state" => "boot" } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end # auto submit a job to collect the host information. diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index b43aa06..d81f28b 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -44,6 +44,6 @@ class Sched "job_state" => (@env.get?("job_state") || "update").to_s, "time" => @env.get?("time").to_s } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/mq.cr: add new function reconnect
by Wu Zhende 15 Mar '21

15 Mar '21

When the connection to the mq is disconnected, an interface needs to be provided to re-initiate the connection. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/mq.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/lib/mq.cr b/src/lib/mq.cr index 87c67e9..980585d 100644 --- a/src/lib/mq.cr +++ b/src/lib/mq.cr @@ -5,17 +5,34 @@ require "singleton" require "amqp-client" require "amq-protocol" +require "./json_logger" + class MQClient getter ch : AMQP::Client::Channel MQ_HOST = ENV.has_key?("MQ_HOST") ? ENV["MQ_HOST"] : "172.17.0.1" MQ_PORT = (ENV.has_key?("MQ_PORT") ? ENV["MQ_PORT"] : 5672).to_i32 def initialize(host = MQ_HOST, port = MQ_PORT) + @log = JSONLogger.new @client = AMQP::Client.new("amqp://#{host}:#{port}") conn = @client.connect @ch = conn.channel.as(AMQP::Client::Channel) end + def reconnect + conn = @client.connect + @ch = conn.channel.as(AMQP::Client::Channel) + @log.info({ + "msg" => "rabbitmq reconnected successfully", + "source" => "mq_client" + }.to_json) + rescue e + @log.warn({ + "msg" => e.to_s, + "source" => "mq_client" + }.to_json) + end + def self.instance Singleton::Of(self).instance end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 15 Mar '21

15 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 83 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..43a9ce341 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,83 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + echo $ansible_host + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] || { + echo "Ansible failed to install." + exit 1 + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + exit 1 + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook failed to run!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

2 2

[PATCH lkp-tests 2/2] distro/aliyun: revoke commit e21801797 and use a new approach
by Wei Jihui 15 Mar '21

15 Mar '21

commit:e21801797 is for use dnf instead of yum in centos7 use dnf it can raise error when yum install print "No package * available" [why] install dnf will installs many dependency packages. this makes cci-depends can not download some packages. [how] yum install do not use dnf and analyze installation output, if has "No package * available" then exit 1 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/aliyun | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 253121e0..24f16287 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -13,16 +13,6 @@ support_repo_epel() fi } -update_yum_into_dnf() -{ - if ! rpm -q dnf >/dev/null; then - yum install -y -q dnf - ln -sf /usr/bin/dnf /usr/bin/yum - else - dnf makecache timer >/dev/null - fi -} - distro_install_depends() { local script @@ -30,7 +20,6 @@ distro_install_depends() local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel - [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf for script in $scripts do @@ -71,10 +60,10 @@ pack_benchmark() yum_download() { [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel - [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -y -q --downloadonly $*" - yum install -y -q --downloadonly "$@" + echo "yum install -y --downloadonly $@ | grep -e \"No\ package .*\ available\"" + yum install -y --downloadonly "$@" | grep -e "No\ package .*\ available" && exit 1 + return 0 } yum_pack() -- 2.23.0

1 0

[PATCH lkp-tests 1/2] distro/aliyun: fix yum install args
by Wei Jihui 15 Mar '21

15 Mar '21

[error] Ignored option -q, -v, -d or -e (probably due to merging: -yq != -y -q) Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/aliyun | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 83f058c3..253121e0 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -9,14 +9,14 @@ shopt -s nullglob support_repo_epel() { if ! rpm -q epel-release >/dev/null; then - yum install -yq epel-release + yum install -y -q epel-release fi } update_yum_into_dnf() { if ! rpm -q dnf >/dev/null; then - yum install -yq dnf + yum install -y -q dnf ln -sf /usr/bin/dnf /usr/bin/yum else dnf makecache timer >/dev/null @@ -46,7 +46,7 @@ distro_install_depends() for pkg in $packages do - rpm --quiet --query $pkg || yum install -yq $pkg + rpm --quiet --query $pkg || yum install -y -q $pkg done done } @@ -73,8 +73,8 @@ yum_download() [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -yq --downloadonly $*" - yum install -yq --downloadonly "$@" + echo "yum install -y -q --downloadonly $*" + yum install -y -q --downloadonly "$@" } yum_pack() @@ -137,9 +137,9 @@ install_gcc7() wget -O /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm https://rpmfind.net/linux/centos/7/extras/x86_64/Packages/centos-release-sc… rpm -ivh --ignoresize /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm else - yum install -yq centos-release-scl-rh + yum install -y -q centos-release-scl-rh fi - yum install -yq "devtoolset-7-gcc*" + yum install -y -q "devtoolset-7-gcc*" source /opt/rh/devtoolset-7/enable } @@ -148,7 +148,7 @@ download_ocfs2_tools() local pkgs=$1 mkdir -p /opt/rpms - yum install -yq wget + yum install -y -q wget wget -nc -P /opt/rpms https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/ocfs2-… @@ -160,7 +160,7 @@ download_ocfs2_tools() fi # Try to download the dependences of ocfs2-tools/ocfs2-tools-devel - yum install -yq --downloadonly "$download_pkgs" + yum install -y -q --downloadonly "$download_pkgs" } install_ocfs2_tools() @@ -169,8 +169,8 @@ install_ocfs2_tools() download_ocfs2_tools "$pkgs" - yum install -yq /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm + yum install -y -q /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm echo "$pkgs" | grep -q 'ocfs2-tools-devel' && \ - yum install -yq /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm + yum install -y -q /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm } -- 2.23.0

1 0

[PATCH compass-ci] fix: port conflict between srv-http-cci and scheduler_https
by Yu Chuan 15 Mar '21

15 Mar '21

[Why] container/scheduler_https already use 11306 port of host, so change the port of container/srv-http-cci to 11307. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/scheduler/start | 2 +- container/srv-http/start-cci | 2 +- sparrow/3-code/dev-env | 2 +- src/scheduler/constants.cr | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index d83fcc21fb0b..e655dd4a94b2 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -37,7 +37,7 @@ defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' -defaults['SRV_HTTP_CCI_PORT'] ||= '11306' +defaults['SRV_HTTP_CCI_PORT'] ||= '11307' env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci index 78ab1490795b..84864761e689 100755 --- a/container/srv-http/start-cci +++ b/container/srv-http/start-cci @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-cci - -p 11306:11300 + -p 11307:11300 -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 403ccb127813..93a1667e73b1 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,7 +18,7 @@ server_name=$(hostname | cut -f1 -d.) : ${SRV_HTTP_RESULT_PORT:=11300} : ${SRV_HTTP_OS_PORT:=11304} : ${SRV_HTTP_GIT_PORT:=11305} -: ${SRV_HTTP_CCI_PORT:=11306} +: ${SRV_HTTP_CCI_PORT:=11307} : ${LOGGING_ES_HOST:=$server_ip} : ${LOGGING_ES_PORT:=9202} : ${INITRD_HTTP_HOST:=$server_ip} diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 3fdb9839e90d..a5e52b8f7a6d 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -26,7 +26,7 @@ OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1 OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 SRV_HTTP_CCI_HOST = (ENV.has_key?("SRV_HTTP_CCI_HOST") ? ENV["SRV_HTTP_CCI_HOST"] : "172.17.0.1") -SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11306") +SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11307") CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") LAB_REPO = "lab-z9" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package/common: install bunny for mq
by Wu Zhende 15 Mar '21

15 Mar '21

multi-qemu and multi-docker need bunny to connect rabbitmq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 0324f58..8942247 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer -- 2.23.0

1 0