- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] src/lib/job.cr: open the path for uploading rpm packages.
by Li Ping 16 Mar '21

16 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9e18a4c..40aaa62 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -278,7 +278,7 @@ class Job mount_type = "nfs" if mount_type == "cifs" common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") + common_dir = "#{tmp_os}-#{tmp_os_version}" if @hash.has_key?("rpmbuild-pkg") return common_dir end @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/compare-results.zh.md: translate to Chinese
by Lin Jiaxin 16 Mar '21

16 Mar '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- doc/manual/compare-results.zh.md | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/doc/manual/compare-results.zh.md b/doc/manual/compare-results.zh.md index 1b43d6b..b96add9 100644 --- a/doc/manual/compare-results.zh.md +++ b/doc/manual/compare-results.zh.md @@ -1,29 +1,26 @@ -# Introduction of compare feature - The compare feature used to analyze the results of different jobs, showing the performance waves and changes - under different influencing factors, for users to analyze performance influencing factor. +# 比较功能介绍 + 比较功能用于分析不同任务的结果，显示不同影响因素下的性能波动和变化，供用户分析性能影响因素。 ## url: https://compass-ci.openeuler.org/compare -## options directions -### filters: +## 选项说明 +### 过滤器: - suite: iperf, netperf, mysql, ... - OS: openeuler 20.03, centos 7.6, ... - os_arch: aarch64, x86 - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... - we can combine the above options arbitrarily to limit compare scope. - choose at least one option as filter. + 我们可以任意组合以上选项来限制比较范围，但至少选择一个选项作为过滤器。 -### dimension - Dimension can select: os, os_version, os_arch, suite, tbox_group. - Within filter, we will compare all different job result by dimension - and keep other test conditions are same. +### 维度 + 可选维度: os, os_version, os_arch, suite, tbox_group. + 在过滤器中，我们将按所选维度比较所有不同的任务结果，并保持其他测试条件一致。 -## example: - filter: suite = iperf - dimension: os_version +## 示例: + 过滤器: suite = iperf + 维度: os_version - result: - os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + 结果: + os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # 其他测试条件一致 20.09 20.03 metric -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/web-backend-nginx: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend-nginx/nginx.conf | 2 +- container/web-backend-nginx/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/web-backend-nginx/nginx.conf b/container/web-backend-nginx/nginx.conf index c9a4ec4..50ced4c 100644 --- a/container/web-backend-nginx/nginx.conf +++ b/container/web-backend-nginx/nginx.conf @@ -8,7 +8,7 @@ http { default_type application/octet-stream; include /etc/nginx/conf.d/*.conf; server { - listen 11320 ssl; + listen 20003 ssl; server_name api.compass-ci.openeuler.org; ssl_certificate /opt/cert/web-backend.crt; ssl_certificate_key /opt/cert/web-backend.key; diff --git a/container/web-backend-nginx/start b/container/web-backend-nginx/start index 4da4c8b..5360385 100755 --- a/container/web-backend-nginx/start +++ b/container/web-backend-nginx/start @@ -10,7 +10,7 @@ docker_rm web-backend-nginx cmd=( docker run --name web-backend-nginx - -p 11320:11320 + -p 20003:20003 -d -v /etc/ssl/certs:/opt/cert -- 2.23.0

1 1

[PATCH compass-ci 1/2] container/web-backend: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/start | 2 +- container/web-backend/web-backend | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/web-backend/start b/container/web-backend/start index 6731d27..e680255 100755 --- a/container/web-backend/start +++ b/container/web-backend/start @@ -18,7 +18,7 @@ cmd=( --name web-backend --restart=always -d - -p 32767:32767 + -p 10002:10002 -e ES_PORT=9200 -e SRV_HTTP_RESULT_HOST=$SRV_HTTP_RESULT_HOST diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 74eda85..33a3fae 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -9,7 +9,7 @@ CCI_SRC = ENV['CCI_SRC'] || '/c/compass-ci' require "#{CCI_SRC}/src/lib/web_backend.rb" set :bind, '0.0.0.0' -set :port, 32767 +set :port, 10002 set :show_exceptions, false set :environment, :production -- 2.23.0

1 1

[PATCH compass-ci 2/2] answerback-email: update tool with new port
by Luan Shengde 16 Mar '21

16 Mar '21

assign-account service has assigned new port: 29999 => 10000 send-internet-mail service has assigned new port: 49000 => 10001 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index f573f0f..f4b6ef3 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -28,9 +28,9 @@ names = Set.new %w[ defaults = relevant_defaults(names) JUMPER_HOST ||= defaults['JUMPER_HOST'] -JUMPER_PORT ||= defaults['JUMPER_PORT'] || 29999 +JUMPER_PORT ||= defaults['JUMPER_PORT'] || 10000 SEND_MAIL_HOST ||= defaults['SEND_MAIL_HOST'] || 'localhost' -SEND_MAIL_PORT ||= defaults['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_PORT ||= defaults['SEND_MAIL_PORT'] || 10001 LAB ||= defaults['lab'] TOOL_SERVER = 'z9' -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/assign-account: update for new port
by Luan Shengde 16 Mar '21

16 Mar '21

the assign-account servive has been assigned new port: 29999 => 10000 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 2 +- container/assign-account/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index c5d77db..abbab0f 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -10,7 +10,7 @@ require 'yaml' require_relative 'get_account_info' set :bind, '0.0.0.0' -set :port, 29999 +set :port, 10000 post '/assign_account' do begin diff --git a/container/assign-account/start b/container/assign-account/start index 793191a..b926853 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -14,7 +14,7 @@ cmd=( -v /etc:/etc:rw -v /srv/account-info:/opt/account_data/:rw -v /home:/home:rw - -p 29999:29999 + -p 10000:10000 assign-account ) -- 2.23.0

1 0

[PATCH compass-ci] constants: change service default ports
by Cao Xueliang 16 Mar '21

16 Mar '21

Reference: 6354ba631e0fd5f795a1dc28b65e2002bd3de668 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/constants.rb | 6 +++--- src/scheduler/constants.cr | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index 8d8e488..6c30aaf 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -14,9 +14,9 @@ SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_RESULT_HOST ||= config['SRV_HTTP_RESULT_HOST'] || ENV['SRV_HTTP_RESULT_HOST'] || '172.17.0.1' SRV_HTTP_OS_HOST ||= config['SRV_HTTP_OS_HOST'] || ENV['SRV_HTTP_OS_HOST'] || '172.17.0.1' SRV_HTTP_GIT_HOST ||= config['SRV_HTTP_GIT_HOST'] || ENV['SRV_HTTP_GIT_HOST'] || '172.17.0.1' -SRV_HTTP_RESULT_PORT ||= config['SRV_HTTP_RESULT_PORT'] || ENV['SRV_HTTP_RESULT_PORT'] || 11300 -SRV_HTTP_OS_PORT ||= config['SRV_HTTP_OS_PORT'] || ENV['SRV_HTTP_OS_PORT'] || 11304 -SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || 11305 +SRV_HTTP_RESULT_PORT ||= config['SRV_HTTP_RESULT_PORT'] || ENV['SRV_HTTP_RESULT_PORT'] || 20007 +SRV_HTTP_OS_PORT ||= config['SRV_HTTP_OS_PORT'] || ENV['SRV_HTTP_OS_PORT'] || 20009 +SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || 20010 SRV_HTTP_DOMAIN ||= config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index a5e52b8..d18c88d 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -26,7 +26,7 @@ OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1 OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 SRV_HTTP_CCI_HOST = (ENV.has_key?("SRV_HTTP_CCI_HOST") ? ENV["SRV_HTTP_CCI_HOST"] : "172.17.0.1") -SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11307") +SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : 20011).to_i32 CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") LAB_REPO = "lab-z9" -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: change service ports
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/start-cci | 2 +- container/srv-http/start-git | 2 +- container/srv-http/start-initrd | 2 +- container/srv-http/start-os | 2 +- container/srv-http/start-pub | 2 +- container/srv-http/start-result | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci index 8486476..00d8e4c 100755 --- a/container/srv-http/start-cci +++ b/container/srv-http/start-cci @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-cci - -p 11307:11300 + -p 20011:11300 -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-git b/container/srv-http/start-git index f2292bf..c1181a7 100755 --- a/container/srv-http/start-git +++ b/container/srv-http/start-git @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-git - -p 11305:11300 + -p 20010:11300 -v /srv/git/archlinux:/srv/git/archlinux:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 495cc25..466ec1b 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-initrd - -p 11303:11300 + -p 20008:11300 -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro diff --git a/container/srv-http/start-os b/container/srv-http/start-os index bb344cf..d39e04e 100755 --- a/container/srv-http/start-os +++ b/container/srv-http/start-os @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-os - -p 11304:11300 + -p 20009:11300 -v /srv/os:/srv/os:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-pub b/container/srv-http/start-pub index dc1864c..f42b117 100755 --- a/container/srv-http/start-pub +++ b/container/srv-http/start-pub @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-pub - -p 11302:11300 + -p 20006:11300 -v /srv/pub:/srv/pub:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-result b/container/srv-http/start-result index f9ecc97..bf0287d 100755 --- a/container/srv-http/start-result +++ b/container/srv-http/start-result @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-result - -p 11300:11300 + -p 20007:11300 -v /srv/result:/srv/result:ro -v /etc/localtime:/etc/localtime:ro -d -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: open the path for uploading rpm packages.
by Li Ping 16 Mar '21

16 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index f8cfa97..40aaa62 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -278,7 +278,7 @@ class Job mount_type = "nfs" if mount_type == "cifs" common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") + common_dir = "#{tmp_os}-#{tmp_os_version}" if @hash.has_key?("rpmbuild-pkg") return common_dir end @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["upstream_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 16 Mar '21

16 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 83 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..9ecb4d2c5 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,83 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + echo $ansible_host + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] || { + echo "Ansible failed to install." + exit 1 + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + exit 1 + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook is running failed!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

1 1

[PATCH v4 compass-ci 2/2] multi-docker: add reboot function
by Wu Zhende 16 Mar '21

16 Mar '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 34 +++++++++++++++++++++++++++++++++- providers/docker/run.sh | 2 ++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index f4e954e..9d6854f 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -7,6 +7,8 @@ require 'open-uri' require 'json' require 'set' require 'fileutils' + +require '../lib/mq_client' require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' @@ -19,6 +21,8 @@ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 LOG_DIR = '/srv/cci/serial/logs' +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || 'localhost' +MQ_PORT = ENV['MQ_PORT'] || 5672 def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -84,7 +88,8 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, + { 'job_id' => hash['job_id'], + 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path, 'log_dir' => "#{LOG_DIR}/#{hostname}" @@ -145,6 +150,29 @@ def loop_main(hostname, queues) end end +def loop_reboot_docker(hostname) + loop do + begin + reboot_docker(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_docker(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) + queue = mq.queue(hostname, {:durable => true}) + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + puts msg + machine_info = JSON.parse(msg) + job_id = machine_info['job_id'] + system "docker rm -f #{job_id};echo $?" + mq.ack(info) + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -153,6 +181,10 @@ def save_pid(pids) end def multi_docker(hostname, nr_container, queues) + Process.fork do + loop_reboot_docker(hostname) + end + pids = [] nr_container.to_i.times do |i| pid = Process.fork do diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 2dfe36d..80950dd 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -4,6 +4,7 @@ . $LKP_SRC/lib/yaml.sh +: ${job_id:=$$} : ${docker_image:="centos:7"} : ${load_path:="${HOME}/jobs"} : ${hostname:="dc-1g-1"} @@ -23,6 +24,7 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm + --name ${job_id} --hostname $host.compass-ci.net -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 16 Mar '21

16 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH v4 compass-ci] providers/multi-qemu: optimize
by Wu Zhende 16 Mar '21

16 Mar '21

1. optimize function name 2. the host and port are transferred when the MQClient is initalized 3. reboot vm cyclically Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 5979cc9..d3f8406 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -104,8 +104,19 @@ def save_pid(arr) f.close end -def reboot_deadline_machine(hostname) - mq = MQClient.new +def loop_reboot_vm(hostname) + loop do + begin + reboot_vm(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_vm(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) queue = mq.queue(hostname, {:durable => true}) queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| puts msg @@ -118,10 +129,7 @@ end def multiqemu Process.fork do - reboot_deadline_machine(HOSTNAME) - rescue StandardError => e - puts e.backtrace - sleep 5 + loop_reboot_vm(HOSTNAME) end pids = [] -- 2.23.0

1 0

[PATCH compass-ci] container/webhook: change the service port
by Li Yuanchao 15 Mar '21

15 Mar '21

For the port policy. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/webhook/start b/container/webhook/start index 373396a..1d121a4 100755 --- a/container/webhook/start +++ b/container/webhook/start @@ -12,7 +12,7 @@ cmd = %w[ --restart=always --name webhook -d - -p 11301:11301 + -p 20005:20005 node:webhook ] -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: change the local git repo path
by Lin Jiaxin 15 Mar '21

15 Mar '21

As the path of repo mirrors under /srv/git changes, introduce upstream_dir to get new path. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 43932865..350d8fdf 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -6,6 +6,7 @@ # - pkg_ver # - config # upstream_url +# upstream_dir # We'll use the below global env vars, but not treat them as parameters to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ @@ -123,9 +124,9 @@ replace_source() do echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { - url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" + url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#commit=${upstream_commit}" [ -n "$git_tag" ] && { - url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#tag=${git_tag}" + url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#tag=${git_tag}" } } } -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/auto_submit: change the upstream_repo path to submit
by Lin Jiaxin 15 Mar '21

15 Mar '21

Reference: 5d6cea1bd531409f0444abf14088fdecd8b08c2d Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index faebb2f..a27075b 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -33,7 +33,7 @@ class AutoSubmit git_repo = newrefs_info['git_repo'] puts "git_repo : #{git_repo}" url = Array(newrefs_info['url'])[0] - submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{git_repo} upstream_url=#{url}"] + submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{newrefs_info['belong']}/#{git_repo} upstream_url=#{url}"] if newrefs_info['pkgbuild_repo'] pkgbuild_repo = get_pkgbuild_repo(newrefs_info['pkgbuild_repo']) -- 2.23.0

2 2

[PATCH compass-ci] optimize doc
by Luan Shengde 15 Mar '21

15 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-tests-container.zh.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/build-lkp-tests-container.zh.md b/doc/manual/build-lkp-tests-container.zh.md index 10a1611..407ddbf 100644 --- a/doc/manual/build-lkp-tests-container.zh.md +++ b/doc/manual/build-lkp-tests-container.zh.md @@ -60,4 +60,4 @@ submit命令： - 了解更多submit命令的使用方法，请参考：[submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… + 了解更多submit命令的使用方法，请参考：[submit用户手册](https://gitee.com/wu_fengguang/compass-c… -- 2.23.0

1 0

[PATCH compass-ci 2/2] service/scheduler: fix the mq connection exception
by Wu Zhende 15 Mar '21

15 Mar '21

[Why] If the connection to the mq service is disconnected, the interface for sending messages is abnormal, affecting the main logic. [How] Start a thread to send mq messages. Reconnect and send every 5 seconds, three attempts. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 10 ++++++++++ src/scheduler/close_job.cr | 2 +- src/scheduler/find_job_boot.cr | 2 +- src/scheduler/update_job_parameter.cr | 2 +- 4 files changed, 13 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f5aba03..9528f54 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -218,4 +218,14 @@ class Sched return nil end + + def mq_pushlish_confirm(queue, msg) + 3.times do + @mq.pushlish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end end diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index ea1dbf5..e014361 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -45,7 +45,7 @@ class Sched "job_state" => "close", "time" => get_time } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..9e83537 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -32,7 +32,7 @@ class Sched "time" => get_time, "job_state" => "boot" } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end # auto submit a job to collect the host information. diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index b43aa06..d81f28b 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -44,6 +44,6 @@ class Sched "job_state" => (@env.get?("job_state") || "update").to_s, "time" => @env.get?("time").to_s } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/mq.cr: add new function reconnect
by Wu Zhende 15 Mar '21

15 Mar '21

When the connection to the mq is disconnected, an interface needs to be provided to re-initiate the connection. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/mq.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/lib/mq.cr b/src/lib/mq.cr index 87c67e9..980585d 100644 --- a/src/lib/mq.cr +++ b/src/lib/mq.cr @@ -5,17 +5,34 @@ require "singleton" require "amqp-client" require "amq-protocol" +require "./json_logger" + class MQClient getter ch : AMQP::Client::Channel MQ_HOST = ENV.has_key?("MQ_HOST") ? ENV["MQ_HOST"] : "172.17.0.1" MQ_PORT = (ENV.has_key?("MQ_PORT") ? ENV["MQ_PORT"] : 5672).to_i32 def initialize(host = MQ_HOST, port = MQ_PORT) + @log = JSONLogger.new @client = AMQP::Client.new("amqp://#{host}:#{port}") conn = @client.connect @ch = conn.channel.as(AMQP::Client::Channel) end + def reconnect + conn = @client.connect + @ch = conn.channel.as(AMQP::Client::Channel) + @log.info({ + "msg" => "rabbitmq reconnected successfully", + "source" => "mq_client" + }.to_json) + rescue e + @log.warn({ + "msg" => e.to_s, + "source" => "mq_client" + }.to_json) + end + def self.instance Singleton::Of(self).instance end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 15 Mar '21

15 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 83 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..43a9ce341 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,83 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + echo $ansible_host + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] || { + echo "Ansible failed to install." + exit 1 + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + exit 1 + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook failed to run!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

2 2

[PATCH lkp-tests 2/2] distro/aliyun: revoke commit e21801797 and use a new approach
by Wei Jihui 15 Mar '21

15 Mar '21

commit:e21801797 is for use dnf instead of yum in centos7 use dnf it can raise error when yum install print "No package * available" [why] install dnf will installs many dependency packages. this makes cci-depends can not download some packages. [how] yum install do not use dnf and analyze installation output, if has "No package * available" then exit 1 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/aliyun | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 253121e0..24f16287 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -13,16 +13,6 @@ support_repo_epel() fi } -update_yum_into_dnf() -{ - if ! rpm -q dnf >/dev/null; then - yum install -y -q dnf - ln -sf /usr/bin/dnf /usr/bin/yum - else - dnf makecache timer >/dev/null - fi -} - distro_install_depends() { local script @@ -30,7 +20,6 @@ distro_install_depends() local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel - [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf for script in $scripts do @@ -71,10 +60,10 @@ pack_benchmark() yum_download() { [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel - [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -y -q --downloadonly $*" - yum install -y -q --downloadonly "$@" + echo "yum install -y --downloadonly $@ | grep -e \"No\ package .*\ available\"" + yum install -y --downloadonly "$@" | grep -e "No\ package .*\ available" && exit 1 + return 0 } yum_pack() -- 2.23.0

1 0

[PATCH lkp-tests 1/2] distro/aliyun: fix yum install args
by Wei Jihui 15 Mar '21

15 Mar '21

[error] Ignored option -q, -v, -d or -e (probably due to merging: -yq != -y -q) Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/aliyun | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 83f058c3..253121e0 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -9,14 +9,14 @@ shopt -s nullglob support_repo_epel() { if ! rpm -q epel-release >/dev/null; then - yum install -yq epel-release + yum install -y -q epel-release fi } update_yum_into_dnf() { if ! rpm -q dnf >/dev/null; then - yum install -yq dnf + yum install -y -q dnf ln -sf /usr/bin/dnf /usr/bin/yum else dnf makecache timer >/dev/null @@ -46,7 +46,7 @@ distro_install_depends() for pkg in $packages do - rpm --quiet --query $pkg || yum install -yq $pkg + rpm --quiet --query $pkg || yum install -y -q $pkg done done } @@ -73,8 +73,8 @@ yum_download() [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -yq --downloadonly $*" - yum install -yq --downloadonly "$@" + echo "yum install -y -q --downloadonly $*" + yum install -y -q --downloadonly "$@" } yum_pack() @@ -137,9 +137,9 @@ install_gcc7() wget -O /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm https://rpmfind.net/linux/centos/7/extras/x86_64/Packages/centos-release-sc… rpm -ivh --ignoresize /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm else - yum install -yq centos-release-scl-rh + yum install -y -q centos-release-scl-rh fi - yum install -yq "devtoolset-7-gcc*" + yum install -y -q "devtoolset-7-gcc*" source /opt/rh/devtoolset-7/enable } @@ -148,7 +148,7 @@ download_ocfs2_tools() local pkgs=$1 mkdir -p /opt/rpms - yum install -yq wget + yum install -y -q wget wget -nc -P /opt/rpms https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/ocfs2-… @@ -160,7 +160,7 @@ download_ocfs2_tools() fi # Try to download the dependences of ocfs2-tools/ocfs2-tools-devel - yum install -yq --downloadonly "$download_pkgs" + yum install -y -q --downloadonly "$download_pkgs" } install_ocfs2_tools() @@ -169,8 +169,8 @@ install_ocfs2_tools() download_ocfs2_tools "$pkgs" - yum install -yq /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm + yum install -y -q /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm echo "$pkgs" | grep -q 'ocfs2-tools-devel' && \ - yum install -yq /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm + yum install -y -q /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm } -- 2.23.0

1 0

[PATCH compass-ci] fix: port conflict between srv-http-cci and scheduler_https
by Yu Chuan 15 Mar '21

15 Mar '21

[Why] container/scheduler_https already use 11306 port of host, so change the port of container/srv-http-cci to 11307. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/scheduler/start | 2 +- container/srv-http/start-cci | 2 +- sparrow/3-code/dev-env | 2 +- src/scheduler/constants.cr | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index d83fcc21fb0b..e655dd4a94b2 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -37,7 +37,7 @@ defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' -defaults['SRV_HTTP_CCI_PORT'] ||= '11306' +defaults['SRV_HTTP_CCI_PORT'] ||= '11307' env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci index 78ab1490795b..84864761e689 100755 --- a/container/srv-http/start-cci +++ b/container/srv-http/start-cci @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-cci - -p 11306:11300 + -p 11307:11300 -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 403ccb127813..93a1667e73b1 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,7 +18,7 @@ server_name=$(hostname | cut -f1 -d.) : ${SRV_HTTP_RESULT_PORT:=11300} : ${SRV_HTTP_OS_PORT:=11304} : ${SRV_HTTP_GIT_PORT:=11305} -: ${SRV_HTTP_CCI_PORT:=11306} +: ${SRV_HTTP_CCI_PORT:=11307} : ${LOGGING_ES_HOST:=$server_ip} : ${LOGGING_ES_PORT:=9202} : ${INITRD_HTTP_HOST:=$server_ip} diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 3fdb9839e90d..a5e52b8f7a6d 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -26,7 +26,7 @@ OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1 OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 SRV_HTTP_CCI_HOST = (ENV.has_key?("SRV_HTTP_CCI_HOST") ? ENV["SRV_HTTP_CCI_HOST"] : "172.17.0.1") -SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11306") +SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11307") CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") LAB_REPO = "lab-z9" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package/common: install bunny for mq
by Wu Zhende 15 Mar '21

15 Mar '21

multi-qemu and multi-docker need bunny to connect rabbitmq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 0324f58..8942247 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer -- 2.23.0

1 0

[PATCH v3 compass-ci] providers/multi-qemu: optimize
by Wu Zhende 15 Mar '21

15 Mar '21

1. optimize function name 2. the host and port are transferred when the MQClient is initalized 3. reboot vm cyclically Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 5979cc9..d3f8406 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -104,8 +104,19 @@ def save_pid(arr) f.close end -def reboot_deadline_machine(hostname) - mq = MQClient.new +def loop_reboot_vm(hostname) + loop do + begin + reboot_vm(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_vm(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) queue = mq.queue(hostname, {:durable => true}) queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| puts msg @@ -118,10 +129,7 @@ end def multiqemu Process.fork do - reboot_deadline_machine(HOSTNAME) - rescue StandardError => e - puts e.backtrace - sleep 5 + loop_reboot_vm(HOSTNAME) end pids = [] -- 2.23.0

2 1

2024

2023

2022

2021

2020

Compass-ci