- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] src/lib/job.cr: change rpm_repo to upstream_repo
by Li Ping 16 Mar '21

16 Mar '21

src/lib/job.cr: open the path for uploading rpm packages. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9e18a4c..40aaa62 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -278,7 +278,7 @@ class Job mount_type = "nfs" if mount_type == "cifs" common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") + common_dir = "#{tmp_os}-#{tmp_os_version}" if @hash.has_key?("rpmbuild-pkg") return common_dir end @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 1

[PATCH v2 compass-ci] src/lib/job.cr: open the path for uploading rpm packages.
by Li Ping 16 Mar '21

16 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9e18a4c..40aaa62 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -278,7 +278,7 @@ class Job mount_type = "nfs" if mount_type == "cifs" common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") + common_dir = "#{tmp_os}-#{tmp_os_version}" if @hash.has_key?("rpmbuild-pkg") return common_dir end @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/compare-results.zh.md: translate to Chinese
by Lin Jiaxin 16 Mar '21

16 Mar '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- doc/manual/compare-results.zh.md | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/doc/manual/compare-results.zh.md b/doc/manual/compare-results.zh.md index 1b43d6b..b96add9 100644 --- a/doc/manual/compare-results.zh.md +++ b/doc/manual/compare-results.zh.md @@ -1,29 +1,26 @@ -# Introduction of compare feature - The compare feature used to analyze the results of different jobs, showing the performance waves and changes - under different influencing factors, for users to analyze performance influencing factor. +# 比较功能介绍 + 比较功能用于分析不同任务的结果，显示不同影响因素下的性能波动和变化，供用户分析性能影响因素。 ## url: https://compass-ci.openeuler.org/compare -## options directions -### filters: +## 选项说明 +### 过滤器: - suite: iperf, netperf, mysql, ... - OS: openeuler 20.03, centos 7.6, ... - os_arch: aarch64, x86 - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... - we can combine the above options arbitrarily to limit compare scope. - choose at least one option as filter. + 我们可以任意组合以上选项来限制比较范围，但至少选择一个选项作为过滤器。 -### dimension - Dimension can select: os, os_version, os_arch, suite, tbox_group. - Within filter, we will compare all different job result by dimension - and keep other test conditions are same. +### 维度 + 可选维度: os, os_version, os_arch, suite, tbox_group. + 在过滤器中，我们将按所选维度比较所有不同的任务结果，并保持其他测试条件一致。 -## example: - filter: suite = iperf - dimension: os_version +## 示例: + 过滤器: suite = iperf + 维度: os_version - result: - os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + 结果: + os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # 其他测试条件一致 20.09 20.03 metric -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/web-backend-nginx: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend-nginx/nginx.conf | 2 +- container/web-backend-nginx/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/web-backend-nginx/nginx.conf b/container/web-backend-nginx/nginx.conf index c9a4ec4..50ced4c 100644 --- a/container/web-backend-nginx/nginx.conf +++ b/container/web-backend-nginx/nginx.conf @@ -8,7 +8,7 @@ http { default_type application/octet-stream; include /etc/nginx/conf.d/*.conf; server { - listen 11320 ssl; + listen 20003 ssl; server_name api.compass-ci.openeuler.org; ssl_certificate /opt/cert/web-backend.crt; ssl_certificate_key /opt/cert/web-backend.key; diff --git a/container/web-backend-nginx/start b/container/web-backend-nginx/start index 4da4c8b..5360385 100755 --- a/container/web-backend-nginx/start +++ b/container/web-backend-nginx/start @@ -10,7 +10,7 @@ docker_rm web-backend-nginx cmd=( docker run --name web-backend-nginx - -p 11320:11320 + -p 20003:20003 -d -v /etc/ssl/certs:/opt/cert -- 2.23.0

1 1

[PATCH compass-ci 1/2] container/web-backend: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/start | 2 +- container/web-backend/web-backend | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/web-backend/start b/container/web-backend/start index 6731d27..e680255 100755 --- a/container/web-backend/start +++ b/container/web-backend/start @@ -18,7 +18,7 @@ cmd=( --name web-backend --restart=always -d - -p 32767:32767 + -p 10002:10002 -e ES_PORT=9200 -e SRV_HTTP_RESULT_HOST=$SRV_HTTP_RESULT_HOST diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 74eda85..33a3fae 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -9,7 +9,7 @@ CCI_SRC = ENV['CCI_SRC'] || '/c/compass-ci' require "#{CCI_SRC}/src/lib/web_backend.rb" set :bind, '0.0.0.0' -set :port, 32767 +set :port, 10002 set :show_exceptions, false set :environment, :production -- 2.23.0

1 1

[PATCH compass-ci 2/2] answerback-email: update tool with new port
by Luan Shengde 16 Mar '21

16 Mar '21

assign-account service has assigned new port: 29999 => 10000 send-internet-mail service has assigned new port: 49000 => 10001 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index f573f0f..f4b6ef3 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -28,9 +28,9 @@ names = Set.new %w[ defaults = relevant_defaults(names) JUMPER_HOST ||= defaults['JUMPER_HOST'] -JUMPER_PORT ||= defaults['JUMPER_PORT'] || 29999 +JUMPER_PORT ||= defaults['JUMPER_PORT'] || 10000 SEND_MAIL_HOST ||= defaults['SEND_MAIL_HOST'] || 'localhost' -SEND_MAIL_PORT ||= defaults['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_PORT ||= defaults['SEND_MAIL_PORT'] || 10001 LAB ||= defaults['lab'] TOOL_SERVER = 'z9' -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/assign-account: update for new port
by Luan Shengde 16 Mar '21

16 Mar '21

the assign-account servive has been assigned new port: 29999 => 10000 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 2 +- container/assign-account/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index c5d77db..abbab0f 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -10,7 +10,7 @@ require 'yaml' require_relative 'get_account_info' set :bind, '0.0.0.0' -set :port, 29999 +set :port, 10000 post '/assign_account' do begin diff --git a/container/assign-account/start b/container/assign-account/start index 793191a..b926853 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -14,7 +14,7 @@ cmd=( -v /etc:/etc:rw -v /srv/account-info:/opt/account_data/:rw -v /home:/home:rw - -p 29999:29999 + -p 10000:10000 assign-account ) -- 2.23.0

1 0

[PATCH compass-ci] constants: change service default ports
by Cao Xueliang 16 Mar '21

16 Mar '21

Reference: 6354ba631e0fd5f795a1dc28b65e2002bd3de668 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/constants.rb | 6 +++--- src/scheduler/constants.cr | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index 8d8e488..6c30aaf 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -14,9 +14,9 @@ SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_RESULT_HOST ||= config['SRV_HTTP_RESULT_HOST'] || ENV['SRV_HTTP_RESULT_HOST'] || '172.17.0.1' SRV_HTTP_OS_HOST ||= config['SRV_HTTP_OS_HOST'] || ENV['SRV_HTTP_OS_HOST'] || '172.17.0.1' SRV_HTTP_GIT_HOST ||= config['SRV_HTTP_GIT_HOST'] || ENV['SRV_HTTP_GIT_HOST'] || '172.17.0.1' -SRV_HTTP_RESULT_PORT ||= config['SRV_HTTP_RESULT_PORT'] || ENV['SRV_HTTP_RESULT_PORT'] || 11300 -SRV_HTTP_OS_PORT ||= config['SRV_HTTP_OS_PORT'] || ENV['SRV_HTTP_OS_PORT'] || 11304 -SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || 11305 +SRV_HTTP_RESULT_PORT ||= config['SRV_HTTP_RESULT_PORT'] || ENV['SRV_HTTP_RESULT_PORT'] || 20007 +SRV_HTTP_OS_PORT ||= config['SRV_HTTP_OS_PORT'] || ENV['SRV_HTTP_OS_PORT'] || 20009 +SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || 20010 SRV_HTTP_DOMAIN ||= config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index a5e52b8..d18c88d 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -26,7 +26,7 @@ OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1 OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 SRV_HTTP_CCI_HOST = (ENV.has_key?("SRV_HTTP_CCI_HOST") ? ENV["SRV_HTTP_CCI_HOST"] : "172.17.0.1") -SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11307") +SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : 20011).to_i32 CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") LAB_REPO = "lab-z9" -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: change service ports
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/start-cci | 2 +- container/srv-http/start-git | 2 +- container/srv-http/start-initrd | 2 +- container/srv-http/start-os | 2 +- container/srv-http/start-pub | 2 +- container/srv-http/start-result | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci index 8486476..00d8e4c 100755 --- a/container/srv-http/start-cci +++ b/container/srv-http/start-cci @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-cci - -p 11307:11300 + -p 20011:11300 -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-git b/container/srv-http/start-git index f2292bf..c1181a7 100755 --- a/container/srv-http/start-git +++ b/container/srv-http/start-git @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-git - -p 11305:11300 + -p 20010:11300 -v /srv/git/archlinux:/srv/git/archlinux:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 495cc25..466ec1b 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-initrd - -p 11303:11300 + -p 20008:11300 -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro diff --git a/container/srv-http/start-os b/container/srv-http/start-os index bb344cf..d39e04e 100755 --- a/container/srv-http/start-os +++ b/container/srv-http/start-os @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-os - -p 11304:11300 + -p 20009:11300 -v /srv/os:/srv/os:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-pub b/container/srv-http/start-pub index dc1864c..f42b117 100755 --- a/container/srv-http/start-pub +++ b/container/srv-http/start-pub @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-pub - -p 11302:11300 + -p 20006:11300 -v /srv/pub:/srv/pub:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-result b/container/srv-http/start-result index f9ecc97..bf0287d 100755 --- a/container/srv-http/start-result +++ b/container/srv-http/start-result @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-result - -p 11300:11300 + -p 20007:11300 -v /srv/result:/srv/result:ro -v /etc/localtime:/etc/localtime:ro -d -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: open the path for uploading rpm packages.
by Li Ping 16 Mar '21

16 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index f8cfa97..40aaa62 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -278,7 +278,7 @@ class Job mount_type = "nfs" if mount_type == "cifs" common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") + common_dir = "#{tmp_os}-#{tmp_os_version}" if @hash.has_key?("rpmbuild-pkg") return common_dir end @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["upstream_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 16 Mar '21

16 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 83 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..9ecb4d2c5 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,83 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + echo $ansible_host + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] || { + echo "Ansible failed to install." + exit 1 + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + exit 1 + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook is running failed!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

1 1

[PATCH v4 compass-ci 2/2] multi-docker: add reboot function
by Wu Zhende 16 Mar '21

16 Mar '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 34 +++++++++++++++++++++++++++++++++- providers/docker/run.sh | 2 ++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index f4e954e..9d6854f 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -7,6 +7,8 @@ require 'open-uri' require 'json' require 'set' require 'fileutils' + +require '../lib/mq_client' require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' @@ -19,6 +21,8 @@ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 LOG_DIR = '/srv/cci/serial/logs' +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || 'localhost' +MQ_PORT = ENV['MQ_PORT'] || 5672 def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -84,7 +88,8 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, + { 'job_id' => hash['job_id'], + 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path, 'log_dir' => "#{LOG_DIR}/#{hostname}" @@ -145,6 +150,29 @@ def loop_main(hostname, queues) end end +def loop_reboot_docker(hostname) + loop do + begin + reboot_docker(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_docker(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) + queue = mq.queue(hostname, {:durable => true}) + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + puts msg + machine_info = JSON.parse(msg) + job_id = machine_info['job_id'] + system "docker rm -f #{job_id};echo $?" + mq.ack(info) + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -153,6 +181,10 @@ def save_pid(pids) end def multi_docker(hostname, nr_container, queues) + Process.fork do + loop_reboot_docker(hostname) + end + pids = [] nr_container.to_i.times do |i| pid = Process.fork do diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 2dfe36d..80950dd 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -4,6 +4,7 @@ . $LKP_SRC/lib/yaml.sh +: ${job_id:=$$} : ${docker_image:="centos:7"} : ${load_path:="${HOME}/jobs"} : ${hostname:="dc-1g-1"} @@ -23,6 +24,7 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm + --name ${job_id} --hostname $host.compass-ci.net -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 16 Mar '21

16 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH v4 compass-ci] providers/multi-qemu: optimize
by Wu Zhende 16 Mar '21

16 Mar '21

1. optimize function name 2. the host and port are transferred when the MQClient is initalized 3. reboot vm cyclically Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 5979cc9..d3f8406 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -104,8 +104,19 @@ def save_pid(arr) f.close end -def reboot_deadline_machine(hostname) - mq = MQClient.new +def loop_reboot_vm(hostname) + loop do + begin + reboot_vm(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_vm(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) queue = mq.queue(hostname, {:durable => true}) queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| puts msg @@ -118,10 +129,7 @@ end def multiqemu Process.fork do - reboot_deadline_machine(HOSTNAME) - rescue StandardError => e - puts e.backtrace - sleep 5 + loop_reboot_vm(HOSTNAME) end pids = [] -- 2.23.0

1 0

[PATCH compass-ci] container/webhook: change the service port
by Li Yuanchao 15 Mar '21

15 Mar '21

For the port policy. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/webhook/start b/container/webhook/start index 373396a..1d121a4 100755 --- a/container/webhook/start +++ b/container/webhook/start @@ -12,7 +12,7 @@ cmd = %w[ --restart=always --name webhook -d - -p 11301:11301 + -p 20005:20005 node:webhook ] -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: change the local git repo path
by Lin Jiaxin 15 Mar '21

15 Mar '21

As the path of repo mirrors under /srv/git changes, introduce upstream_dir to get new path. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 43932865..350d8fdf 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -6,6 +6,7 @@ # - pkg_ver # - config # upstream_url +# upstream_dir # We'll use the below global env vars, but not treat them as parameters to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ @@ -123,9 +124,9 @@ replace_source() do echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { - url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" + url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#commit=${upstream_commit}" [ -n "$git_tag" ] && { - url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#tag=${git_tag}" + url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#tag=${git_tag}" } } } -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/auto_submit: change the upstream_repo path to submit
by Lin Jiaxin 15 Mar '21

15 Mar '21

Reference: 5d6cea1bd531409f0444abf14088fdecd8b08c2d Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index faebb2f..a27075b 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -33,7 +33,7 @@ class AutoSubmit git_repo = newrefs_info['git_repo'] puts "git_repo : #{git_repo}" url = Array(newrefs_info['url'])[0] - submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{git_repo} upstream_url=#{url}"] + submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{newrefs_info['belong']}/#{git_repo} upstream_url=#{url}"] if newrefs_info['pkgbuild_repo'] pkgbuild_repo = get_pkgbuild_repo(newrefs_info['pkgbuild_repo']) -- 2.23.0

2 2

[PATCH compass-ci] optimize doc
by Luan Shengde 15 Mar '21

15 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-tests-container.zh.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/build-lkp-tests-container.zh.md b/doc/manual/build-lkp-tests-container.zh.md index 10a1611..407ddbf 100644 --- a/doc/manual/build-lkp-tests-container.zh.md +++ b/doc/manual/build-lkp-tests-container.zh.md @@ -60,4 +60,4 @@ submit命令： - 了解更多submit命令的使用方法，请参考：[submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… + 了解更多submit命令的使用方法，请参考：[submit用户手册](https://gitee.com/wu_fengguang/compass-c… -- 2.23.0

1 0

[PATCH compass-ci 2/2] service/scheduler: fix the mq connection exception
by Wu Zhende 15 Mar '21

15 Mar '21

[Why] If the connection to the mq service is disconnected, the interface for sending messages is abnormal, affecting the main logic. [How] Start a thread to send mq messages. Reconnect and send every 5 seconds, three attempts. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 10 ++++++++++ src/scheduler/close_job.cr | 2 +- src/scheduler/find_job_boot.cr | 2 +- src/scheduler/update_job_parameter.cr | 2 +- 4 files changed, 13 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f5aba03..9528f54 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -218,4 +218,14 @@ class Sched return nil end + + def mq_pushlish_confirm(queue, msg) + 3.times do + @mq.pushlish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end end diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index ea1dbf5..e014361 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -45,7 +45,7 @@ class Sched "job_state" => "close", "time" => get_time } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..9e83537 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -32,7 +32,7 @@ class Sched "time" => get_time, "job_state" => "boot" } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end # auto submit a job to collect the host information. diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index b43aa06..d81f28b 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -44,6 +44,6 @@ class Sched "job_state" => (@env.get?("job_state") || "update").to_s, "time" => @env.get?("time").to_s } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/mq.cr: add new function reconnect
by Wu Zhende 15 Mar '21

15 Mar '21

When the connection to the mq is disconnected, an interface needs to be provided to re-initiate the connection. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/mq.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/lib/mq.cr b/src/lib/mq.cr index 87c67e9..980585d 100644 --- a/src/lib/mq.cr +++ b/src/lib/mq.cr @@ -5,17 +5,34 @@ require "singleton" require "amqp-client" require "amq-protocol" +require "./json_logger" + class MQClient getter ch : AMQP::Client::Channel MQ_HOST = ENV.has_key?("MQ_HOST") ? ENV["MQ_HOST"] : "172.17.0.1" MQ_PORT = (ENV.has_key?("MQ_PORT") ? ENV["MQ_PORT"] : 5672).to_i32 def initialize(host = MQ_HOST, port = MQ_PORT) + @log = JSONLogger.new @client = AMQP::Client.new("amqp://#{host}:#{port}") conn = @client.connect @ch = conn.channel.as(AMQP::Client::Channel) end + def reconnect + conn = @client.connect + @ch = conn.channel.as(AMQP::Client::Channel) + @log.info({ + "msg" => "rabbitmq reconnected successfully", + "source" => "mq_client" + }.to_json) + rescue e + @log.warn({ + "msg" => e.to_s, + "source" => "mq_client" + }.to_json) + end + def self.instance Singleton::Of(self).instance end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 15 Mar '21

15 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 83 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..43a9ce341 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,83 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + echo $ansible_host + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] || { + echo "Ansible failed to install." + exit 1 + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + exit 1 + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook failed to run!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

2 2

[PATCH lkp-tests 2/2] distro/aliyun: revoke commit e21801797 and use a new approach
by Wei Jihui 15 Mar '21

15 Mar '21

commit:e21801797 is for use dnf instead of yum in centos7 use dnf it can raise error when yum install print "No package * available" [why] install dnf will installs many dependency packages. this makes cci-depends can not download some packages. [how] yum install do not use dnf and analyze installation output, if has "No package * available" then exit 1 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/aliyun | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 253121e0..24f16287 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -13,16 +13,6 @@ support_repo_epel() fi } -update_yum_into_dnf() -{ - if ! rpm -q dnf >/dev/null; then - yum install -y -q dnf - ln -sf /usr/bin/dnf /usr/bin/yum - else - dnf makecache timer >/dev/null - fi -} - distro_install_depends() { local script @@ -30,7 +20,6 @@ distro_install_depends() local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel - [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf for script in $scripts do @@ -71,10 +60,10 @@ pack_benchmark() yum_download() { [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel - [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -y -q --downloadonly $*" - yum install -y -q --downloadonly "$@" + echo "yum install -y --downloadonly $@ | grep -e \"No\ package .*\ available\"" + yum install -y --downloadonly "$@" | grep -e "No\ package .*\ available" && exit 1 + return 0 } yum_pack() -- 2.23.0

1 0

[PATCH lkp-tests 1/2] distro/aliyun: fix yum install args
by Wei Jihui 15 Mar '21

15 Mar '21

[error] Ignored option -q, -v, -d or -e (probably due to merging: -yq != -y -q) Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/aliyun | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 83f058c3..253121e0 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -9,14 +9,14 @@ shopt -s nullglob support_repo_epel() { if ! rpm -q epel-release >/dev/null; then - yum install -yq epel-release + yum install -y -q epel-release fi } update_yum_into_dnf() { if ! rpm -q dnf >/dev/null; then - yum install -yq dnf + yum install -y -q dnf ln -sf /usr/bin/dnf /usr/bin/yum else dnf makecache timer >/dev/null @@ -46,7 +46,7 @@ distro_install_depends() for pkg in $packages do - rpm --quiet --query $pkg || yum install -yq $pkg + rpm --quiet --query $pkg || yum install -y -q $pkg done done } @@ -73,8 +73,8 @@ yum_download() [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -yq --downloadonly $*" - yum install -yq --downloadonly "$@" + echo "yum install -y -q --downloadonly $*" + yum install -y -q --downloadonly "$@" } yum_pack() @@ -137,9 +137,9 @@ install_gcc7() wget -O /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm https://rpmfind.net/linux/centos/7/extras/x86_64/Packages/centos-release-sc… rpm -ivh --ignoresize /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm else - yum install -yq centos-release-scl-rh + yum install -y -q centos-release-scl-rh fi - yum install -yq "devtoolset-7-gcc*" + yum install -y -q "devtoolset-7-gcc*" source /opt/rh/devtoolset-7/enable } @@ -148,7 +148,7 @@ download_ocfs2_tools() local pkgs=$1 mkdir -p /opt/rpms - yum install -yq wget + yum install -y -q wget wget -nc -P /opt/rpms https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/ocfs2-… @@ -160,7 +160,7 @@ download_ocfs2_tools() fi # Try to download the dependences of ocfs2-tools/ocfs2-tools-devel - yum install -yq --downloadonly "$download_pkgs" + yum install -y -q --downloadonly "$download_pkgs" } install_ocfs2_tools() @@ -169,8 +169,8 @@ install_ocfs2_tools() download_ocfs2_tools "$pkgs" - yum install -yq /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm + yum install -y -q /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm echo "$pkgs" | grep -q 'ocfs2-tools-devel' && \ - yum install -yq /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm + yum install -y -q /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm } -- 2.23.0

1 0

[PATCH compass-ci] fix: port conflict between srv-http-cci and scheduler_https
by Yu Chuan 15 Mar '21

15 Mar '21

[Why] container/scheduler_https already use 11306 port of host, so change the port of container/srv-http-cci to 11307. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/scheduler/start | 2 +- container/srv-http/start-cci | 2 +- sparrow/3-code/dev-env | 2 +- src/scheduler/constants.cr | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index d83fcc21fb0b..e655dd4a94b2 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -37,7 +37,7 @@ defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' -defaults['SRV_HTTP_CCI_PORT'] ||= '11306' +defaults['SRV_HTTP_CCI_PORT'] ||= '11307' env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci index 78ab1490795b..84864761e689 100755 --- a/container/srv-http/start-cci +++ b/container/srv-http/start-cci @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-cci - -p 11306:11300 + -p 11307:11300 -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 403ccb127813..93a1667e73b1 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,7 +18,7 @@ server_name=$(hostname | cut -f1 -d.) : ${SRV_HTTP_RESULT_PORT:=11300} : ${SRV_HTTP_OS_PORT:=11304} : ${SRV_HTTP_GIT_PORT:=11305} -: ${SRV_HTTP_CCI_PORT:=11306} +: ${SRV_HTTP_CCI_PORT:=11307} : ${LOGGING_ES_HOST:=$server_ip} : ${LOGGING_ES_PORT:=9202} : ${INITRD_HTTP_HOST:=$server_ip} diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 3fdb9839e90d..a5e52b8f7a6d 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -26,7 +26,7 @@ OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1 OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 SRV_HTTP_CCI_HOST = (ENV.has_key?("SRV_HTTP_CCI_HOST") ? ENV["SRV_HTTP_CCI_HOST"] : "172.17.0.1") -SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11306") +SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11307") CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") LAB_REPO = "lab-z9" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package/common: install bunny for mq
by Wu Zhende 15 Mar '21

15 Mar '21

multi-qemu and multi-docker need bunny to connect rabbitmq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 0324f58..8942247 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer -- 2.23.0

1 0

[PATCH v3 compass-ci] providers/multi-qemu: optimize
by Wu Zhende 15 Mar '21

15 Mar '21

1. optimize function name 2. the host and port are transferred when the MQClient is initalized 3. reboot vm cyclically Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 5979cc9..d3f8406 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -104,8 +104,19 @@ def save_pid(arr) f.close end -def reboot_deadline_machine(hostname) - mq = MQClient.new +def loop_reboot_vm(hostname) + loop do + begin + reboot_vm(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_vm(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) queue = mq.queue(hostname, {:durable => true}) queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| puts msg @@ -118,10 +129,7 @@ end def multiqemu Process.fork do - reboot_deadline_machine(HOSTNAME) - rescue StandardError => e - puts e.backtrace - sleep 5 + loop_reboot_vm(HOSTNAME) end pids = [] -- 2.23.0

2 1

[PATCH v3 lkp-tests 1/2] tests/rpmbuild-pkg: change rpmbuild-pkg upload_dir
by Li Ping 15 Mar '21

15 Mar '21

--- tests/rpmbuild-pkg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index b641e97d1..a8c8eb6b1 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,7 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" +sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() { @@ -24,7 +24,7 @@ download_rpm_repo() { local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in "${filelist[@]}" + for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" -- 2.23.0

2 4

[PATCH v2 compass-ci] providers/multi-qemu: optimize
by Wu Zhende 15 Mar '21

15 Mar '21

1.optimize function name 2.the host and port are transferred when the MQClient is initalized 3.reboot vm cyclically Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 5979cc9..d3f8406 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -104,8 +104,19 @@ def save_pid(arr) f.close end -def reboot_deadline_machine(hostname) - mq = MQClient.new +def loop_reboot_vm(hostname) + loop do + begin + reboot_vm(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_vm(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) queue = mq.queue(hostname, {:durable => true}) queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| puts msg @@ -118,10 +129,7 @@ end def multiqemu Process.fork do - reboot_deadline_machine(HOSTNAME) - rescue StandardError => e - puts e.backtrace - sleep 5 + loop_reboot_vm(HOSTNAME) end pids = [] -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] tests/rpmbuild-pkg: adjusting code for auto_submit
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 1 - tests/rpmbuild-pkg | 23 +++++++++-------------- 2 files changed, 9 insertions(+), 15 deletions(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index 53e145b3b..5860ba029 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,4 @@ suite: rpmbuild-pkg category: functional -rpm_repo: rpmbuild-pkg: diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index a8c8eb6b1..e51c75aa2 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,11 +1,11 @@ #!/bin/bash -# - rpm_repo +# - upstream_repo . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh -[ -n "$rpm_repo" ] || die "rpm_repo is empty" -package_name=${rpm_repo##*/} +[ -n "$upstream_repo" ] || die "upstream_repo is empty" +package_name=${upstream_repo##*/} sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() @@ -14,21 +14,16 @@ init_workspace() rpmdev-setuptree } -get_pkgfile() +download_upstream_repo() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" -} - -download_rpm_repo() -{ - local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') + git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" + cd $package_name || exit + filelist=$(git ls-files) for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" - get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + mv "$pkgfile" "${HOME}/rpmbuild/${dir}/" done } @@ -63,6 +58,6 @@ upload_rpm_pkg() } init_workspace -download_rpm_repo +download_upstream_repo build_rpm upload_rpm_pkg -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] tests/rpmbuild-pkg: adjusting code for auto_submit
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 2 +- tests/rpmbuild-pkg | 23 +++++++++-------------- 2 files changed, 10 insertions(+), 15 deletions(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index 53e145b3b..a70590afe 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,5 @@ suite: rpmbuild-pkg category: functional -rpm_repo: +upstream_repo: rpmbuild-pkg: diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index a8c8eb6b1..d49dc43c7 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,11 +1,11 @@ #!/bin/bash -# - rpm_repo +# - upstream_repo . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh -[ -n "$rpm_repo" ] || die "rpm_repo is empty" -package_name=${rpm_repo##*/} +[ -n "$upstream_repo" ] || die "upstream_repo is empty" +package_name=${upstream_repo##*/} sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() @@ -14,21 +14,16 @@ init_workspace() rpmdev-setuptree } -get_pkgfile() +download_upstream_repo() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" -} - -download_rpm_repo() -{ - local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') + git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" + cd $package_name || exit + filelist=$(git ls-files) for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" - get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + cat "$pkgfile" > "${HOME}/rpmbuild/${dir}/$pkgfile" done } @@ -63,6 +58,6 @@ upload_rpm_pkg() } init_workspace -download_rpm_repo +download_upstream_repo build_rpm upload_rpm_pkg -- 2.23.0

2 2

[PATCH v2 compass-ci] result-webdav-webdav/nginx.conf: do not overwrite the uploaded config with the same name
by Lin Jiaxin 12 Mar '21

12 Mar '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- container/result-webdav/nginx.conf | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index cc21926..e0e0ee2 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -27,6 +27,22 @@ http { dav_methods PUT MKCOL; dav_access user:rw group:rw all:rw; + set $config_upload_condition "false"; + if ( $uri ~ "^/cci/build-config.*" ) + { + set $config_upload_condition "true"; + } + + if ( -e $request_filename ) + { + set $config_upload_condition "${config_upload_condition}+true"; + } + + if ( $config_upload_condition = "true+true" ) + { + return 200; + } + access_by_lua_block { function split(str, reps) local resultstrlist = {} -- 2.23.0

1 0

[PATCH compass-ci 2/2] ult-webdav/nginx.conf: do not overwrite the uploaded config with the same name
by Lin Jiaxin 12 Mar '21

12 Mar '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- container/result-webdav/nginx.conf | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index cc21926..e0e0ee2 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -27,6 +27,22 @@ http { dav_methods PUT MKCOL; dav_access user:rw group:rw all:rw; + set $config_upload_condition "false"; + if ( $uri ~ "^/cci/build-config.*" ) + { + set $config_upload_condition "true"; + } + + if ( -e $request_filename ) + { + set $config_upload_condition "${config_upload_condition}+true"; + } + + if ( $config_upload_condition = "true+true" ) + { + return 200; + } + access_by_lua_block { function split(str, reps) local resultstrlist = {} -- 2.23.0

1 1

[PATCH compass-ci] src/lib/job.cr: change rpm_repo to upstream_repo
by Li Ping 12 Mar '21

12 Mar '21

--- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9e18a4c..f8cfa97 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["upstream_repo"].to_s.split("/")[-1]}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] tests/rpmbuild-pkg: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

--- tests/rpmbuild-pkg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index b641e97d1..a8c8eb6b1 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,7 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" +sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() { @@ -24,7 +24,7 @@ download_rpm_repo() { local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in "${filelist[@]}" + for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" -- 2.23.0

1 0

[PATCH compass-ci] providers/multi-qemu: optimize
by Wu Zhende 12 Mar '21

12 Mar '21

1.optimize function name 2.the host and port are transferred when the MQClient is initalized 3.reboot vm cyclically Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 5979cc9..d3f8406 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -104,8 +104,19 @@ def save_pid(arr) f.close end -def reboot_deadline_machine(hostname) - mq = MQClient.new +def loop_reboot_vm(hostname) + loop do + begin + reboot_vm(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_vm(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) queue = mq.queue(hostname, {:durable => true}) queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| puts msg @@ -118,10 +129,7 @@ end def multiqemu Process.fork do - reboot_deadline_machine(HOSTNAME) - rescue StandardError => e - puts e.backtrace - sleep 5 + loop_reboot_vm(HOSTNAME) end pids = [] -- 2.23.0

2 1

[PATCH lkp-tests] tests/rpmbuild-pkg: adjusting code for auto_submit
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- tests/rpmbuild-pkg | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index a8c8eb6b1..f49ba3a0b 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,11 +1,11 @@ #!/bin/bash -# - rpm_repo +# - upstream_repo . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh -[ -n "$rpm_repo" ] || die "rpm_repo is empty" -package_name=${rpm_repo##*/} +[ -n "$upstream_repo" ] || die "upstream_repo is empty" +package_name=${upstream_repo##*/} sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() @@ -14,21 +14,16 @@ init_workspace() rpmdev-setuptree } -get_pkgfile() +download_upstream_repo() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" -} - -download_rpm_repo() -{ - local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') + git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone ${package_name} git repo failed: git://$GIT_SERVER/openeuler/${upstream_repo}" + cd $package_name || exit + filelist=$(git ls-files) for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" - get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + cat "$pkgfile" > "${HOME}/rpmbuild/${dir}/$pkgfile" done } @@ -63,6 +58,6 @@ upload_rpm_pkg() } init_workspace -download_rpm_repo +download_upstream_repo build_rpm upload_rpm_pkg -- 2.23.0

2 2

[PATCH compass-ci] container/dracut-initrd: add basename into openeuler's 21xx+ kernel
by Xu Xijian 12 Mar '21

12 Mar '21

[why] It seems openeuler 21xx+ kernel not support cmd basename, our overlay need this cmd, so add basename into dracut module to enable initrd. [errlog] //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 50: basename: command not found //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 65: basename: command not found Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 45e895c..494588a 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -17,7 +17,7 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh +sed -i "/install() {/a\ inst /sbin/mke2fs /usr/bin/basename" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" [ "$(sed -n '$p' $pre_mount_file)" = "exit 0" ] && sed -i '$d' "$pre_mount_file" cat set-local-sysroot.sh >> "$pre_mount_file" -- 2.23.0

2 2

[PATCH v2 compass-ci] src/lib/job.cr: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2177b22..9e18a4c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -277,11 +277,8 @@ class Job # same usage for client mount_type = "nfs" if mount_type == "cifs" - if @hash["rpmbuild-pkg"]? - common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - else - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - end + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") return common_dir end -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2177b22..1fbf646 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -277,11 +277,8 @@ class Job # same usage for client mount_type = "nfs" if mount_type == "cifs" - if @hash["rpmbuild-pkg"]? - common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - else - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - end + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash["rpmbuild-pkg"]? return common_dir end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] multi-docker: add reboot function
by Wu Zhende 12 Mar '21

12 Mar '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 34 +++++++++++++++++++++++++++++++++- providers/docker/run.sh | 2 ++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index f4e954e..9d6854f 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -7,6 +7,8 @@ require 'open-uri' require 'json' require 'set' require 'fileutils' + +require '../lib/mq_client' require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' @@ -19,6 +21,8 @@ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 LOG_DIR = '/srv/cci/serial/logs' +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || 'localhost' +MQ_PORT = ENV['MQ_PORT'] || 5672 def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -84,7 +88,8 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, + { 'job_id' => hash['job_id'], + 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path, 'log_dir' => "#{LOG_DIR}/#{hostname}" @@ -145,6 +150,29 @@ def loop_main(hostname, queues) end end +def loop_reboot_docker(hostname) + loop do + begin + reboot_docker(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_docker(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) + queue = mq.queue(hostname, {:durable => true}) + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + puts msg + machine_info = JSON.parse(msg) + job_id = machine_info['job_id'] + system "docker rm -f #{job_id};echo $?" + mq.ack(info) + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -153,6 +181,10 @@ def save_pid(pids) end def multi_docker(hostname, nr_container, queues) + Process.fork do + loop_reboot_docker(hostname) + end + pids = [] nr_container.to_i.times do |i| pid = Process.fork do diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 2dfe36d..80950dd 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -4,6 +4,7 @@ . $LKP_SRC/lib/yaml.sh +: ${job_id:=$$} : ${docker_image:="centos:7"} : ${load_path:="${HOME}/jobs"} : ${hostname:="dc-1g-1"} @@ -23,6 +24,7 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm + --name ${job_id} --hostname $host.compass-ci.net -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH compass-ci] extract-stats: wait_for refresh doc to ES DB
by Lu Weitao 12 Mar '21

12 Mar '21

[Why] We need update doc(job) to ES completed, then send delimiter task to task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index dbb3fe2..44c327d 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -81,6 +81,7 @@ class StatsWorker { :index => "jobs", :type => "_doc", :id => job_id, + :refresh => "wait_for", :body => {:doc => update_content}, } ) -- 2.23.0

1 0

[PATCH compass-ci] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code/dev-env: fix failed to submit job
by Liu Yinsi 12 Mar '21

12 Mar '21

[error] root@taishan200-2280-2s64p-256g--a123 /c/compass-ci/container# submit host-info.yaml submit_id=9bca6036-4602-44ab-92df-729058ac1818 Traceback (most recent call last): 5: from /c/lkp-tests/sbin/submit:136:in `<main>' 4: from /c/lkp-tests/sbin/submit:136:in `each' 3: from /c/lkp-tests/sbin/submit:150:in `block in <main>' 2: from /c/lkp-tests/lib/job.rb:744:in `each_jobs' 1: from /c/lkp-tests/lib/job.rb:736:in `each_job' /c/lkp-tests/sbin/submit:151:in `block (2 levels) in <main>': Please configure SCHED_HOST (RuntimeError) [why] for user locally deploy compass-ci, env $SCHED_HOST not exists in /etc/compass-ci/defaults/$server_name.yaml, cause failed to submit, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 403ccb1..fdf7894 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -36,6 +36,8 @@ server_name=$(hostname | cut -f1 -d.) mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF +SCHED_HOST: $sched_host +SCHED_PORT: $sched_port lab: nolab EOF -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

--- tests/rpmbuild-pkg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index b641e97d1..a8c8eb6b1 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,7 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" +sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() { @@ -24,7 +24,7 @@ download_rpm_repo() { local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in "${filelist[@]}" + for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] doc/job/os_mount.md: change the fields of job.yaml
by Xu Xijian 11 Mar '21

11 Mar '21

The fields that users need to give is the suffix of partitions, so give them appropriate names. use_root_partition => src_lv_suffix save_root_partition => boot_lv_suffix Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/os_mount.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 6c1637b..eab95ed 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -56,23 +56,25 @@ The brief flow is as follows: ## persistent rootfs data -When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_custom_params` will help you: `save_root_partition`, `use_root_partition`. +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_custom_params` will help you: 'src_lv_suffix', 'boot_lv_suffix'. The brief flow is as follows: ``` 1. boot and request scheduler for job. 2. initrd stage: + use_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${src_lv_suffix} + save_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${boot_lv_suffix} - firstly, we need two logical volume: ${src_lv} -- src logical volume: - - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if have ${use_root_partition}, src_lv=${use_root_partition} - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} ${boot_lv} -- boot logical volume: - - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if have ${save_root_partition}, boot_lv=${save_root_partition} - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} - if ${src_lv} not exists: - - if have ${use_root_partition}, exit 1. - - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if have ${use_root_partition}, reboot. + - if no ${use_root_partition}, create ${src_lv}, and sync the rootfs from cluster nfs server. - if ${boot_lv} != ${src_lv}: - if ${boot_lv} exists: delete it - create ${boot_lv} as the snapshot of ${src_lv}. @@ -87,12 +89,12 @@ Demo usage: rootfs data of job-20210218.yaml so that it can be used by the subsequent jobs. Then you need add the follow field in your job-20210218.yaml: - kernel_custom_params: save_root_partition=zhangsan_local_for_iperf_20210218 + kernel_custom_params: boot_lv_suffix=zhangsan_local_for_iperf_20210218 - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of job-20210218.yaml. Then you need add the follow field in your job-20210219.yaml: - kernel_custom_params: use_root_partition=zhangsan_local_for_iperf_20210218 + kernel_custom_params: src_lv_suffix=zhangsan_local_for_iperf_20210218 ``` Notes: @@ -109,16 +111,19 @@ Notes: os_arch: aarch64 os_version: 20.03 os_mount: local - kernel_custom_params: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + kernel_custom_params: src_lv_suffix=zhangsan_local_for_iperf_20210218 boot_lv_suffix=zhangsan_local_for_iperf_20210219 ``` 2. scheduler return the custom_ipxe to testbox + os_info = "#{os}_#{os_arch}_#{os_version}" + use_root_partition = "/dev/mapper/os-#{os_info}_#{src_lv_suffix}" + save_root_partition = "/dev/mapper/os-#{os_info}_#{boot_lv_suffix}" ``` #!ipxe dhcp initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp - kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_custom_params} + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 initrd=initrd.lkp local use_root_partition=#{use_root_partition} save_root_partition=#{save_root_partition} rw ... boot ``` -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container/dracut-initrd: change someof the handle logic according to scheme doc
by Xu Xijian 11 Mar '21

11 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index dc9000b..240551a 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -5,9 +5,9 @@ analyse_kernel_cmdline_params() { # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') timestamp="$(basename $root)" - os="$(echo $root_src | awk -F '/|-' '{print $2}')" - os_arch="$(echo $root_src | awk -F '/|-' '{print $3}')" - os_version="$(echo $root_src | awk -F '/|-' '{print $4}')" + os="$(echo $rootfs_src | awk -F '/|-' '{print $2}')" + os_arch="$(echo $rootfs_src | awk -F '/|-' '{print $3}')" + os_version="$(echo $rootfs_src | awk -F '/|-' '{print $4}')" os_info="${os}_${os_arch}_${os_version}" export rootfs_src timestamp os_info } @@ -58,22 +58,22 @@ analyse_kernel_cmdline_params sed -i "s/^locking_type = .*/locking_type = 1/" /etc/lvm/lvm.conf use_root_partition="$(getarg use_root_partition=)" -if [ -n "$use_root_partition" ]; then - src_lv="$use_root_partition" # /dev/mapper/os-${os_info}_${timestamp} +if [ -z "$use_root_partition" ]; then + src_lv="/dev/mapper/os-${os_info}_$timestamp" sync_src_lv "$src_lv" else - src_lv="/dev/mapper/os-${os_info}_$timestamp" - [ -e "src_lv" ] || { + src_lv="$use_root_partition" + [ -e "$src_lv" ] || { echo "warn dracut: FATAL: no src_lv with local mount, reboot" reboot } fi save_root_partition="$(getarg save_root_partition=)" -if [ -n "$save_root_partition" ]; then - boot_lv="$save_root_partition" -else +if [ -z "$save_root_partition" ]; then boot_lv="/dev/mapper/os-${os_info}" +else + boot_lv="$save_root_partition" fi snapshot_boot_lv "$src_lv" "$boot_lv" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/dracut-initrd: add basename into openeuler's 21xx+ kernel
by Xu Xijian 11 Mar '21

11 Mar '21

[why] It seems openeuler 21xx+ kernel not support cmd basename, our overlay need this cmd, so add basename into dracut module to enable initrd. [errlog] //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 50: basename: command not found //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 65: basename: command not found Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 45e895c..494588a 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -17,7 +17,7 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh +sed -i "/install() {/a\ inst /sbin/mke2fs /usr/bin/basename" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" [ "$(sed -n '$p' $pre_mount_file)" = "exit 0" ] && sed -i '$d' "$pre_mount_file" cat set-local-sysroot.sh >> "$pre_mount_file" -- 2.23.0

1 0

[PATCH lkp-tests] add multi-qemu-x86.yaml
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/multi-qemu-x86.yaml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 jobs/multi-qemu-x86.yaml diff --git a/jobs/multi-qemu-x86.yaml b/jobs/multi-qemu-x86.yaml new file mode 100644 index 00000000..5539c1ad --- /dev/null +++ b/jobs/multi-qemu-x86.yaml @@ -0,0 +1,23 @@ +suite: multi-qemu +category: functional +simplify-ci: +runtime: 100d +multi-qemu-0: + nr_vm: 5 + tbox_group: vm-2p1g-x86 +multi-qemu-1: + nr_vm: 5 + tbox_group: vm-2p2g-x86 +multi-qemu-2: + nr_vm: 5 + tbox_group: vm-2p4g-x86 +multi-qemu-3: + nr_vm: 10 + tbox_group: vm-2p8g-x86 +multi-qemu-4: + nr_vm: 5 + tbox_group: vm-2p16g-x86 +multi-qemu-5: + nr_vm: 1 + tbox_group: vm-2p32g-x86 +sleep: -- 2.23.0

1 0