- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib: update port
by Luan Shengde 16 Mar '21

16 Mar '21

service assigned new port: send-internet-mail: 49000 => 10001 assign-account: 29999 => 10000 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 4 ++-- lib/constants.rb | 2 +- lib/mail-post.rb | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb index 4831545..f551a9d 100644 --- a/lib/assign_account_client.rb +++ b/lib/assign_account_client.rb @@ -21,9 +21,9 @@ names = Set.new %w[ defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] -JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 +JUMPER_PORT = defaults['JUMPER_PORT'] || 10000 SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 10001 LAB = ENV['lab'] # used for other codes calling to assign account for user diff --git a/lib/constants.rb b/lib/constants.rb index 6c30aaf..03b1e40 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -9,7 +9,7 @@ ES_HOST ||= config['ES_HOST'] || '172.17.0.1' ES_PORT ||= config['ES_PORT'] || 9200 SEND_MAIL_HOST ||= config['SEND_MAIL_HOST'] || '172.17.0.1' -SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 10001 SRV_HTTP_RESULT_HOST ||= config['SRV_HTTP_RESULT_HOST'] || ENV['SRV_HTTP_RESULT_HOST'] || '172.17.0.1' SRV_HTTP_OS_HOST ||= config['SRV_HTTP_OS_HOST'] || ENV['SRV_HTTP_OS_HOST'] || '172.17.0.1' diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 4eb5800..84fc3c1 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -76,7 +76,7 @@ def check_send_mail(mail_info) raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? - return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '10001' mail_info = check_email_mapping(mail_info.clone) mail_info = check_email_limit(mail_info.clone) @@ -101,7 +101,7 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' + return if ENV['SEND_MAIL_PORT'].to_s != '10001' return if ENV['HOST_SERVER'] != 'z9' store_email(mail) -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] container/web-backend-nginx: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend-nginx/nginx.conf | 4 ++-- container/web-backend-nginx/start | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/container/web-backend-nginx/nginx.conf b/container/web-backend-nginx/nginx.conf index c9a4ec4..381888a 100644 --- a/container/web-backend-nginx/nginx.conf +++ b/container/web-backend-nginx/nginx.conf @@ -8,7 +8,7 @@ http { default_type application/octet-stream; include /etc/nginx/conf.d/*.conf; server { - listen 11320 ssl; + listen 20003 ssl; server_name api.compass-ci.openeuler.org; ssl_certificate /opt/cert/web-backend.crt; ssl_certificate_key /opt/cert/web-backend.key; @@ -20,7 +20,7 @@ http { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forward-For $proxy_add_x_forwarded_for; # web-backend server - proxy_pass http://172.17.0.1:32767; + proxy_pass http://172.17.0.1:10002; } location ~ ^/(result|pub) { diff --git a/container/web-backend-nginx/start b/container/web-backend-nginx/start index 4da4c8b..5360385 100755 --- a/container/web-backend-nginx/start +++ b/container/web-backend-nginx/start @@ -10,7 +10,7 @@ docker_rm web-backend-nginx cmd=( docker run --name web-backend-nginx - -p 11320:11320 + -p 20003:20003 -d -v /etc/ssl/certs:/opt/cert -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/web-backend: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/start | 2 +- container/web-backend/web-backend | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/web-backend/start b/container/web-backend/start index 6731d27..e680255 100755 --- a/container/web-backend/start +++ b/container/web-backend/start @@ -18,7 +18,7 @@ cmd=( --name web-backend --restart=always -d - -p 32767:32767 + -p 10002:10002 -e ES_PORT=9200 -e SRV_HTTP_RESULT_HOST=$SRV_HTTP_RESULT_HOST diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 74eda85..33a3fae 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -9,7 +9,7 @@ CCI_SRC = ENV['CCI_SRC'] || '/c/compass-ci' require "#{CCI_SRC}/src/lib/web_backend.rb" set :bind, '0.0.0.0' -set :port, 32767 +set :port, 10002 set :show_exceptions, false set :environment, :production -- 2.23.0

1 0

[PATCH lkp-tests] lib/email.sh: change default send email port
by Wei Jihui 16 Mar '21

16 Mar '21

49000 --> 10001 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/email.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/email.sh b/lib/email.sh index a0c99c24..63d5a51e 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -87,5 +87,5 @@ send_email() ${subject}_content - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$email_content" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-10001}/send_mail_text" -d "$email_content" } -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 16 Mar '21

16 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 81 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..bf75b19b7 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,81 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] || { + echo "Ansible failed to install." + exit 1 + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git + if [ $? -eq 0 ]; then + echo "$benchmark is successfully cloned" + else + echo "Please check whether your ansible_repo is correct!" + exit 1 + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook is running failed!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

2 3

[PATCH lkp-tests] stat/ansible_test: stats result for ansible_test
by Wang Chenglong 16 Mar '21

16 Mar '21

The result of ansible_test are counted and recorded in the ES. query it with the 'es-jobs' command. sum.stats.ansible_test.error.Unsupported-pkg-mgr-This-role-is-depends-on-apt-yum-or-pkgng: 1 raw.stats.ansible_test.error.Unsupported-pkg-mgr-This-role-is-depends-on-apt-yum-or-pkgng.message: ["{\"changed\": false, \"msg\": \"Unsupported pkg mgr. This role is depends on apt, yum, or pkgng.\"}"] Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 55 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 44 insertions(+), 11 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..38d6e278c 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,48 @@ #!/usr/bin/env ruby -while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line +require 'json' + +def output_error(error_msg) + return if error_msg.nil? + + if error_msg.is_a? Array + error_msg.each do |i| + error_id = common_error_id i + puts "error.#{error_id}: 1" + puts "error.#{error_id}.message: #{@ansible_failed_info}" + end + else + error_id = common_error_id error_msg + puts "error.#{error_id}: 1" + puts "error.#{error_id}.message: #{@ansible_failed_info}" end end + +def common_error_id(line) + line.gsub!(/\n|\t/, '.') + line =`echo "#{line}"|awk -F'.' '{print $1}'` + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/\^/, '') + line.gsub!(/\.|,/, '') + line.gsub!(/<|>/, '') # error.Request-failed-<urlopen-error-timed-out>: 1 + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/ /, '-') + line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one + line = line.chomp + line +end + +while (line = STDIN.gets) + next unless line =~ /(FAILED!|failed:).*=>(.*)/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.parse @ansible_failed_info + + output_error ansible_failed_json['msg'] + output_error ansible_failed_json['message'] + output_error ansible_failed_json['cmd'] + output_error ansible_failed_json['failures'] +end -- 2.23.0

2 2

[PATCH compass-ci] dev-env: update port
by Luan Shengde 16 Mar '21

16 Mar '21

send-internet-mail has been assigned new port: 49000 -> 10001 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index b740a15..e448dcb 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -10,7 +10,7 @@ server_name=$(hostname | cut -f1 -d.) : ${ES_PORT:=9200} : ${OS_HTTP_HOST:=$server_ip} : ${SEND_MAIL_HOST:=$server_ip} -: ${SEND_MAIL_PORT:=49000} +: ${SEND_MAIL_PORT:=10001} : ${SRV_HTTP_RESULT_HOST:=$server_ip} : ${SRV_HTTP_OS_HOST:=$server_ip} : ${SRV_HTTP_GIT_HOST:=$server_ip} -- 2.23.0

1 0

[PATCH v2 compass-ci] src/lib/job.cr: change rpm_repo to upstream_repo
by Li Ping 16 Mar '21

16 Mar '21

src/lib/job.cr: open the path for uploading rpm packages. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9e18a4c..40aaa62 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -278,7 +278,7 @@ class Job mount_type = "nfs" if mount_type == "cifs" common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") + common_dir = "#{tmp_os}-#{tmp_os_version}" if @hash.has_key?("rpmbuild-pkg") return common_dir end @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 1

[PATCH v2 compass-ci] src/lib/job.cr: open the path for uploading rpm packages.
by Li Ping 16 Mar '21

16 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9e18a4c..40aaa62 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -278,7 +278,7 @@ class Job mount_type = "nfs" if mount_type == "cifs" common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") + common_dir = "#{tmp_os}-#{tmp_os_version}" if @hash.has_key?("rpmbuild-pkg") return common_dir end @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/compare-results.zh.md: translate to Chinese
by Lin Jiaxin 16 Mar '21

16 Mar '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- doc/manual/compare-results.zh.md | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/doc/manual/compare-results.zh.md b/doc/manual/compare-results.zh.md index 1b43d6b..b96add9 100644 --- a/doc/manual/compare-results.zh.md +++ b/doc/manual/compare-results.zh.md @@ -1,29 +1,26 @@ -# Introduction of compare feature - The compare feature used to analyze the results of different jobs, showing the performance waves and changes - under different influencing factors, for users to analyze performance influencing factor. +# 比较功能介绍 + 比较功能用于分析不同任务的结果，显示不同影响因素下的性能波动和变化，供用户分析性能影响因素。 ## url: https://compass-ci.openeuler.org/compare -## options directions -### filters: +## 选项说明 +### 过滤器: - suite: iperf, netperf, mysql, ... - OS: openeuler 20.03, centos 7.6, ... - os_arch: aarch64, x86 - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... - we can combine the above options arbitrarily to limit compare scope. - choose at least one option as filter. + 我们可以任意组合以上选项来限制比较范围，但至少选择一个选项作为过滤器。 -### dimension - Dimension can select: os, os_version, os_arch, suite, tbox_group. - Within filter, we will compare all different job result by dimension - and keep other test conditions are same. +### 维度 + 可选维度: os, os_version, os_arch, suite, tbox_group. + 在过滤器中，我们将按所选维度比较所有不同的任务结果，并保持其他测试条件一致。 -## example: - filter: suite = iperf - dimension: os_version +## 示例: + 过滤器: suite = iperf + 维度: os_version - result: - os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + 结果: + os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # 其他测试条件一致 20.09 20.03 metric -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/web-backend-nginx: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend-nginx/nginx.conf | 2 +- container/web-backend-nginx/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/web-backend-nginx/nginx.conf b/container/web-backend-nginx/nginx.conf index c9a4ec4..50ced4c 100644 --- a/container/web-backend-nginx/nginx.conf +++ b/container/web-backend-nginx/nginx.conf @@ -8,7 +8,7 @@ http { default_type application/octet-stream; include /etc/nginx/conf.d/*.conf; server { - listen 11320 ssl; + listen 20003 ssl; server_name api.compass-ci.openeuler.org; ssl_certificate /opt/cert/web-backend.crt; ssl_certificate_key /opt/cert/web-backend.key; diff --git a/container/web-backend-nginx/start b/container/web-backend-nginx/start index 4da4c8b..5360385 100755 --- a/container/web-backend-nginx/start +++ b/container/web-backend-nginx/start @@ -10,7 +10,7 @@ docker_rm web-backend-nginx cmd=( docker run --name web-backend-nginx - -p 11320:11320 + -p 20003:20003 -d -v /etc/ssl/certs:/opt/cert -- 2.23.0

1 1

[PATCH compass-ci 1/2] container/web-backend: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/start | 2 +- container/web-backend/web-backend | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/web-backend/start b/container/web-backend/start index 6731d27..e680255 100755 --- a/container/web-backend/start +++ b/container/web-backend/start @@ -18,7 +18,7 @@ cmd=( --name web-backend --restart=always -d - -p 32767:32767 + -p 10002:10002 -e ES_PORT=9200 -e SRV_HTTP_RESULT_HOST=$SRV_HTTP_RESULT_HOST diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 74eda85..33a3fae 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -9,7 +9,7 @@ CCI_SRC = ENV['CCI_SRC'] || '/c/compass-ci' require "#{CCI_SRC}/src/lib/web_backend.rb" set :bind, '0.0.0.0' -set :port, 32767 +set :port, 10002 set :show_exceptions, false set :environment, :production -- 2.23.0

1 1

[PATCH compass-ci 2/2] answerback-email: update tool with new port
by Luan Shengde 16 Mar '21

16 Mar '21

assign-account service has assigned new port: 29999 => 10000 send-internet-mail service has assigned new port: 49000 => 10001 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index f573f0f..f4b6ef3 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -28,9 +28,9 @@ names = Set.new %w[ defaults = relevant_defaults(names) JUMPER_HOST ||= defaults['JUMPER_HOST'] -JUMPER_PORT ||= defaults['JUMPER_PORT'] || 29999 +JUMPER_PORT ||= defaults['JUMPER_PORT'] || 10000 SEND_MAIL_HOST ||= defaults['SEND_MAIL_HOST'] || 'localhost' -SEND_MAIL_PORT ||= defaults['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_PORT ||= defaults['SEND_MAIL_PORT'] || 10001 LAB ||= defaults['lab'] TOOL_SERVER = 'z9' -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/assign-account: update for new port
by Luan Shengde 16 Mar '21

16 Mar '21

the assign-account servive has been assigned new port: 29999 => 10000 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 2 +- container/assign-account/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index c5d77db..abbab0f 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -10,7 +10,7 @@ require 'yaml' require_relative 'get_account_info' set :bind, '0.0.0.0' -set :port, 29999 +set :port, 10000 post '/assign_account' do begin diff --git a/container/assign-account/start b/container/assign-account/start index 793191a..b926853 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -14,7 +14,7 @@ cmd=( -v /etc:/etc:rw -v /srv/account-info:/opt/account_data/:rw -v /home:/home:rw - -p 29999:29999 + -p 10000:10000 assign-account ) -- 2.23.0

1 0

[PATCH compass-ci] constants: change service default ports
by Cao Xueliang 16 Mar '21

16 Mar '21

Reference: 6354ba631e0fd5f795a1dc28b65e2002bd3de668 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/constants.rb | 6 +++--- src/scheduler/constants.cr | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index 8d8e488..6c30aaf 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -14,9 +14,9 @@ SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_RESULT_HOST ||= config['SRV_HTTP_RESULT_HOST'] || ENV['SRV_HTTP_RESULT_HOST'] || '172.17.0.1' SRV_HTTP_OS_HOST ||= config['SRV_HTTP_OS_HOST'] || ENV['SRV_HTTP_OS_HOST'] || '172.17.0.1' SRV_HTTP_GIT_HOST ||= config['SRV_HTTP_GIT_HOST'] || ENV['SRV_HTTP_GIT_HOST'] || '172.17.0.1' -SRV_HTTP_RESULT_PORT ||= config['SRV_HTTP_RESULT_PORT'] || ENV['SRV_HTTP_RESULT_PORT'] || 11300 -SRV_HTTP_OS_PORT ||= config['SRV_HTTP_OS_PORT'] || ENV['SRV_HTTP_OS_PORT'] || 11304 -SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || 11305 +SRV_HTTP_RESULT_PORT ||= config['SRV_HTTP_RESULT_PORT'] || ENV['SRV_HTTP_RESULT_PORT'] || 20007 +SRV_HTTP_OS_PORT ||= config['SRV_HTTP_OS_PORT'] || ENV['SRV_HTTP_OS_PORT'] || 20009 +SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || 20010 SRV_HTTP_DOMAIN ||= config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index a5e52b8..d18c88d 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -26,7 +26,7 @@ OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1 OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 SRV_HTTP_CCI_HOST = (ENV.has_key?("SRV_HTTP_CCI_HOST") ? ENV["SRV_HTTP_CCI_HOST"] : "172.17.0.1") -SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11307") +SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : 20011).to_i32 CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") LAB_REPO = "lab-z9" -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: change service ports
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/start-cci | 2 +- container/srv-http/start-git | 2 +- container/srv-http/start-initrd | 2 +- container/srv-http/start-os | 2 +- container/srv-http/start-pub | 2 +- container/srv-http/start-result | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci index 8486476..00d8e4c 100755 --- a/container/srv-http/start-cci +++ b/container/srv-http/start-cci @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-cci - -p 11307:11300 + -p 20011:11300 -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-git b/container/srv-http/start-git index f2292bf..c1181a7 100755 --- a/container/srv-http/start-git +++ b/container/srv-http/start-git @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-git - -p 11305:11300 + -p 20010:11300 -v /srv/git/archlinux:/srv/git/archlinux:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 495cc25..466ec1b 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-initrd - -p 11303:11300 + -p 20008:11300 -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro diff --git a/container/srv-http/start-os b/container/srv-http/start-os index bb344cf..d39e04e 100755 --- a/container/srv-http/start-os +++ b/container/srv-http/start-os @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-os - -p 11304:11300 + -p 20009:11300 -v /srv/os:/srv/os:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-pub b/container/srv-http/start-pub index dc1864c..f42b117 100755 --- a/container/srv-http/start-pub +++ b/container/srv-http/start-pub @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-pub - -p 11302:11300 + -p 20006:11300 -v /srv/pub:/srv/pub:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-result b/container/srv-http/start-result index f9ecc97..bf0287d 100755 --- a/container/srv-http/start-result +++ b/container/srv-http/start-result @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-result - -p 11300:11300 + -p 20007:11300 -v /srv/result:/srv/result:ro -v /etc/localtime:/etc/localtime:ro -d -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: open the path for uploading rpm packages.
by Li Ping 16 Mar '21

16 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index f8cfa97..40aaa62 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -278,7 +278,7 @@ class Job mount_type = "nfs" if mount_type == "cifs" common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") + common_dir = "#{tmp_os}-#{tmp_os_version}" if @hash.has_key?("rpmbuild-pkg") return common_dir end @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["upstream_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 16 Mar '21

16 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 83 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..9ecb4d2c5 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,83 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + echo $ansible_host + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] || { + echo "Ansible failed to install." + exit 1 + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + exit 1 + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook is running failed!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

1 1

[PATCH v4 compass-ci 2/2] multi-docker: add reboot function
by Wu Zhende 16 Mar '21

16 Mar '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 34 +++++++++++++++++++++++++++++++++- providers/docker/run.sh | 2 ++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index f4e954e..9d6854f 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -7,6 +7,8 @@ require 'open-uri' require 'json' require 'set' require 'fileutils' + +require '../lib/mq_client' require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' @@ -19,6 +21,8 @@ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 LOG_DIR = '/srv/cci/serial/logs' +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || 'localhost' +MQ_PORT = ENV['MQ_PORT'] || 5672 def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -84,7 +88,8 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, + { 'job_id' => hash['job_id'], + 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path, 'log_dir' => "#{LOG_DIR}/#{hostname}" @@ -145,6 +150,29 @@ def loop_main(hostname, queues) end end +def loop_reboot_docker(hostname) + loop do + begin + reboot_docker(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_docker(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) + queue = mq.queue(hostname, {:durable => true}) + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + puts msg + machine_info = JSON.parse(msg) + job_id = machine_info['job_id'] + system "docker rm -f #{job_id};echo $?" + mq.ack(info) + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -153,6 +181,10 @@ def save_pid(pids) end def multi_docker(hostname, nr_container, queues) + Process.fork do + loop_reboot_docker(hostname) + end + pids = [] nr_container.to_i.times do |i| pid = Process.fork do diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 2dfe36d..80950dd 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -4,6 +4,7 @@ . $LKP_SRC/lib/yaml.sh +: ${job_id:=$$} : ${docker_image:="centos:7"} : ${load_path:="${HOME}/jobs"} : ${hostname:="dc-1g-1"} @@ -23,6 +24,7 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm + --name ${job_id} --hostname $host.compass-ci.net -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 16 Mar '21

16 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH v4 compass-ci] providers/multi-qemu: optimize
by Wu Zhende 16 Mar '21

16 Mar '21

1. optimize function name 2. the host and port are transferred when the MQClient is initalized 3. reboot vm cyclically Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 5979cc9..d3f8406 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -104,8 +104,19 @@ def save_pid(arr) f.close end -def reboot_deadline_machine(hostname) - mq = MQClient.new +def loop_reboot_vm(hostname) + loop do + begin + reboot_vm(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_vm(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) queue = mq.queue(hostname, {:durable => true}) queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| puts msg @@ -118,10 +129,7 @@ end def multiqemu Process.fork do - reboot_deadline_machine(HOSTNAME) - rescue StandardError => e - puts e.backtrace - sleep 5 + loop_reboot_vm(HOSTNAME) end pids = [] -- 2.23.0

1 0

[PATCH compass-ci] container/webhook: change the service port
by Li Yuanchao 15 Mar '21

15 Mar '21

For the port policy. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/webhook/start b/container/webhook/start index 373396a..1d121a4 100755 --- a/container/webhook/start +++ b/container/webhook/start @@ -12,7 +12,7 @@ cmd = %w[ --restart=always --name webhook -d - -p 11301:11301 + -p 20005:20005 node:webhook ] -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: change the local git repo path
by Lin Jiaxin 15 Mar '21

15 Mar '21

As the path of repo mirrors under /srv/git changes, introduce upstream_dir to get new path. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 43932865..350d8fdf 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -6,6 +6,7 @@ # - pkg_ver # - config # upstream_url +# upstream_dir # We'll use the below global env vars, but not treat them as parameters to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ @@ -123,9 +124,9 @@ replace_source() do echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { - url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" + url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#commit=${upstream_commit}" [ -n "$git_tag" ] && { - url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#tag=${git_tag}" + url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#tag=${git_tag}" } } } -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/auto_submit: change the upstream_repo path to submit
by Lin Jiaxin 15 Mar '21

15 Mar '21

Reference: 5d6cea1bd531409f0444abf14088fdecd8b08c2d Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index faebb2f..a27075b 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -33,7 +33,7 @@ class AutoSubmit git_repo = newrefs_info['git_repo'] puts "git_repo : #{git_repo}" url = Array(newrefs_info['url'])[0] - submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{git_repo} upstream_url=#{url}"] + submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{newrefs_info['belong']}/#{git_repo} upstream_url=#{url}"] if newrefs_info['pkgbuild_repo'] pkgbuild_repo = get_pkgbuild_repo(newrefs_info['pkgbuild_repo']) -- 2.23.0

2 2

[PATCH compass-ci] optimize doc
by Luan Shengde 15 Mar '21

15 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-tests-container.zh.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/build-lkp-tests-container.zh.md b/doc/manual/build-lkp-tests-container.zh.md index 10a1611..407ddbf 100644 --- a/doc/manual/build-lkp-tests-container.zh.md +++ b/doc/manual/build-lkp-tests-container.zh.md @@ -60,4 +60,4 @@ submit命令： - 了解更多submit命令的使用方法，请参考：[submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… + 了解更多submit命令的使用方法，请参考：[submit用户手册](https://gitee.com/wu_fengguang/compass-c… -- 2.23.0

1 0

[PATCH compass-ci 2/2] service/scheduler: fix the mq connection exception
by Wu Zhende 15 Mar '21

15 Mar '21

[Why] If the connection to the mq service is disconnected, the interface for sending messages is abnormal, affecting the main logic. [How] Start a thread to send mq messages. Reconnect and send every 5 seconds, three attempts. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 10 ++++++++++ src/scheduler/close_job.cr | 2 +- src/scheduler/find_job_boot.cr | 2 +- src/scheduler/update_job_parameter.cr | 2 +- 4 files changed, 13 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f5aba03..9528f54 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -218,4 +218,14 @@ class Sched return nil end + + def mq_pushlish_confirm(queue, msg) + 3.times do + @mq.pushlish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end end diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index ea1dbf5..e014361 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -45,7 +45,7 @@ class Sched "job_state" => "close", "time" => get_time } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..9e83537 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -32,7 +32,7 @@ class Sched "time" => get_time, "job_state" => "boot" } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end # auto submit a job to collect the host information. diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index b43aa06..d81f28b 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -44,6 +44,6 @@ class Sched "job_state" => (@env.get?("job_state") || "update").to_s, "time" => @env.get?("time").to_s } - @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) + spawn mq_pushlish_confirm(JOB_MQ, mq_msg.to_json) end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/mq.cr: add new function reconnect
by Wu Zhende 15 Mar '21

15 Mar '21

When the connection to the mq is disconnected, an interface needs to be provided to re-initiate the connection. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/mq.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/src/lib/mq.cr b/src/lib/mq.cr index 87c67e9..980585d 100644 --- a/src/lib/mq.cr +++ b/src/lib/mq.cr @@ -5,17 +5,34 @@ require "singleton" require "amqp-client" require "amq-protocol" +require "./json_logger" + class MQClient getter ch : AMQP::Client::Channel MQ_HOST = ENV.has_key?("MQ_HOST") ? ENV["MQ_HOST"] : "172.17.0.1" MQ_PORT = (ENV.has_key?("MQ_PORT") ? ENV["MQ_PORT"] : 5672).to_i32 def initialize(host = MQ_HOST, port = MQ_PORT) + @log = JSONLogger.new @client = AMQP::Client.new("amqp://#{host}:#{port}") conn = @client.connect @ch = conn.channel.as(AMQP::Client::Channel) end + def reconnect + conn = @client.connect + @ch = conn.channel.as(AMQP::Client::Channel) + @log.info({ + "msg" => "rabbitmq reconnected successfully", + "source" => "mq_client" + }.to_json) + rescue e + @log.warn({ + "msg" => e.to_s, + "source" => "mq_client" + }.to_json) + end + def self.instance Singleton::Of(self).instance end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 15 Mar '21

15 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 83 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..43a9ce341 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,83 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + echo $ansible_host + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] || { + echo "Ansible failed to install." + exit 1 + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + exit 1 + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook failed to run!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

2 2

[PATCH lkp-tests 2/2] distro/aliyun: revoke commit e21801797 and use a new approach
by Wei Jihui 15 Mar '21

15 Mar '21

commit:e21801797 is for use dnf instead of yum in centos7 use dnf it can raise error when yum install print "No package * available" [why] install dnf will installs many dependency packages. this makes cci-depends can not download some packages. [how] yum install do not use dnf and analyze installation output, if has "No package * available" then exit 1 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/aliyun | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 253121e0..24f16287 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -13,16 +13,6 @@ support_repo_epel() fi } -update_yum_into_dnf() -{ - if ! rpm -q dnf >/dev/null; then - yum install -y -q dnf - ln -sf /usr/bin/dnf /usr/bin/yum - else - dnf makecache timer >/dev/null - fi -} - distro_install_depends() { local script @@ -30,7 +20,6 @@ distro_install_depends() local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel - [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf for script in $scripts do @@ -71,10 +60,10 @@ pack_benchmark() yum_download() { [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel - [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -y -q --downloadonly $*" - yum install -y -q --downloadonly "$@" + echo "yum install -y --downloadonly $@ | grep -e \"No\ package .*\ available\"" + yum install -y --downloadonly "$@" | grep -e "No\ package .*\ available" && exit 1 + return 0 } yum_pack() -- 2.23.0

1 0

[PATCH lkp-tests 1/2] distro/aliyun: fix yum install args
by Wei Jihui 15 Mar '21

15 Mar '21

[error] Ignored option -q, -v, -d or -e (probably due to merging: -yq != -y -q) Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/aliyun | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 83f058c3..253121e0 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -9,14 +9,14 @@ shopt -s nullglob support_repo_epel() { if ! rpm -q epel-release >/dev/null; then - yum install -yq epel-release + yum install -y -q epel-release fi } update_yum_into_dnf() { if ! rpm -q dnf >/dev/null; then - yum install -yq dnf + yum install -y -q dnf ln -sf /usr/bin/dnf /usr/bin/yum else dnf makecache timer >/dev/null @@ -46,7 +46,7 @@ distro_install_depends() for pkg in $packages do - rpm --quiet --query $pkg || yum install -yq $pkg + rpm --quiet --query $pkg || yum install -y -q $pkg done done } @@ -73,8 +73,8 @@ yum_download() [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -yq --downloadonly $*" - yum install -yq --downloadonly "$@" + echo "yum install -y -q --downloadonly $*" + yum install -y -q --downloadonly "$@" } yum_pack() @@ -137,9 +137,9 @@ install_gcc7() wget -O /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm https://rpmfind.net/linux/centos/7/extras/x86_64/Packages/centos-release-sc… rpm -ivh --ignoresize /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm else - yum install -yq centos-release-scl-rh + yum install -y -q centos-release-scl-rh fi - yum install -yq "devtoolset-7-gcc*" + yum install -y -q "devtoolset-7-gcc*" source /opt/rh/devtoolset-7/enable } @@ -148,7 +148,7 @@ download_ocfs2_tools() local pkgs=$1 mkdir -p /opt/rpms - yum install -yq wget + yum install -y -q wget wget -nc -P /opt/rpms https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/ocfs2-… @@ -160,7 +160,7 @@ download_ocfs2_tools() fi # Try to download the dependences of ocfs2-tools/ocfs2-tools-devel - yum install -yq --downloadonly "$download_pkgs" + yum install -y -q --downloadonly "$download_pkgs" } install_ocfs2_tools() @@ -169,8 +169,8 @@ install_ocfs2_tools() download_ocfs2_tools "$pkgs" - yum install -yq /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm + yum install -y -q /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm echo "$pkgs" | grep -q 'ocfs2-tools-devel' && \ - yum install -yq /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm + yum install -y -q /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm } -- 2.23.0

1 0

[PATCH compass-ci] fix: port conflict between srv-http-cci and scheduler_https
by Yu Chuan 15 Mar '21

15 Mar '21

[Why] container/scheduler_https already use 11306 port of host, so change the port of container/srv-http-cci to 11307. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/scheduler/start | 2 +- container/srv-http/start-cci | 2 +- sparrow/3-code/dev-env | 2 +- src/scheduler/constants.cr | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index d83fcc21fb0b..e655dd4a94b2 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -37,7 +37,7 @@ defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' -defaults['SRV_HTTP_CCI_PORT'] ||= '11306' +defaults['SRV_HTTP_CCI_PORT'] ||= '11307' env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci index 78ab1490795b..84864761e689 100755 --- a/container/srv-http/start-cci +++ b/container/srv-http/start-cci @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-cci - -p 11306:11300 + -p 11307:11300 -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 403ccb127813..93a1667e73b1 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,7 +18,7 @@ server_name=$(hostname | cut -f1 -d.) : ${SRV_HTTP_RESULT_PORT:=11300} : ${SRV_HTTP_OS_PORT:=11304} : ${SRV_HTTP_GIT_PORT:=11305} -: ${SRV_HTTP_CCI_PORT:=11306} +: ${SRV_HTTP_CCI_PORT:=11307} : ${LOGGING_ES_HOST:=$server_ip} : ${LOGGING_ES_PORT:=9202} : ${INITRD_HTTP_HOST:=$server_ip} diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 3fdb9839e90d..a5e52b8f7a6d 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -26,7 +26,7 @@ OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1 OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 SRV_HTTP_CCI_HOST = (ENV.has_key?("SRV_HTTP_CCI_HOST") ? ENV["SRV_HTTP_CCI_HOST"] : "172.17.0.1") -SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11306") +SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11307") CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") LAB_REPO = "lab-z9" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package/common: install bunny for mq
by Wu Zhende 15 Mar '21

15 Mar '21

multi-qemu and multi-docker need bunny to connect rabbitmq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 0324f58..8942247 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer -- 2.23.0

1 0

[PATCH v3 compass-ci] providers/multi-qemu: optimize
by Wu Zhende 15 Mar '21

15 Mar '21

1. optimize function name 2. the host and port are transferred when the MQClient is initalized 3. reboot vm cyclically Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 5979cc9..d3f8406 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -104,8 +104,19 @@ def save_pid(arr) f.close end -def reboot_deadline_machine(hostname) - mq = MQClient.new +def loop_reboot_vm(hostname) + loop do + begin + reboot_vm(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_vm(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) queue = mq.queue(hostname, {:durable => true}) queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| puts msg @@ -118,10 +129,7 @@ end def multiqemu Process.fork do - reboot_deadline_machine(HOSTNAME) - rescue StandardError => e - puts e.backtrace - sleep 5 + loop_reboot_vm(HOSTNAME) end pids = [] -- 2.23.0

2 1

[PATCH v3 lkp-tests 1/2] tests/rpmbuild-pkg: change rpmbuild-pkg upload_dir
by Li Ping 15 Mar '21

15 Mar '21

--- tests/rpmbuild-pkg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index b641e97d1..a8c8eb6b1 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,7 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" +sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() { @@ -24,7 +24,7 @@ download_rpm_repo() { local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in "${filelist[@]}" + for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" -- 2.23.0

2 4

[PATCH v2 compass-ci] providers/multi-qemu: optimize
by Wu Zhende 15 Mar '21

15 Mar '21

1.optimize function name 2.the host and port are transferred when the MQClient is initalized 3.reboot vm cyclically Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 5979cc9..d3f8406 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -104,8 +104,19 @@ def save_pid(arr) f.close end -def reboot_deadline_machine(hostname) - mq = MQClient.new +def loop_reboot_vm(hostname) + loop do + begin + reboot_vm(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_vm(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) queue = mq.queue(hostname, {:durable => true}) queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| puts msg @@ -118,10 +129,7 @@ end def multiqemu Process.fork do - reboot_deadline_machine(HOSTNAME) - rescue StandardError => e - puts e.backtrace - sleep 5 + loop_reboot_vm(HOSTNAME) end pids = [] -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] tests/rpmbuild-pkg: adjusting code for auto_submit
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 1 - tests/rpmbuild-pkg | 23 +++++++++-------------- 2 files changed, 9 insertions(+), 15 deletions(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index 53e145b3b..5860ba029 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,4 @@ suite: rpmbuild-pkg category: functional -rpm_repo: rpmbuild-pkg: diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index a8c8eb6b1..e51c75aa2 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,11 +1,11 @@ #!/bin/bash -# - rpm_repo +# - upstream_repo . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh -[ -n "$rpm_repo" ] || die "rpm_repo is empty" -package_name=${rpm_repo##*/} +[ -n "$upstream_repo" ] || die "upstream_repo is empty" +package_name=${upstream_repo##*/} sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() @@ -14,21 +14,16 @@ init_workspace() rpmdev-setuptree } -get_pkgfile() +download_upstream_repo() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" -} - -download_rpm_repo() -{ - local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') + git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" + cd $package_name || exit + filelist=$(git ls-files) for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" - get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + mv "$pkgfile" "${HOME}/rpmbuild/${dir}/" done } @@ -63,6 +58,6 @@ upload_rpm_pkg() } init_workspace -download_rpm_repo +download_upstream_repo build_rpm upload_rpm_pkg -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] tests/rpmbuild-pkg: adjusting code for auto_submit
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/rpmbuild-pkg.yaml | 2 +- tests/rpmbuild-pkg | 23 +++++++++-------------- 2 files changed, 10 insertions(+), 15 deletions(-) diff --git a/jobs/rpmbuild-pkg.yaml b/jobs/rpmbuild-pkg.yaml index 53e145b3b..a70590afe 100644 --- a/jobs/rpmbuild-pkg.yaml +++ b/jobs/rpmbuild-pkg.yaml @@ -1,5 +1,5 @@ suite: rpmbuild-pkg category: functional -rpm_repo: +upstream_repo: rpmbuild-pkg: diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index a8c8eb6b1..d49dc43c7 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,11 +1,11 @@ #!/bin/bash -# - rpm_repo +# - upstream_repo . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh -[ -n "$rpm_repo" ] || die "rpm_repo is empty" -package_name=${rpm_repo##*/} +[ -n "$upstream_repo" ] || die "upstream_repo is empty" +package_name=${upstream_repo##*/} sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() @@ -14,21 +14,16 @@ init_workspace() rpmdev-setuptree } -get_pkgfile() +download_upstream_repo() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" -} - -download_rpm_repo() -{ - local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') + git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" + cd $package_name || exit + filelist=$(git ls-files) for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" - get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + cat "$pkgfile" > "${HOME}/rpmbuild/${dir}/$pkgfile" done } @@ -63,6 +58,6 @@ upload_rpm_pkg() } init_workspace -download_rpm_repo +download_upstream_repo build_rpm upload_rpm_pkg -- 2.23.0

2 2

[PATCH v2 compass-ci] result-webdav-webdav/nginx.conf: do not overwrite the uploaded config with the same name
by Lin Jiaxin 12 Mar '21

12 Mar '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- container/result-webdav/nginx.conf | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index cc21926..e0e0ee2 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -27,6 +27,22 @@ http { dav_methods PUT MKCOL; dav_access user:rw group:rw all:rw; + set $config_upload_condition "false"; + if ( $uri ~ "^/cci/build-config.*" ) + { + set $config_upload_condition "true"; + } + + if ( -e $request_filename ) + { + set $config_upload_condition "${config_upload_condition}+true"; + } + + if ( $config_upload_condition = "true+true" ) + { + return 200; + } + access_by_lua_block { function split(str, reps) local resultstrlist = {} -- 2.23.0

1 0

[PATCH compass-ci 2/2] ult-webdav/nginx.conf: do not overwrite the uploaded config with the same name
by Lin Jiaxin 12 Mar '21

12 Mar '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- container/result-webdav/nginx.conf | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index cc21926..e0e0ee2 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -27,6 +27,22 @@ http { dav_methods PUT MKCOL; dav_access user:rw group:rw all:rw; + set $config_upload_condition "false"; + if ( $uri ~ "^/cci/build-config.*" ) + { + set $config_upload_condition "true"; + } + + if ( -e $request_filename ) + { + set $config_upload_condition "${config_upload_condition}+true"; + } + + if ( $config_upload_condition = "true+true" ) + { + return 200; + } + access_by_lua_block { function split(str, reps) local resultstrlist = {} -- 2.23.0

1 1

[PATCH compass-ci] src/lib/job.cr: change rpm_repo to upstream_repo
by Li Ping 12 Mar '21

12 Mar '21

--- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9e18a4c..f8cfa97 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["upstream_repo"].to_s.split("/")[-1]}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] tests/rpmbuild-pkg: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

--- tests/rpmbuild-pkg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index b641e97d1..a8c8eb6b1 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,7 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" +sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() { @@ -24,7 +24,7 @@ download_rpm_repo() { local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in "${filelist[@]}" + for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" -- 2.23.0

1 0

[PATCH compass-ci] providers/multi-qemu: optimize
by Wu Zhende 12 Mar '21

12 Mar '21

1.optimize function name 2.the host and port are transferred when the MQClient is initalized 3.reboot vm cyclically Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 5979cc9..d3f8406 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -104,8 +104,19 @@ def save_pid(arr) f.close end -def reboot_deadline_machine(hostname) - mq = MQClient.new +def loop_reboot_vm(hostname) + loop do + begin + reboot_vm(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_vm(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) queue = mq.queue(hostname, {:durable => true}) queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| puts msg @@ -118,10 +129,7 @@ end def multiqemu Process.fork do - reboot_deadline_machine(HOSTNAME) - rescue StandardError => e - puts e.backtrace - sleep 5 + loop_reboot_vm(HOSTNAME) end pids = [] -- 2.23.0

2 1

[PATCH lkp-tests] tests/rpmbuild-pkg: adjusting code for auto_submit
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- tests/rpmbuild-pkg | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index a8c8eb6b1..f49ba3a0b 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,11 +1,11 @@ #!/bin/bash -# - rpm_repo +# - upstream_repo . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh -[ -n "$rpm_repo" ] || die "rpm_repo is empty" -package_name=${rpm_repo##*/} +[ -n "$upstream_repo" ] || die "upstream_repo is empty" +package_name=${upstream_repo##*/} sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() @@ -14,21 +14,16 @@ init_workspace() rpmdev-setuptree } -get_pkgfile() +download_upstream_repo() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" -} - -download_rpm_repo() -{ - local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') + git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone ${package_name} git repo failed: git://$GIT_SERVER/openeuler/${upstream_repo}" + cd $package_name || exit + filelist=$(git ls-files) for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" - get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + cat "$pkgfile" > "${HOME}/rpmbuild/${dir}/$pkgfile" done } @@ -63,6 +58,6 @@ upload_rpm_pkg() } init_workspace -download_rpm_repo +download_upstream_repo build_rpm upload_rpm_pkg -- 2.23.0

2 2

[PATCH compass-ci] container/dracut-initrd: add basename into openeuler's 21xx+ kernel
by Xu Xijian 12 Mar '21

12 Mar '21

[why] It seems openeuler 21xx+ kernel not support cmd basename, our overlay need this cmd, so add basename into dracut module to enable initrd. [errlog] //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 50: basename: command not found //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 65: basename: command not found Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 45e895c..494588a 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -17,7 +17,7 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh +sed -i "/install() {/a\ inst /sbin/mke2fs /usr/bin/basename" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" [ "$(sed -n '$p' $pre_mount_file)" = "exit 0" ] && sed -i '$d' "$pre_mount_file" cat set-local-sysroot.sh >> "$pre_mount_file" -- 2.23.0

2 2

[PATCH v2 compass-ci] src/lib/job.cr: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2177b22..9e18a4c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -277,11 +277,8 @@ class Job # same usage for client mount_type = "nfs" if mount_type == "cifs" - if @hash["rpmbuild-pkg"]? - common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - else - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - end + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") return common_dir end -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2177b22..1fbf646 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -277,11 +277,8 @@ class Job # same usage for client mount_type = "nfs" if mount_type == "cifs" - if @hash["rpmbuild-pkg"]? - common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - else - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - end + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash["rpmbuild-pkg"]? return common_dir end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] multi-docker: add reboot function
by Wu Zhende 12 Mar '21

12 Mar '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 34 +++++++++++++++++++++++++++++++++- providers/docker/run.sh | 2 ++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index f4e954e..9d6854f 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -7,6 +7,8 @@ require 'open-uri' require 'json' require 'set' require 'fileutils' + +require '../lib/mq_client' require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' @@ -19,6 +21,8 @@ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 LOG_DIR = '/srv/cci/serial/logs' +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || 'localhost' +MQ_PORT = ENV['MQ_PORT'] || 5672 def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -84,7 +88,8 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, + { 'job_id' => hash['job_id'], + 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path, 'log_dir' => "#{LOG_DIR}/#{hostname}" @@ -145,6 +150,29 @@ def loop_main(hostname, queues) end end +def loop_reboot_docker(hostname) + loop do + begin + reboot_docker(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_docker(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) + queue = mq.queue(hostname, {:durable => true}) + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + puts msg + machine_info = JSON.parse(msg) + job_id = machine_info['job_id'] + system "docker rm -f #{job_id};echo $?" + mq.ack(info) + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -153,6 +181,10 @@ def save_pid(pids) end def multi_docker(hostname, nr_container, queues) + Process.fork do + loop_reboot_docker(hostname) + end + pids = [] nr_container.to_i.times do |i| pid = Process.fork do diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 2dfe36d..80950dd 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -4,6 +4,7 @@ . $LKP_SRC/lib/yaml.sh +: ${job_id:=$$} : ${docker_image:="centos:7"} : ${load_path:="${HOME}/jobs"} : ${hostname:="dc-1g-1"} @@ -23,6 +24,7 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm + --name ${job_id} --hostname $host.compass-ci.net -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH compass-ci] extract-stats: wait_for refresh doc to ES DB
by Lu Weitao 12 Mar '21

12 Mar '21

[Why] We need update doc(job) to ES completed, then send delimiter task to task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index dbb3fe2..44c327d 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -81,6 +81,7 @@ class StatsWorker { :index => "jobs", :type => "_doc", :id => job_id, + :refresh => "wait_for", :body => {:doc => update_content}, } ) -- 2.23.0

1 0

[PATCH compass-ci] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code/dev-env: fix failed to submit job
by Liu Yinsi 12 Mar '21

12 Mar '21

[error] root@taishan200-2280-2s64p-256g--a123 /c/compass-ci/container# submit host-info.yaml submit_id=9bca6036-4602-44ab-92df-729058ac1818 Traceback (most recent call last): 5: from /c/lkp-tests/sbin/submit:136:in `<main>' 4: from /c/lkp-tests/sbin/submit:136:in `each' 3: from /c/lkp-tests/sbin/submit:150:in `block in <main>' 2: from /c/lkp-tests/lib/job.rb:744:in `each_jobs' 1: from /c/lkp-tests/lib/job.rb:736:in `each_job' /c/lkp-tests/sbin/submit:151:in `block (2 levels) in <main>': Please configure SCHED_HOST (RuntimeError) [why] for user locally deploy compass-ci, env $SCHED_HOST not exists in /etc/compass-ci/defaults/$server_name.yaml, cause failed to submit, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 403ccb1..fdf7894 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -36,6 +36,8 @@ server_name=$(hostname | cut -f1 -d.) mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF +SCHED_HOST: $sched_host +SCHED_PORT: $sched_port lab: nolab EOF -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

--- tests/rpmbuild-pkg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index b641e97d1..a8c8eb6b1 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,7 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" +sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() { @@ -24,7 +24,7 @@ download_rpm_repo() { local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in "${filelist[@]}" + for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] doc/job/os_mount.md: change the fields of job.yaml
by Xu Xijian 11 Mar '21

11 Mar '21

The fields that users need to give is the suffix of partitions, so give them appropriate names. use_root_partition => src_lv_suffix save_root_partition => boot_lv_suffix Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/os_mount.md | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 6c1637b..eab95ed 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -56,23 +56,25 @@ The brief flow is as follows: ## persistent rootfs data -When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_custom_params` will help you: `save_root_partition`, `use_root_partition`. +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_custom_params` will help you: 'src_lv_suffix', 'boot_lv_suffix'. The brief flow is as follows: ``` 1. boot and request scheduler for job. 2. initrd stage: + use_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${src_lv_suffix} + save_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${boot_lv_suffix} - firstly, we need two logical volume: ${src_lv} -- src logical volume: - - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if have ${use_root_partition}, src_lv=${use_root_partition} - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} ${boot_lv} -- boot logical volume: - - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if have ${save_root_partition}, boot_lv=${save_root_partition} - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} - if ${src_lv} not exists: - - if have ${use_root_partition}, exit 1. - - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if have ${use_root_partition}, reboot. + - if no ${use_root_partition}, create ${src_lv}, and sync the rootfs from cluster nfs server. - if ${boot_lv} != ${src_lv}: - if ${boot_lv} exists: delete it - create ${boot_lv} as the snapshot of ${src_lv}. @@ -87,12 +89,12 @@ Demo usage: rootfs data of job-20210218.yaml so that it can be used by the subsequent jobs. Then you need add the follow field in your job-20210218.yaml: - kernel_custom_params: save_root_partition=zhangsan_local_for_iperf_20210218 + kernel_custom_params: boot_lv_suffix=zhangsan_local_for_iperf_20210218 - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of job-20210218.yaml. Then you need add the follow field in your job-20210219.yaml: - kernel_custom_params: use_root_partition=zhangsan_local_for_iperf_20210218 + kernel_custom_params: src_lv_suffix=zhangsan_local_for_iperf_20210218 ``` Notes: @@ -109,16 +111,19 @@ Notes: os_arch: aarch64 os_version: 20.03 os_mount: local - kernel_custom_params: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + kernel_custom_params: src_lv_suffix=zhangsan_local_for_iperf_20210218 boot_lv_suffix=zhangsan_local_for_iperf_20210219 ``` 2. scheduler return the custom_ipxe to testbox + os_info = "#{os}_#{os_arch}_#{os_version}" + use_root_partition = "/dev/mapper/os-#{os_info}_#{src_lv_suffix}" + save_root_partition = "/dev/mapper/os-#{os_info}_#{boot_lv_suffix}" ``` #!ipxe dhcp initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp - kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_custom_params} + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 initrd=initrd.lkp local use_root_partition=#{use_root_partition} save_root_partition=#{save_root_partition} rw ... boot ``` -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container/dracut-initrd: change someof the handle logic according to scheme doc
by Xu Xijian 11 Mar '21

11 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 20 +++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index dc9000b..240551a 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -5,9 +5,9 @@ analyse_kernel_cmdline_params() { # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') timestamp="$(basename $root)" - os="$(echo $root_src | awk -F '/|-' '{print $2}')" - os_arch="$(echo $root_src | awk -F '/|-' '{print $3}')" - os_version="$(echo $root_src | awk -F '/|-' '{print $4}')" + os="$(echo $rootfs_src | awk -F '/|-' '{print $2}')" + os_arch="$(echo $rootfs_src | awk -F '/|-' '{print $3}')" + os_version="$(echo $rootfs_src | awk -F '/|-' '{print $4}')" os_info="${os}_${os_arch}_${os_version}" export rootfs_src timestamp os_info } @@ -58,22 +58,22 @@ analyse_kernel_cmdline_params sed -i "s/^locking_type = .*/locking_type = 1/" /etc/lvm/lvm.conf use_root_partition="$(getarg use_root_partition=)" -if [ -n "$use_root_partition" ]; then - src_lv="$use_root_partition" # /dev/mapper/os-${os_info}_${timestamp} +if [ -z "$use_root_partition" ]; then + src_lv="/dev/mapper/os-${os_info}_$timestamp" sync_src_lv "$src_lv" else - src_lv="/dev/mapper/os-${os_info}_$timestamp" - [ -e "src_lv" ] || { + src_lv="$use_root_partition" + [ -e "$src_lv" ] || { echo "warn dracut: FATAL: no src_lv with local mount, reboot" reboot } fi save_root_partition="$(getarg save_root_partition=)" -if [ -n "$save_root_partition" ]; then - boot_lv="$save_root_partition" -else +if [ -z "$save_root_partition" ]; then boot_lv="/dev/mapper/os-${os_info}" +else + boot_lv="$save_root_partition" fi snapshot_boot_lv "$src_lv" "$boot_lv" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/dracut-initrd: add basename into openeuler's 21xx+ kernel
by Xu Xijian 11 Mar '21

11 Mar '21

[why] It seems openeuler 21xx+ kernel not support cmd basename, our overlay need this cmd, so add basename into dracut module to enable initrd. [errlog] //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 50: basename: command not found //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 65: basename: command not found Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 45e895c..494588a 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -17,7 +17,7 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh +sed -i "/install() {/a\ inst /sbin/mke2fs /usr/bin/basename" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" [ "$(sed -n '$p' $pre_mount_file)" = "exit 0" ] && sed -i '$d' "$pre_mount_file" cat set-local-sysroot.sh >> "$pre_mount_file" -- 2.23.0

1 0

[PATCH lkp-tests] add multi-qemu-x86.yaml
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/multi-qemu-x86.yaml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 jobs/multi-qemu-x86.yaml diff --git a/jobs/multi-qemu-x86.yaml b/jobs/multi-qemu-x86.yaml new file mode 100644 index 00000000..5539c1ad --- /dev/null +++ b/jobs/multi-qemu-x86.yaml @@ -0,0 +1,23 @@ +suite: multi-qemu +category: functional +simplify-ci: +runtime: 100d +multi-qemu-0: + nr_vm: 5 + tbox_group: vm-2p1g-x86 +multi-qemu-1: + nr_vm: 5 + tbox_group: vm-2p2g-x86 +multi-qemu-2: + nr_vm: 5 + tbox_group: vm-2p4g-x86 +multi-qemu-3: + nr_vm: 10 + tbox_group: vm-2p8g-x86 +multi-qemu-4: + nr_vm: 5 + tbox_group: vm-2p16g-x86 +multi-qemu-5: + nr_vm: 1 + tbox_group: vm-2p32g-x86 +sleep: -- 2.23.0

1 0

[PATCH v2 lab-z9 2/2] add x86 vm host file
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/vm-1p1g-x86 | 3 ++- hosts/vm-2p16g-x86 | 3 ++- hosts/vm-2p1g-x86 | 3 ++- hosts/vm-2p32g-x86 | 3 ++- hosts/vm-2p4g-x86 | 3 ++- hosts/vm-2p8g-x86 | 3 ++- 6 files changed, 12 insertions(+), 6 deletions(-) diff --git a/hosts/vm-1p1g-x86 b/hosts/vm-1p1g-x86 index 5c0e081..b278e80 100644 --- a/hosts/vm-1p1g-x86 +++ b/hosts/vm-1p1g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 1 memory: 1G diff --git a/hosts/vm-2p16g-x86 b/hosts/vm-2p16g-x86 index 8abda03..cd49afc 100644 --- a/hosts/vm-2p16g-x86 +++ b/hosts/vm-2p16g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 2 memory: 16G diff --git a/hosts/vm-2p1g-x86 b/hosts/vm-2p1g-x86 index 60658d3..4fffd2a 100644 --- a/hosts/vm-2p1g-x86 +++ b/hosts/vm-2p1g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 2 memory: 1G diff --git a/hosts/vm-2p32g-x86 b/hosts/vm-2p32g-x86 index 2d7bb13..c7cc583 100644 --- a/hosts/vm-2p32g-x86 +++ b/hosts/vm-2p32g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 2 memory: 32G diff --git a/hosts/vm-2p4g-x86 b/hosts/vm-2p4g-x86 index 638f803..41d0a3b 100644 --- a/hosts/vm-2p4g-x86 +++ b/hosts/vm-2p4g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 2 memory: 4G diff --git a/hosts/vm-2p8g-x86 b/hosts/vm-2p8g-x86 index caa6a88..8bc5cad 100644 --- a/hosts/vm-2p8g-x86 +++ b/hosts/vm-2p8g-x86 @@ -1,5 +1,6 @@ +arch: aarch64 provider: qemu -template: pxe +template: kvm nr_node: 1 nr_cpu: 2 memory: 8G -- 2.23.0

1 0

[PATCH v2 lab-z9 1/2] add x86 vm host file
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/vm-1p1g-x86 | 6 ++++++ hosts/vm-2p16g-x86 | 6 ++++++ hosts/vm-2p1g-x86 | 6 ++++++ hosts/vm-2p32g-x86 | 6 ++++++ hosts/vm-2p4g-x86 | 6 ++++++ hosts/vm-2p8g-x86 | 6 ++++++ 6 files changed, 36 insertions(+) create mode 100644 hosts/vm-1p1g-x86 create mode 100644 hosts/vm-2p16g-x86 create mode 100644 hosts/vm-2p1g-x86 create mode 100644 hosts/vm-2p32g-x86 create mode 100644 hosts/vm-2p4g-x86 create mode 100644 hosts/vm-2p8g-x86 diff --git a/hosts/vm-1p1g-x86 b/hosts/vm-1p1g-x86 new file mode 100644 index 0000000..5c0e081 --- /dev/null +++ b/hosts/vm-1p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 1 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p16g-x86 b/hosts/vm-2p16g-x86 new file mode 100644 index 0000000..8abda03 --- /dev/null +++ b/hosts/vm-2p16g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 16G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g-x86 b/hosts/vm-2p1g-x86 new file mode 100644 index 0000000..60658d3 --- /dev/null +++ b/hosts/vm-2p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p32g-x86 b/hosts/vm-2p32g-x86 new file mode 100644 index 0000000..2d7bb13 --- /dev/null +++ b/hosts/vm-2p32g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 32G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g-x86 b/hosts/vm-2p4g-x86 new file mode 100644 index 0000000..638f803 --- /dev/null +++ b/hosts/vm-2p4g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g-x86 b/hosts/vm-2p8g-x86 new file mode 100644 index 0000000..caa6a88 --- /dev/null +++ b/hosts/vm-2p8g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 8G +hdd_partitions: /dev/vda -- 2.23.0

1 0

[PATCH v1 lab-z9] add x86 vm host file
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/vm-1p1g-x86 | 6 ++++++ hosts/vm-2p16g-x86 | 6 ++++++ hosts/vm-2p1g-x86 | 6 ++++++ hosts/vm-2p32g-x86 | 6 ++++++ hosts/vm-2p4g-x86 | 6 ++++++ hosts/vm-2p8g-x86 | 6 ++++++ 6 files changed, 36 insertions(+) create mode 100644 hosts/vm-1p1g-x86 create mode 100644 hosts/vm-2p16g-x86 create mode 100644 hosts/vm-2p1g-x86 create mode 100644 hosts/vm-2p32g-x86 create mode 100644 hosts/vm-2p4g-x86 create mode 100644 hosts/vm-2p8g-x86 diff --git a/hosts/vm-1p1g-x86 b/hosts/vm-1p1g-x86 new file mode 100644 index 0000000..5c0e081 --- /dev/null +++ b/hosts/vm-1p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 1 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p16g-x86 b/hosts/vm-2p16g-x86 new file mode 100644 index 0000000..8abda03 --- /dev/null +++ b/hosts/vm-2p16g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 16G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g-x86 b/hosts/vm-2p1g-x86 new file mode 100644 index 0000000..60658d3 --- /dev/null +++ b/hosts/vm-2p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p32g-x86 b/hosts/vm-2p32g-x86 new file mode 100644 index 0000000..2d7bb13 --- /dev/null +++ b/hosts/vm-2p32g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 32G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g-x86 b/hosts/vm-2p4g-x86 new file mode 100644 index 0000000..638f803 --- /dev/null +++ b/hosts/vm-2p4g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g-x86 b/hosts/vm-2p8g-x86 new file mode 100644 index 0000000..caa6a88 --- /dev/null +++ b/hosts/vm-2p8g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 8G +hdd_partitions: /dev/vda -- 2.23.0

1 0

[PATCH lkp-tests 2/2] add x86 vm host file
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/vm-1p1g-x86 | 6 ++++++ hosts/vm-2p16g-x86 | 6 ++++++ hosts/vm-2p1g-x86 | 6 ++++++ hosts/vm-2p32g-x86 | 6 ++++++ hosts/vm-2p4g-x86 | 6 ++++++ hosts/vm-2p8g-x86 | 6 ++++++ 6 files changed, 36 insertions(+) create mode 100644 hosts/vm-1p1g-x86 create mode 100644 hosts/vm-2p16g-x86 create mode 100644 hosts/vm-2p1g-x86 create mode 100644 hosts/vm-2p32g-x86 create mode 100644 hosts/vm-2p4g-x86 create mode 100644 hosts/vm-2p8g-x86 diff --git a/hosts/vm-1p1g-x86 b/hosts/vm-1p1g-x86 new file mode 100644 index 00000000..5c0e081c --- /dev/null +++ b/hosts/vm-1p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 1 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p16g-x86 b/hosts/vm-2p16g-x86 new file mode 100644 index 00000000..8abda03f --- /dev/null +++ b/hosts/vm-2p16g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 16G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g-x86 b/hosts/vm-2p1g-x86 new file mode 100644 index 00000000..60658d38 --- /dev/null +++ b/hosts/vm-2p1g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p32g-x86 b/hosts/vm-2p32g-x86 new file mode 100644 index 00000000..2d7bb132 --- /dev/null +++ b/hosts/vm-2p32g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 32G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g-x86 b/hosts/vm-2p4g-x86 new file mode 100644 index 00000000..638f803f --- /dev/null +++ b/hosts/vm-2p4g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g-x86 b/hosts/vm-2p8g-x86 new file mode 100644 index 00000000..caa6a88f --- /dev/null +++ b/hosts/vm-2p8g-x86 @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 8G +hdd_partitions: /dev/vda -- 2.23.0

1 0

[PATCH lkp-tests 1/2] add a host group in lkp-tests/hosts
by Zhang Dewan 11 Mar '21

11 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g | 1 + 1 file changed, 1 insertion(+) create mode 100644 hosts/taishan200-2280-2s48p-256g diff --git a/hosts/taishan200-2280-2s48p-256g b/hosts/taishan200-2280-2s48p-256g new file mode 100644 index 00000000..6399b995 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g @@ -0,0 +1 @@ +arch: aarch64 -- 2.23.0

1 0

help star
by Wu Fengguang 11 Mar '21

11 Mar '21

Xueliang is our help star, thanks Xueliang! ~/todo/bin% ./help-star 19.6 Wu Fengguang 18.6 Cao Xueliang 15.9 Wei Jihui 15.2 Yu Chuan 10.8 Wu Zhende 8.8 Luan Shengde 7.8 Liu Yinsi 6.8 Wang Chenglong 6.7 Lin Jiaxin 5.6 Bai Jing 5.4 Wang Yong 5.3 Cui Yili 5.0 Xu Xijian 4.8 Li Yuanchao 4.6 Liu Shaofei 3.3 Lu Weitao 3.1 Zhang Dewan 1.8 Du Kaitian 0.9 Li Ping 0.0 Zhang Yale

1 0

review star
by Wu Fengguang 11 Mar '21

11 Mar '21

Jihui is our review star again, thanks Jihui! ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 30 17 26 24 Xu Xijian 80 18 27 22 Wei Jihui 87 5 14 19 Yu Chuan 51 4 5 9 Cao Xueliang 32 13 32 7 Luan Shengde 16 3 3 3 Wu Zhende 5 2 1 2 Cui Yili 15 5 7 2 Wu Fengguang 5 2 2 1 Lin Jiaxin 9 0 0 0 Zhang Dewan 11 3 8 0 Liu Shaofei 3 0 0 0 Wang Yong 21 0 0 0 Liu Yinsi 13 0 0 0 Li Yuanchao 13 0 0 0 Bai Jing 26 1 2 0 Lu Weitao 12 1 1 0 Wang Chenglong 28 0 0 0 Li Ping

1 0

git count
by Wu Fengguang 11 Mar '21

11 Mar '21

Yu Chuan is our code star, thanks Yu Chuan! ~/todo/bin% ./git-count 7 18 Yu Chuan 10 Xu Xijian 10 Wei Jihui 8 Wu Zhende 7 Liu Yinsi 5 Li Yuanchao 5 Luan Shengde 4 Bai Jing 4 Lu Weitao 4 Li Ping 3 Cao Xueliang 2 Liu Shaofei 2 Wang Yong 2 Lin Jiaxin 1 Zhang Dewan 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale 0 Cui Yili 0 Wang Chenglong

1 0

[PATCH lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 11 Mar '21

11 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 84 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..36164bcae --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,84 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + echo $ansible_host + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] && { + echo "Ansible is successfully deployed!" + } + } +} + +download_ansible_role() +{ + + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git || exit 1 + # git clone git://$GIT_SERVER/$ansible_repo.git || exit + # ansible-galaxy install $username.$rolename + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook failed to run!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

3 4

[PATCH v3 lkp-tests] add new type error
by Bai Jing 11 Mar '21

11 Mar '21

[input] Error: Unknown repo: 'powertools' [output] unknown-repo.powertools: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 +++++++- tests/openeuler_docker | 43 +++++++++++++++++++++++------------------- 2 files changed, 31 insertions(+), 20 deletions(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index f85c9ca63..4c65c6bb3 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -6,12 +6,18 @@ sys_set = Set[] while (line = STDIN.gets) case line.chomp! - # Error: Unable to find a match: docker-registry + # Error: Unable to find a match: docker-registry mock xx when /Error: Unable to find a match: (.+)/ $1.split.each do |repo| repo_set << repo end + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + puts "unknown-repo.#{repo}: 1" + puts "unknown-repo.#{repo}.message: #{line}" + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" diff --git a/tests/openeuler_docker b/tests/openeuler_docker index b77b890a9..5270bde6e 100755 --- a/tests/openeuler_docker +++ b/tests/openeuler_docker @@ -13,7 +13,7 @@ cd $BENCHMARK_ROOT -centos_one_ok=false +nr_openeuler_fail=0 fit_repo() { @@ -21,7 +21,7 @@ fit_repo() git clone "$dockerfile_url" || exit 0 file_dir=$(find ./ -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]') [ -n "$file_dir" ] || { - echo "current-repo not support centos" + echo "current-repo-not-support-centos: 1" exit 0 } } @@ -42,21 +42,23 @@ centos_docker() [ -n "$image" ] || image=${file#*/}_$i if [ -f "$file_dir/build.sh" ] && [ -x "$file_dir/build.sh" ]; then "$file_dir/build.sh" > /dev/null 2> /dev/null && { - centos_one_ok=true handle_dockerfile } else docker_build > /dev/null 2> /dev/null && { - centos_one_ok=true handle_dockerfile } fi let i++ done - [ "$centos_one_ok" == "false" ] && { - echo "centos all failed" + if [ "$nr_openeuler_fail" != 0 ]; then + echo "openeuler failed times: $nr_openeuler_fail" + echo "openeuler_valid_url: $dockerfile_url" + exit 1 + else + echo "current-repo-not-support-centos: 1" exit 0 - } + fi } handle_FROM() @@ -81,19 +83,18 @@ handle_epelrpm() } } -handle_groupadd() +add_base_commands() { - grep -qw "groupadd*" "$file" && { - sed -i '/groupadd*/i\ yum -y install shadow && \\' "$file" - } + # grep -qw "groupadd*" "$file" && { + # fix missing useradd, groupadd, chpasswd, etc. commands + sed -i '/FROM /a\RUN yum -y install shadow tar' "$file" + # } } handle_epel_repo() { - grep -qwE '.* wget .*(epel|CentOS-Base)\.repo' "$file" && { - sed 's|^|#|g' "$file" - } + sed -i '/.* wget .*(epel|CentOS-Base)\.repo/ s|^|#|g' "$file" } handle_dockerfile() @@ -101,15 +102,18 @@ handle_dockerfile() flag=1 handle_FROM handle_epel - handle_groupadd - docker_build "-openeuler" - [ $? -ne 0 ] || docker_run "-openeuler" + add_base_commands + if docker_build "-openeuler"; then + docker_run "-openeuler" + else + let nr_openeuler_fail++ + fi } docker_build() { docker build -f "$file" -t "${image}$1" . - [ $? -ne 0 ] && { + [ $? != 0 ] && { [ "$flag" == "1" ] && echo "${image}$1.build.fail: 1" return 1 } @@ -120,7 +124,8 @@ docker_build() docker_run() { docker run -d --name "${image}" "${image}$1" - docker_stats=$(docker inspect ${image} | grep Running | awk -F ' ' '{print $2}' | tr -d ',') + wait + docker_stats=$(docker inspect -f '{{.State.Running}}' ${image}) [ $docker_stats == false ] && { echo "${image}.run.fail: 1" return 1 -- 2.23.0

2 2

[PATCH v4 lkp-tests] add new type error
by Bai Jing 11 Mar '21

11 Mar '21

[input] Error: Unknown repo: 'powertools' [output] unknown-repo.powertools: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/stats/openeuler_docker b/stats/openeuler_docker index f85c9ca63..4c65c6bb3 100755 --- a/stats/openeuler_docker +++ b/stats/openeuler_docker @@ -6,12 +6,18 @@ sys_set = Set[] while (line = STDIN.gets) case line.chomp! - # Error: Unable to find a match: docker-registry + # Error: Unable to find a match: docker-registry mock xx when /Error: Unable to find a match: (.+)/ $1.split.each do |repo| repo_set << repo end + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + puts "unknown-repo.#{repo}: 1" + puts "unknown-repo.#{repo}.message: #{line}" + # /bin/sh: passwd: command not found when /\/bin\/sh: (.+): command not found/ puts "sh.command-not-found.#{$1}: 1" -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/web-backend: add API /get_table_error
by Lu Weitao 11 Mar '21

11 Mar '21

[Example] curl -X GET 'localhost:30000/get_job_error?group_id=wcl_ansible-openeuler-03-10' return: { "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, "table_fields":["job_id","error_id","error_message","link to result"], "data_table":[ [ "crystal.1354921", "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-xxx.message", "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene… ate/crystal.1354921" ], [ "crystal.1354929", ... ] ... ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 19 +++++++++ src/lib/web_backend.rb | 67 +++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 74eda85..fa2a53d 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,22 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_job_error?suite=virttest&tbox_group=vm-2p8g +# Response +# { +# "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, +# "table_fields":["job_id","error_id","error_message","link to result"], +# "data_table":[ +# [ +# "crystal.1354921", +# "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-failed.message", +# "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.} +# "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…" +# ], +# ... +# ] +# } +get '/get_job_error' do + get_job_error(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..f09fc3d 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/params_group.rb" require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -548,3 +549,69 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + + +# ------------------------------------------------------------------------------------------- +# job error table like: +# job_id error_id error_message link to result +# ------------------------------------------------------------------------------------- +# crystal.630608 "stderr.xxx" "messag:xxxx" https://$host:$port/$result_root +# ... +# ------------------------------------------------------------------------------------------- + +def get_job_error(params) + begin + body = job_error_body(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get error table error'] + end + + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def job_error_body(params) + error_table = get_error_table(params) + { + filter: params, + table_fields: ["job_id", "error_id", "error_message", "link to result"], + data_table: error_table, + }.to_json +end + +def get_error_table(filter_items) + error_table = [] + + job_list = get_job_list(filter_items) + job_list.each do |job| + error_table += get_error_from_job(job) + end + + error_table +end + +def get_job_list(items) + es = ESQuery.new(ES_HOST, ES_PORT) + query_results = es.multi_field_query(items) + + extract_jobs_list(query_results['hits']['hits']) +end + +# get all error_id from one job +def get_error_from_job(job) + job_error = [] + job['stats'].each do |metric, value| + next unless metric.end_with?('.message', 'log') + + result_host = ENV['SRV_HTTP_RESULT_HOST'] || SRV_HTTP_RESULT_HOST + result_port = ENV['SRV_HTTP_RESULT_PORT'] || SRV_HTTP_RESULT_PORT + job_error << [ + job['id'], + metric, + value, + "http://#{result_host}:#{result_port}#{job['result_root']}" + ] + end + + job_error +end -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] src/scheduler: add local mount of kernel_common_params
by Xu Xijian 11 Mar '21

11 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index bb5983f..8ad5029 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -4,8 +4,12 @@ class Job private def kernel_common_params common_params = "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job ip=dhcp" - return "#{common_params} rw" if "#{self.os_mount}" == "local" - return "#{common_params} rootovl ro" + return "#{common_params} rootovl ro" unless "#{self.os_mount}" == "local" + + os_info = "#{os}_#{os_arch}_#{os_version}" + use_root_partition = "/dev/mapper/os-#{os_info}_#{src_lv_suffix}" if @hash["src_lv_suffix"]? != nil + save_root_partition = "/dev/mapper/os-#{os_info}_#{boot_lv_suffix}" if @hash["boot_lv_suffix"]? != nil + return "#{common_params} local use_root_partition=#{use_root_partition} save_root_partition=#{save_root_partition} rw" end private def kernel_custom_params @@ -20,7 +24,7 @@ class Job "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino", "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", - "local" => "root=/dev/mapper/os-#{self.os}_#{self.os_arch}_#{self.os_version}", + "local" => "root=#{OS_HTTP_HOST}:#{os_real_path}", # root is just used to temporarily mount a root in initqueue stage when lvm is not ready "container" => "", } -- 2.23.0

2 1

[PATCH v4 compass-ci 2/2] lib/dump-stat.rb: refactor lkp-tests/sbin/dump-stat
by Lu Weitao 11 Mar '21

11 Mar '21

[Why] support handle hash from lkp-tests/stats/$script.rb Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/dump_stat.rb | 194 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 194 insertions(+) create mode 100644 lib/dump_stat.rb diff --git a/lib/dump_stat.rb b/lib/dump_stat.rb new file mode 100644 index 0000000..1c0695e --- /dev/null +++ b/lib/dump_stat.rb @@ -0,0 +1,194 @@ +# SPDX-License-Identifier: GPL-2.0-only + +LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/statistics" +require "#{LKP_SRC}/lib/bounds" +require "#{LKP_SRC}/lib/yaml" +require "#{LKP_SRC}/lib/job" +require "#{LKP_SRC}/lib/string_ext" +require "#{LKP_SRC}/lib/log" +require 'set' + +UNSTRUCTURED_MONITORS = %w[ftrace].to_set + +def warn_stat(msg, monitor) + log_warn msg + log_warn "check #{RESULT_ROOT}/#{monitor}" +end + +# dump stat which input by lkp-tests/stats/$script return +# input: +# eg-1: +# { +# "pgfree" => [275506, 280018], +# ... +# } +# eg-2: +# { +# "iperf.tcp.sender.bps" => 34804801216.197174, +# "iperf.tcp.receiver.bps" => "34804762215.18231" +# } +module DumpStat + def self.dump_stat(monitor, stat_result) + @result = {} + @invalid_records = [] + @record_index = 0 + @monitor = monitor + + stat_result.each do |key, value| + key = key.resolve_invalid_bytes + next if key[0] == '#' + next if value.empty? || value == 0 + next if monitor =~ /^(dmesg|kmsg)$/ && key =~ /^(message|pattern):/ + + if key =~ /[ \t]/ + @invalid_records.push @record_index + warn_stat "whitespace in stats name: #{key}", @monitor + return nil # for exit current stats/script dump-stat + end + next if assign_log_message(key, value) + + k = @monitor + '.' + key + @result[k] ||= [] + fill_zero(k) + if value.is_a?(String) + value = check_string_value(k, value, @monitor) + next unless value + return nil unless number?(value, @invalid_records) + + value = value.index('.') ? value.to_f : value.to_i + elsif value.is_a?(Array) + (0..value.size - 1).each do |i| + next unless value[i].is_a?(String) + + value[i] = check_string_value(k, value[i], @monitor) + next unless value[i] + return nil unless number?(value[i], @invalid_records) + + value[i] = value[i].index('.') ? value[i].to_f : value[i].to_i + valid_stats_verification(k, value[i]) + end + @result[k] = value + next + end + valid_stats_verification(k, value) + @result[k].push value + end + return nil if @result.empty? + + remove_zero_stats + delete_invalid_number(@result, @invalid_records, @monitor) + cols_verifation + return nil unless useful_result?(@result) + + save_json(@result, "#{RESULT_ROOT}/#{(a)monitor}.json", @result.size * @min_cols > 1000) + end + + # keep message | log line which key end with .message|.log + def self.assign_log_message(key, value) + if key.end_with?('.message', '.log') + k = @monitor + '.' + key + @result[k] = value + return true + end + + false + end + + def self.fill_zero(key) + size = @result[key].size + if @record_index < size + @record_index = size + elsif (@record_index - size).positive? + # fill 0 for missing values + @result[key].concat([0] * (@record_index - size)) + end + end + + def self.valid_stats_verification(key, value) + return nil if valid_stats_range? key, value + + @invalid_records.push @record_index + puts "outside valid range: #{value} in #{key} #{RESULT_ROOT}" + end + + def self.remove_zero_stats + @max_cols = 0 + @min_cols = Float::INFINITY + @min_cols_stat = '' + @max_cols_stat = '' + zero_stats = [] + @result.each do |key, val| + next if key.end_with?('.message', '.log') + + if @max_cols < val.size + @max_cols = val.size + @max_cols_stat = key + end + if @min_cols > val.size + @min_cols = val.size + @min_cols_stat = key + end + next if val[0] != 0 + next if val[-1] != 0 + next if val.sum != 0 + + zero_stats << key + end + zero_stats.each { |x| @result.delete x } + end + + def self.cols_verifation + return nil unless @min_cols < @max_cols && !UNSTRUCTURED_MONITORS.include?(@monitor) + + if @min_cols == @max_cols - 1 + @result.each { |_k, y| y.pop if y.size == @max_cols } + puts "Last record seems incomplete. Truncated #{RESULT_ROOT}/#{(a)monitor}.json" + else + warn_stat "Not a matrix: value size is different - #{@min_cols_stat}: #{@min_cols} != #{@max_cols_stat}: #{@max_cols}: #{RESULT_ROOT}/#{(a)monitor}.json", @monitor + end + end +end + +def check_string_value(key, value, monitor) + # value terminator is expected. If not, throw out an error warning. + warn_stat "no line terminator in stats value: #{value}", monitor if value.chomp!.nil? + + value.strip! + if value.empty? + warn_stat "empty stat value of #{key}", monitor + return nil + end + + return value +end + +# only number is valid +def number?(value, invalid_records) + unless value.numeric? + invalid_records.push record_index + warn_stat "invalid stats value: #{value}", monitor + return nil + end + + true +end + +def delete_invalid_number(result, invalid_records, monitor) + return nil if monitor == 'ftrace' + + invalid_records.reverse_each do |index| + result.each do |_k, value| + value.delete_at index + end + end +end + +def useful_result?(result) + return nil if result.empty? + return nil if result.values[0].size.zero? + return nil if result.values[-1].size.zero? + + true +end -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] sbin/result2stats: support call function from stats/script
by Lu Weitao 11 Mar '21

11 Mar '21

[Why] old logical: lkp-tests/stats/script < script_log > tmpfile lkp-tests/sbin/dump-stat < tmpfile > $script.json this flow will escape special characters [How] use return of function to keep original characters lkp-tests/stats/script.rb func(read(script_log)) --> hash compass-ci/lib/dump-stats.rb hash --> dump_stats() --> $script.json [Dependence] lkp-tests/stats/$script.rb, for example: file_name: lkp-tests/stats/proc-vmstats.rb function name: proc_vmstats() input: Array(String) ["time: 231231234", "nr_free_page: 89721", ...] output: Hash(String, Array(Number|String) | Number | String) { "time" => [231231234, 231231235], "nr_free_page" => [89721,88564], ... } or { "time" => 231231234, "nr_free_page" => 89721, ... } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats.rb | 21 ++++++++++++++++----- lib/stats_wrapper.rb | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 5 deletions(-) diff --git a/lib/stats.rb b/lib/stats.rb index c01d38d..a02f074 100644 --- a/lib/stats.rb +++ b/lib/stats.rb @@ -14,11 +14,18 @@ end def available_stats vailable_stats = [] + available_stats_func = [] Dir.open(LKP_SRC + '/stats').each do |file| - vailable_stats << file unless file.start_with?('.') + next if file.start_with?('.') + + if file.end_with?('.rb') + available_stats_func << file.chomp('.rb') + else + vailable_stats << file + end end - vailable_stats + [vailable_stats, available_stats_func] end def assign_default_stats(stats_list) @@ -32,13 +39,17 @@ class Stats def initialize(result_root) @result_root = result_root @job = load_yaml(File.join(result_root, 'job.yaml')) - @available_stats = available_stats + @available_stats, @available_stats_func = available_stats end def extract_stats stats_list = assign_stats_list stats_list.each do |stat| - StatsWrapper.wrapper(*stat) + if @available_stats_func.include?(stat) + StatsWrapper.wrapper_func(*stat) + else + StatsWrapper.wrapper(*stat) + end end end @@ -48,7 +59,7 @@ class Stats stats_list << ['time', @job['suite'] + '.time'] stats_list << 'stderr' @job.each_key do |k| - if @available_stats.include?(k) + if @available_stats.include?(k) || @available_stats_func.include?(k) stats_list << k end end diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index 8a6d6a9..faa81df 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -6,6 +6,11 @@ LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' require "#{LKP_SRC}/lib/log.rb" require 'tempfile' require 'English' +require_relative './dump_stat' + +Dir[File.expand_path("#{LKP_SRC}/stats/*.rb")].uniq.each do |file| + require file +end PROGRAM_DIR = "#{LKP_SRC}/stats" @@ -35,6 +40,27 @@ module StatsWrapper delete_log_package end + def self.wrapper_func(program, program_time = nil) + @program = program + @stats_group = program_time || program + @log = "#{RESULT_ROOT}/#{@stats_group}" + + return unless File.exist?("#{(a)log}.yaml") || pretreatment + + if File.exist?("#{(a)log}.yaml") + log_lines = read_log("#{(a)log}.yaml") + stat_result = parse_simple_log_yaml(log_lines) + else + log_lines = read_log(@log) + call_func_cmd = "#{(a)program.gsub('-', '_')}(log_lines)" # eg: proc_vmstats(log_lines) + stat_result = eval(call_func_cmd) + end + return unless DumpStat.dump_stat(@stats_group, stat_result) + + check_empty_json + delete_log_package + end + def self.pretreatment return unless available_program? return unless unzip_log @@ -261,3 +287,21 @@ module StatsWrapper File.delete(@tmpfile) if File.exist?(@tmpfile) end end + +# read line of log +# return Array(String) +def read_log(log_path) + return nil unless File.exist?(log_path) + + File.readlines(log_path) +end + +def parse_simple_log_yaml(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + log_lines.each do |line| + key, value = line.split(/:?\s+/) + result[key] << value + end + + result +end -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/web-backend/start: add SRV_HTTP_RESULT/PORT to env
by Lu Weitao 11 Mar '21

11 Mar '21

[Why] the API /get_job_error need the host/port Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/start | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/container/web-backend/start b/container/web-backend/start index 941fb11..6731d27 100755 --- a/container/web-backend/start +++ b/container/web-backend/start @@ -9,6 +9,10 @@ . $CCI_SRC/container/defconfig.sh docker_rm web-backend +load_cci_defaults +[[ $SRV_HTTP_RESULT_HOST ]] || SRV_HTTP_RESULT_HOST=$(curl ip.sb) +[[ $SRV_HTTP_RESULT_PORT ]] || SRV_HTTP_RESULT_PORT=11300 + cmd=( docker run --name web-backend @@ -17,6 +21,8 @@ cmd=( -p 32767:32767 -e ES_PORT=9200 + -e SRV_HTTP_RESULT_HOST=$SRV_HTTP_RESULT_HOST + -e SRV_HTTP_RESULT_PORT=$SRV_HTTP_RESULT_PORT -e CCI_SRC=/c/compass-ci -e LKP_SRC=/c/lkp-tests -e UPSTREAM_REPOS_PATH=/c/upstream-repos -- 2.23.0

1 0

[PATCH compass-ci] container: add start depends
by Cao Xueliang 11 Mar '21

11 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/auto-submit/start-depends | 2 +- container/delimiter/start-depends | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/container/auto-submit/start-depends b/container/auto-submit/start-depends index 38fc696..b5a1e0c 100755 --- a/container/auto-submit/start-depends +++ b/container/auto-submit/start-depends @@ -1,2 +1,2 @@ rabbitmq -es +register-accounts diff --git a/container/delimiter/start-depends b/container/delimiter/start-depends index 95df1a1..880f5b1 100755 --- a/container/delimiter/start-depends +++ b/container/delimiter/start-depends @@ -1,2 +1,3 @@ taskqueue send-mail +register-accounts -- 2.23.0

2 2

[PATCH v2 compass-ci 2/3] container/dracut-initrd: set sysroot in local mount situation
by Xu Xijian 11 Mar '21

11 Mar '21

When mount local filesystem in os starting, we will create or use a local existed logical volume to replace remote filesystem. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 container/dracut-initrd/bin/set-local-sysroot.sh diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh new file mode 100644 index 0000000..dc9000b --- /dev/null +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -0,0 +1,83 @@ +#!/bin/sh + +analyse_kernel_cmdline_params() { + rootfs="$(getarg root=)" + # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 + rootfs_src=$(echo $"$rootfs" | sed 's/\///') + timestamp="$(basename $root)" + os="$(echo $root_src | awk -F '/|-' '{print $2}')" + os_arch="$(echo $root_src | awk -F '/|-' '{print $3}')" + os_version="$(echo $root_src | awk -F '/|-' '{print $4}')" + os_info="${os}_${os_arch}_${os_version}" + export rootfs_src timestamp os_info +} + +sync_src_lv() { + local src_lv="$1" + + [ -e "$src_lv" ] && return + + # create logical volume + src_lv_devname="$(basename $src_lv)" + lvm lvcreate -y -L 10G --name "${src_lv_devname#os-}" os + mke2fs -t ext4 -F "$src_lv" + + # sync nfsroot to $src_lv + mkdir -p /mnt1 && mount -t nfs "$rootfs_src" /mnt1 + mkdir -p /mnt2 && mount "$src_lv" /mnt2 + cp -a /mnt1/. /mnt2/ + umount /mnt1 /mnt2 + + # change permission of "$src_lv" to readonly + lvm lvchange --permission r "$src_lv" +} + +snapshot_boot_lv() { + local src_lv="$1" + local boot_lv="$2" + + [ "$src_lv" == "$boot_lv" ] && return + + lvm lvremove --force "$boot_lv" + boot_lv_devname="$(basename $boot_lv)" + lvm lvcreate --size 10G --name ${boot_lv_devname#os-} --snapshot "$src_lv" +} + +set_sysroot() { + boot_lv="$1" + umount "$NEWROOT" + mount "$boot_lv" "$NEWROOT" +} + +if ! getargbool 0 local; then + exit 0 +fi + +analyse_kernel_cmdline_params + +sed -i "s/^locking_type = .*/locking_type = 1/" /etc/lvm/lvm.conf + +use_root_partition="$(getarg use_root_partition=)" +if [ -n "$use_root_partition" ]; then + src_lv="$use_root_partition" # /dev/mapper/os-${os_info}_${timestamp} + sync_src_lv "$src_lv" +else + src_lv="/dev/mapper/os-${os_info}_$timestamp" + [ -e "src_lv" ] || { + echo "warn dracut: FATAL: no src_lv with local mount, reboot" + reboot + } +fi + +save_root_partition="$(getarg save_root_partition=)" +if [ -n "$save_root_partition" ]; then + boot_lv="$save_root_partition" +else + boot_lv="/dev/mapper/os-${os_info}" +fi + +snapshot_boot_lv "$src_lv" "$boot_lv" + +set_sysroot "$boot_lv" + +exit 0 -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/dracut-initrd: add handling of pre-mount stage into setup-dracut.sh
by Xu Xijian 11 Mar '21

11 Mar '21

Add the content of set-local-sysroot.sh into the tail of dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 849480d..45e895c 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -18,3 +18,6 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh +pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" +[ "$(sed -n '$p' $pre_mount_file)" = "exit 0" ] && sed -i '$d' "$pre_mount_file" +cat set-local-sysroot.sh >> "$pre_mount_file" -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] lib/git_mirror.rb: github download workaround
by Li Yuanchao 10 Mar '21

10 Mar '21

There is a problem to download from github by http or https. fatal: unable to access 'https://github.com/liyc-github/AvxToNeon/': Failed to connect to github.com port 443: Connection timed out When it occurs, change url to "git://xxx". Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 8f23af3..50a6c75 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -39,19 +39,27 @@ class GitMirror return url end + def stderr_443?(stderr_info) + return true if stderr_info.include?('Failed to connect to github.com port 443') + + return false + end + def git_clone(url, mirror_dir) url = get_url(Array(url)[0]) 10.times do - %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) + stderr = %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) return true if File.directory?(mirror_dir) && File.exist?("#{mirror_dir}/config") + + url = "git://#{url.split('://')[1]}" if stderr_443?(stderr) end return false end def git_fetch(mirror_dir) if File.exist?("#{mirror_dir}/shallow") + FileUtils.rm("#{mirror_dir}/shallow.lock") if File.exist?("#{mirror_dir}/shallow.lock") %x(git -C #{mirror_dir} fetch --unshallow 2>&1) - return false end fetch_info = %x(git -C #{mirror_dir} fetch 2>&1) @@ -59,6 +67,11 @@ class GitMirror if fetch_info.include?(ERR_MESSAGE) && Dir.empty?(mirror_dir) FileUtils.rmdir(mirror_dir) end + + if stderr_443?(fetch_info) + url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp + %x(git -C #{mirror_dir} remote set-url origin git://#{url.split('://')[1]}) + end return fetch_info.include? '->' end @@ -66,7 +79,7 @@ class GitMirror url = get_url(Array(url)[0]) git_url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp - return true if url == git_url + return true if git_url == url || git_url.include?(url.split('://')[1]) return false end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] sbin/auto_submit: change the path to do git command
by Li Yuanchao 10 Mar '21

10 Mar '21

As the path of repo mirrors under /srv/git changes, the path to execute git command should change too. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 857af29..faebb2f 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -44,14 +44,16 @@ class AutoSubmit submit_argv end - def submit_one_job(submit_argv, git_repo, branch, commit_id) - inactive_time = %x(git -C /srv/git/#{git_repo}.git log --pretty=format:"%cr" -1 #{commit_id}) + def submit_one_job(submit_argv, git_repo, belong, branch, commit_id) + mirror_dir = "/srv/git/#{belong}/#{git_repo}.git" + inactive_time = %x(git -C #{mirror_dir} log --pretty=format:"%cr" -1 #{commit_id}) return if inactive_time =~ /(month|year)/ real_argvs = Array.new(submit_argv) real_argvs.push("upstream_branch=#{branch}") real_argvs.push("upstream_commit=#{commit_id}") - tag = %x(git -C /srv/git/#{git_repo}.git tag --points-at #{commit_id}) + real_argvs.push("upstream_dir=#{belong}") + tag = %x(git -C #{mirror_dir} tag --points-at #{commit_id}) real_argvs.push("upstream_tag=#{tag}") unless tag.empty? system(real_argvs.join(' ')) @@ -60,16 +62,17 @@ class AutoSubmit def submit_specific_branch(submit_argv, newrefs_info, branches) branches.each do |branch_name| branch = "refs/heads/#{branch_name}" - next unless newrefs_info['new_refs']['heads'][branch] + commit_id = newrefs_info['new_refs']['heads'][branch] + next unless commit_id - submit_one_job(submit_argv, newrefs_info['git_repo'], branch_name, newrefs_info['new_refs']['heads'][branch]) + submit_one_job(submit_argv, newrefs_info['git_repo'], newrefs_info['belong'], branch_name, commit_id) end end - def submit_non_specific(non_specific_commits, git_repo, submit_argv) - non_specific_commits.each do |branch, commit_id| + def submit_non_specific(newrefs_info, submit_argv) + newrefs_info['new_refs']['heads'].each do |branch, commit_id| branch = branch.delete_prefix('refs/heads/') - submit_one_job(submit_argv, git_repo, branch, commit_id) + submit_one_job(submit_argv, newrefs_info['git_repo'], newrefs_info['belong'], branch, commit_id) end end @@ -96,7 +99,7 @@ class AutoSubmit if element['branches'] submit_specific_branch(argvs, newrefs_info, element['branches']) else - submit_non_specific(newrefs_info['new_refs']['heads'], newrefs_info['git_repo'], argvs) + submit_non_specific(newrefs_info, argvs) end end end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] lib/git_mirror.rb: don't count new ref if it is not new
by Li Yuanchao 10 Mar '21

10 Mar '21

When repos are added, new_refs_count will increase. But many repos are not updated that day, that cause new_refs_count increase much more than real new refs number. Now only count those really new. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 53c9e59..8f23af3 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -194,7 +194,7 @@ class MirrorMain feedback_info[:new_refs] = new_refs send_message(feedback_info) - new_refs_log(git_repo, new_refs[:heads].length) + new_refs_log(git_repo, new_refs[:heads].length) if last_commit_new?(git_repo) end def do_push(fork_key) @@ -327,7 +327,7 @@ class MirrorMain def update_fork_stat(git_repo, possible_new_refs) update_stat_fetch(git_repo) - update_stat_new_refs(git_repo) if possible_new_refs + update_stat_new_refs(git_repo) if possible_new_refs && last_commit_new?(git_repo) es_repo_update(git_repo) end end @@ -521,4 +521,11 @@ class MirrorMain nr_new_branch: nr_new_branch }.to_json) end + + def last_commit_new?(git_repo) + inactive_time = %x(git -C /srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}.git log --pretty=format:"%cr" -1) + return false if inactive_time =~ /(day|week|month|year)/ + + return true + end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 11/11] install-iso-pre: add step: submit install-iso-sync-rootfs.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 830c6bfd8a6a..ef4a0659fe02 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -236,7 +236,22 @@ submit_install_iso_firstboot_yaml() submit_install_iso_sync_rootfs_yaml() { - : + log_info "start submit install-iso-sync-rootfs.yaml" + + local sync_rootfs_yaml="${LKP_SRC}/jobs/install-iso-sync-rootfs.yaml" + [ -f "${sync_rootfs_yaml}" ] || die "cannot find ${sync_rootfs_yaml}" + + ${LKP_SRC}/sbin/submit \ + ${sync_rootfs_yaml} \ + suite=install-iso-sync-rootfs-${ROOTFS_SUFFIX} \ + iso_os=${iso_os} \ + iso_arch=${iso_arch} \ + iso_version=${iso_version} \ + NFS_SERVER_HOST=${NFS_SERVER_HOST} \ + NFS_ROOT_DIR=${NFS_ROOT_DIR} \ + ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ + VG_NAME=${VG_NAME} \ + testbox=$testbox || die "submit ${sync_rootfs_yaml} failed" } main() -- 2.23.0

1 0

[PATCH v2 lkp-tests 10/11] install-iso-pre: add step: submit install-iso-firstboot.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index cc939ba20ce8..830c6bfd8a6a 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -218,7 +218,20 @@ submit_install_iso_yaml() submit_install_iso_firstboot_yaml() { - : + log_info "start submit install-iso-firstboot.yaml" + + local firstboot_yaml="${LKP_SRC}/jobs/install-iso-firstboot.yaml" + [ -f "${firstboot_yaml}" ] || die "cannot find ${firstboot_yaml}" + + ${LKP_SRC}/sbin/submit \ + ${firstboot_yaml} \ + suite=install-iso-firstboot-${ROOTFS_SUFFIX} \ + os=${iso_os} \ + os_arch=${iso_arch} \ + os_version=${iso_version} \ + os_mount=local \ + kernel_custom_params="src_lv_suffix=${ROOTFS_SUFFIX} boot_lv_suffix=${ROOTFS_SUFFIX}" \ + testbox=$testbox || die "submit ${firstboot_yaml} failed" } submit_install_iso_sync_rootfs_yaml() -- 2.23.0

1 0

[PATCH v2 lkp-tests 09/11] install-iso-pre: add step: submit install-iso.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 000c1fd6811b..cc939ba20ce8 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -185,9 +185,35 @@ prepare_submit() EOF } +prepare_install_iso_yaml() +{ + log_info "start prepare install-iso.yaml" + + local tftp_root="tftp://${TFTP_SERVER_HOST}/EFI/BOOT/${iso_os}/${iso_arch}/${iso_version}" + local custom_ipxe + + custom_ipxe="#!ipxe\\\\n" + custom_ipxe="${custom_ipxe}\\\\ndhcp" + custom_ipxe="${custom_ipxe}\\\\ninitrd ${tftp_root}/initrd.img" + custom_ipxe="${custom_ipxe}\\\\nkernel ${tftp_root}/vmlinux ro inst.geoloc=0 console=ttyS0 smmu.bypassdev=0x1000:0x17 smmu.bypassde1000:0x15 viff video=VGA-1:640x480-32@60me initrd=initrd.img ip=dhcp inst.ks=http://${HTTP_SERVER_HOST}:${HTTP_SERVER_PORT}/os/kickstart/ks/${iso_os}_${iso_arch}_${iso_version}.cfg" + custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" + + log_info "custom_ipxe: $custom_ipxe" + sed -i "/custom_ipxe:/ccustom_ipxe: \"${custom_ipxe}\"" ${install_iso_yaml} +} + submit_install_iso_yaml() { - : + log_info "start submit install-iso.yaml" + + local install_iso_yaml="${LKP_SRC}/jobs/install-iso.yaml" + [ -f "${install_iso_yaml}" ] || die "cannot find ${install_iso_yaml}" + + prepare_install_iso_yaml + + ${LKP_SRC}/sbin/submit \ + ${install_iso_yaml} \ + testbox=$testbox || die "submit ${install_iso_yaml} failed" } submit_install_iso_firstboot_yaml() -- 2.23.0

1 0

[PATCH v2 lkp-tests 08/11] install-iso-pre: add step: prepare_submit
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-pre | 2 ++ tests/install-iso-pre | 18 +++++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/distro/depends/install-iso-pre b/distro/depends/install-iso-pre index 6f88eb919896..4569e1e3cdc0 100644 --- a/distro/depends/install-iso-pre +++ b/distro/depends/install-iso-pre @@ -1 +1,3 @@ lvm2 +ruby +git diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 7b9e6fd2107c..000c1fd6811b 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -166,7 +166,23 @@ prepare_ks() prepare_submit() { - : + ${LKP_SRC}/sbin/install-dependencies.sh + + export CCI_REPOS="/c" + + mkdir -p $CCI_REPOS && cd $CCI_REPOS + git clone $LAB_Z9_URL || die "clone lab-z9 git repo failed: $LAB_Z9_URL" + + local config_yaml="/etc/compass-ci/defaults/sparrow.yaml" + mkdir -p $(dirname $config_yaml) + cat >> $config_yaml <<-EOF + SCHED_HOST: ${SCHED_HOST} + SCHED_PORT: ${SCHED_PORT} + my_name: ${my_name} + my_email: ${my_email} + my_token: ${MY_TOKEN} + lab: z9 + EOF } submit_install_iso_yaml() -- 2.23.0

1 0

[PATCH v2 lkp-tests 07/11] install-iso-pre: add step: prepare_ks
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 72 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index d800704f648c..7b9e6fd2107c 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -89,9 +89,79 @@ prepare_lv() create_lv } +generate_ks() +{ + log_info "start generate $ks_file" + + local nic=$(ip route get ${LKP_SERVER} | awk '{print $3; exit}') + + cat > ${ks_file} <<-EOF + # Use network installation + url --url="http://${HTTP_SERVER_HOST}:${HTTP_SERVER_PORT}/${HTTP_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version}/" + + # Network information + network --bootproto=dhcp --device=${nic} --ipv6=auto --activate + network --hostname=localhost.localdomain + + # Specifies a list of disks for the installation program to use. All other disks are ignored. + ignoredisk --only-use=${rootfs_disk#/dev/} + + # if installation target is a logical volume, bootloader should be none. + bootloader --location=none + logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting + + # Root password + rootpw --iscrypted ${ROOTFS_ROOT_PASSWD} + + # Reboot after installation + reboot --eject + + # Use text mode install + text + + # Keyboard layouts + keyboard --vckeymap=us --xlayouts='us' + + # System language + lang en_US.UTF-8 + + # Do not configure the X Window System + skipx + + # System services + services --enabled="chronyd" + + # System timezone + timezone Asia/Shanghai --isUtc + + %packages + @^minimal-environment + + %end + + %anaconda + pwpolicy root --minlen=8 --minquality=1 --notstrict --nochanges --notempty + pwpolicy user --minlen=8 --minquality=1 --notstrict --nochanges --emptyok + pwpolicy luks --minlen=8 --minquality=1 --notstrict --nochanges --notempty + %end + EOF +} + +upload_ks() +{ + log_info "start upload ks to rsync-server: ${RSYNC_SERVER_HOST}:${RSYNC_SERVER_PORT}" + + rsync -vrtopg --port ${RSYNC_SERVER_PORT} ${ks_file} ${RSYNC_SERVER_HOST}::ks/ || die "upload ks to rsync-server failed, please check." +} + prepare_ks() { - : + log_info "start prepare kickstart file" + + local ks_file=/tmp/${iso_os}_${iso_arch}_${iso_version}.ks + + generate_ks + upload_ks } prepare_submit() -- 2.23.0

1 0

[PATCH v2 lkp-tests 06/11] install-iso-pre: add step: prepare_lv
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-pre | 1 + tests/install-iso-pre | 46 +++++++++++++++++++++++++++++++++- 2 files changed, 46 insertions(+), 1 deletion(-) create mode 100644 distro/depends/install-iso-pre diff --git a/distro/depends/install-iso-pre b/distro/depends/install-iso-pre new file mode 100644 index 000000000000..6f88eb919896 --- /dev/null +++ b/distro/depends/install-iso-pre @@ -0,0 +1 @@ +lvm2 diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 4817f6d8eb37..d800704f648c 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -40,9 +40,53 @@ check_params() done } +create_vg() +{ + log_info "start create vg:$VG_NAME" + + local disk + + for disk in "${rootfs_disk[@]}" + do + log_info "start handle disk: $disk" + pvdisplay ${disk} > /dev/null || pvcreate ${disk} || die "create pv failed: $disk" + + if vgdisplay ${VG_NAME} > /dev/null; then + vgextend ${VG_NAME} ${disk} + else + vgcreate ${VG_NAME} ${disk} + fi + done + +} + +create_lv() +{ + log_info "start create lv:$LV_NAME" + + lvcreate -y -L 10G -n ${LV_NAME} ${VG_NAME} || die "create logical volume failed." +} + prepare_lv() { - : + log_info "start prepare logical volume" + + # activate lvm + vgchange -ay + + # VG_NAME defines in job.yaml + export VG_NAME + export LV_NAME="${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX}" + + # lv existed + lvdisplay /dev/mapper/${VG_NAME}-${LV_NAME} > /dev/null && { + log_info "iso already install to this testbox, so rootfs maybe already generated ok. please check it." + exit 0 + } + + # lv not existed + vgdisplay ${VG_NAME} > /dev/null || create_vg + create_lv } prepare_ks() -- 2.23.0

1 0

[PATCH v2 lkp-tests 05/11] feat: step 4 of iso2rootfs: add entry job
by Yu Chuan 10 Mar '21

10 Mar '21

Framework of install-iso-pre. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso-pre.yaml | 52 +++++++++++++++++++++++ tests/install-iso-pre | 86 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 138 insertions(+) create mode 100644 jobs/install-iso-pre.yaml create mode 100755 tests/install-iso-pre diff --git a/jobs/install-iso-pre.yaml b/jobs/install-iso-pre.yaml new file mode 100644 index 000000000000..a9dbe46b4bf3 --- /dev/null +++ b/jobs/install-iso-pre.yaml @@ -0,0 +1,52 @@ +suite: install-iso-pre + +install-iso-pre: + ################# + # iso info + ################# + iso_os: + iso_arch: + iso_version: + + ################# + # http server info, which stored the installation source + ################# + HTTP_SERVER_HOST: + HTTP_SERVER_PORT: + HTTP_ROOT_DIR: + + ################# + # rsync-server info, used for upload ks + ################# + RSYNC_SERVER_HOST: + RSYNC_SERVER_PORT: + + ################# + # tftp server info, used for kickstart auto install + ################# + TFTP_SERVER_HOST: + + ################# + # nfs server info, used for store rootfs + ################# + NFS_SERVER_HOST: + NFS_ROOT_DIR: + + ################# + # result rootfs suffix, format: $(date "+%Y%m%d%H%M%S") + ################# + ROOTFS_SUFFIX: + + ################# + # name of volume group which stores the logical volume of rootfs + ################# + VG_NAME: + + ################# + # lab_z9 git repo clone url, used for find host file when submit + ################# + LAB_Z9_URL: + +secrets: + ROOTFS_ROOT_PASSWD: + MY_TOKEN: diff --git a/tests/install-iso-pre b/tests/install-iso-pre new file mode 100755 index 000000000000..4817f6d8eb37 --- /dev/null +++ b/tests/install-iso-pre @@ -0,0 +1,86 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +set -e + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/log.sh + +check_params() +{ + log_info "start check params" + + local required_vars=( + "iso_os" + "iso_arch" + "iso_version" + "ROOTFS_SUFFIX" + "VG_NAME" + + "HTTP_SERVER_HOST" + "HTTP_SERVER_PORT" + "HTTP_ROOT_DIR" + "ROOTFS_ROOT_PASSWD" + "rootfs_disk" + + "RSYNC_SERVER_HOST" + "RSYNC_SERVER_PORT" + "TFTP_SERVER_HOST" + "NFS_SERVER_HOST" + "NFS_ROOT_DIR" + + "LAB_Z9_URL" + "MY_TOKEN" + ) + + for i in "${required_vars[@]}" + do + [ -n "$(eval echo \$$i)" ] || die "cannot get value of var: $i" + done +} + +prepare_lv() +{ + : +} + +prepare_ks() +{ + : +} + +prepare_submit() +{ + : +} + +submit_install_iso_yaml() +{ + : +} + +submit_install_iso_firstboot_yaml() +{ + : +} + +submit_install_iso_sync_rootfs_yaml() +{ + : +} + +main() +{ + check_params + + prepare_lv + prepare_ks + prepare_submit + + submit_install_iso_yaml + submit_install_iso_firstboot_yaml + submit_install_iso_sync_rootfs_yaml +} + +main -- 2.23.0

1 0

[PATCH v2 lkp-tests 04/11] install-iso-sync-rootfs: add step: sync_rootfs
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-sync-rootfs | 2 + tests/install-iso-sync-rootfs | 53 +++++++++++++++++++++++++- 2 files changed, 54 insertions(+), 1 deletion(-) create mode 100644 distro/depends/install-iso-sync-rootfs diff --git a/distro/depends/install-iso-sync-rootfs b/distro/depends/install-iso-sync-rootfs new file mode 100644 index 000000000000..2abb6b833291 --- /dev/null +++ b/distro/depends/install-iso-sync-rootfs @@ -0,0 +1,2 @@ +nfs-common +lvm2 diff --git a/tests/install-iso-sync-rootfs b/tests/install-iso-sync-rootfs index e5de4d72240e..34096229ccfb 100755 --- a/tests/install-iso-sync-rootfs +++ b/tests/install-iso-sync-rootfs @@ -7,6 +7,20 @@ set -e . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/log.sh +SKIP_SYNC_DIRS=( + "srv" + "opt" + "mnt" + "media" + "home" + "lost+found" + "dev" + "proc" + "run" + "sys" + "var/log" +) + check_params() { log_info "start check params" @@ -28,9 +42,46 @@ check_params() done } +prepare_exclude_file() +{ + local exclude_file=$1 + + rm -rf $exclude_file + for i in "${SKIP_SYNC_DIRS[@]}" + do + echo $i >> $exclude_file + done +} + sync_rootfs_to_nfs_server() { - : + # mount nfs rootfs + local nfs_rootfs_mount_point=/tmp/nfs_rootfs_${ROOTFS_SUFFIX} + + mkdir -p ${nfs_rootfs_mount_point} + mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" + + # activate the lvm + vgchange -ay + + # mount local logical volume + local lv_rootfs_mount_point=/tmp/lv_rootfs_${ROOTFS_SUFFIX} + + mkdir -p ${lv_rootfs_mount_point} + mount /dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} ${lv_rootfs_mount_point} || die "mount lv rootfs failed" + + # sync rootfs + local exclude_file=/tmp/exclude_file_${ROOTFS_SUFFIX} + + prepare_exclude_file "$exclude_file" + rsync -az ${lv_rootfs_mount_point}/. ${nfs_rootfs_mount_point} --exclude-from=$exclude_file || die "copy out rootfs failed" + cd ${nfs_rootfs_mount_point} && mkdir -p "${SKIP_SYNC_DIRS[@]}" + + # umount nfs rootfs + umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" + + # umount local logical volume + umount $lv_rootfs_mount_point || log_warn "umount failed: $lv_rootfs_mount_point" } main() -- 2.23.0

1 0

[PATCH v2 lkp-tests 03/11] feat: step 3 of iso2rootfs: copy out rootfs in second boot stage
by Yu Chuan 10 Mar '21

10 Mar '21

Framework of install-iso-sync-rootfs. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso-sync-rootfs.yaml | 22 +++++++++++++++ tests/install-iso-sync-rootfs | 46 +++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+) create mode 100644 jobs/install-iso-sync-rootfs.yaml create mode 100755 tests/install-iso-sync-rootfs diff --git a/jobs/install-iso-sync-rootfs.yaml b/jobs/install-iso-sync-rootfs.yaml new file mode 100644 index 000000000000..0235b10c718b --- /dev/null +++ b/jobs/install-iso-sync-rootfs.yaml @@ -0,0 +1,22 @@ +suite: install-iso-sync-rootfs +category: benchmark +install-iso-sync-rootfs: + ################# + # nfs target path: ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version} + ################# + iso_os: + iso_arch: + iso_version: + NFS_SERVER_HOST: + NFS_ROOT_DIR: + + ################# + # result rootfs suffix, format: $(date "+%Y%m%d%H%M%S") + ################# + ROOTFS_SUFFIX: + + ################# + # name of volume group which stores the logical volume of rootfs + ################# + VG_NAME: + diff --git a/tests/install-iso-sync-rootfs b/tests/install-iso-sync-rootfs new file mode 100755 index 000000000000..e5de4d72240e --- /dev/null +++ b/tests/install-iso-sync-rootfs @@ -0,0 +1,46 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +set -e + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/log.sh + +check_params() +{ + log_info "start check params" + + local required_vars=( + "iso_os" + "iso_arch" + "iso_version" + "NFS_SERVER_HOST" + "NFS_ROOT_DIR" + + "ROOTFS_SUFFIX" + "VG_NAME" + ) + + for i in "${required_vars[@]}" + do + [ -n "$(eval echo \$$i)" ] || die "cannot get value of var: $i" + done +} + +sync_rootfs_to_nfs_server() +{ + : +} + +main() +{ + check_params + + sync_rootfs_to_nfs_server + + lvchange --permission r /dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} || + die "change logical volume permission to readonly failed:/dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX}" +} + +main -- 2.23.0

1 0

[PATCH v2 lkp-tests 02/11] feat: step 2 of iso2rootfs: reboot directly in firstboot stage
by Yu Chuan 10 Mar '21

10 Mar '21

[Why] Firstboot stage: the first boot after iso installation is complete. At this stage, the system will run some initial configurations. So we need copy out the rootfs in second boot stage, and do nothing in firstboot stage. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso-firstboot.yaml | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 jobs/install-iso-firstboot.yaml diff --git a/jobs/install-iso-firstboot.yaml b/jobs/install-iso-firstboot.yaml new file mode 100644 index 000000000000..7be9eaaab184 --- /dev/null +++ b/jobs/install-iso-firstboot.yaml @@ -0,0 +1,10 @@ +suite: install-iso-firstboot + +sleep: 1 + +# os_mount must be equal to 'local' +os_mount: local + +# kernel_custom_params must have 'boot_lv_suffix' and 'src_lv_suffix', +# and they must be the same. so can boot from the specify logical volume +kernel_custom_params: -- 2.23.0

1 0

[PATCH v2 lkp-tests 01/11] feat: step 1 of iso2rootfs: install iso to testbox and reboot
by Yu Chuan 10 Mar '21

10 Mar '21

[Why] When testbox request job in pxe -> ipxe step, scheduler returns a response composed of ipxe cmd. So we add 'custom_ipxe' field of job.yaml, then we can customize the response what testbox get, then testbox can auto install iso by executing the specified ipxe cmd. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso.yaml | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 jobs/install-iso.yaml diff --git a/jobs/install-iso.yaml b/jobs/install-iso.yaml new file mode 100644 index 000000000000..bcccc6b76f85 --- /dev/null +++ b/jobs/install-iso.yaml @@ -0,0 +1,3 @@ +suite: install-iso + +custom_ipxe: -- 2.23.0

1 0

[PATCH lkp-tests 11/11] install-iso-pre: add step: submit install-iso-release.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 17a821cf3c14..4d6bae205f19 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -248,7 +248,22 @@ submit_install_iso_firstboot_yaml() submit_install_iso_release_yaml() { - : + log_info "start submit install-iso-release.yaml" + + local release_yaml="${LKP_SRC}/jobs/install-iso-release.yaml" + [ -f "${release_yaml}" ] || die "cannot find ${release_yaml}" + + ${LKP_SRC}/sbin/submit \ + ${release_yaml} \ + suite=install-iso-release-${ROOTFS_SUFFIX} \ + iso_os=${iso_os} \ + iso_arch=${iso_arch} \ + iso_version=${iso_version} \ + NFS_SERVER_HOST=${NFS_SERVER_HOST} \ + NFS_ROOT_DIR=${NFS_ROOT_DIR} \ + ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ + VG_NAME=${VG_NAME} \ + testbox=$testbox || die "submit ${release_yaml} failed" } main() -- 2.23.0

2 2

[PATCH v2 lkp-tests] add test script for dockerfile os migrating
by Bai Jing 10 Mar '21

10 Mar '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/openeuler_docker | 142 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100755 tests/openeuler_docker diff --git a/tests/openeuler_docker b/tests/openeuler_docker new file mode 100755 index 000000000..d713924bf --- /dev/null +++ b/tests/openeuler_docker @@ -0,0 +1,142 @@ +#!/bin/sh +# - dockerfile_url + +# exit 1 if openEuler failed +# exit 0 if +# - no centos support +# - centos failed + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/log.sh + +[ -n "$dockerfile_url" ] || die "dockerfile_url is empty" + +cd $BENCHMARK_ROOT + +nr_openeuler_fail=0 + +fit_repo() +{ + yum reinstall -y git + git clone "$dockerfile_url" || exit 0 + file_dir=$(find ./ -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]') + [ -n "$file_dir" ] || { + echo "current-repo-not-support-centos: 1" + exit 0 + } +} + +openeuler_ready() +{ + docker load < $BENCHMARK_ROOT/openeuler_docker/openeuler-20.03-lts-v1.tar +} + +centos_docker() +{ + flag=0 + i=1 + for file in $(find ./ -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]' | awk -F: '{print $1}') + do + file_dir=${file%/*} + image=$(echo "$file" | awk -F '/' '{print $3}' | tr 'A-Z' 'a-z') + [ -n "$image" ] || image=${file#*/}_$i + if [ -f "$file_dir/build.sh" ] && [ -x "$file_dir/build.sh" ]; then + "$file_dir/build.sh" > /dev/null 2> /dev/null && { + handle_dockerfile + } + else + docker_build > /dev/null 2> /dev/null && { + handle_dockerfile + } + fi + let i++ + done + if [ "$nr_openeuler_fail" != 0 ]; then + echo "openeuler failed times: $nr_openeuler_fail" + echo "openeuler_valid_url: $dockerfile_url" + exit 1 + else + echo "current-repo-not-support-centos: 1" + exit 0 + fi +} + +handle_FROM() +{ + sed -i "s/^FROM .*/FROM openeuler-20.03-lts:v1/g" "$file" + +} + +handle_epel() +{ + grep -qw "epel-release" "$file" && { + sed -i "s/epel-release/bash/g" "$file" + } +} + +handle_epelrpm() +{ + + grep -qw "https://.*epel-release.*rpm" "$file" && { + sed -i "s|https://.*epel-release.*rpm|bash|g" "$file" + + } +} + +add_base_commands() +{ + # grep -qw "groupadd*" "$file" && { + # fix missing useradd, groupadd, chpasswd, etc. commands + sed -i '/FROM /a\RUN yum -y install shadow tar' "$file" +} + +handle_epel_repo() +{ + sed -i '/.* wget .*(epel|CentOS-Base)\.repo/ s|^|#|g' "$file" +} + +handle_dockerfile() +{ + flag=1 + handle_FROM + handle_epel + add_base_commands + if docker_build "-openeuler"; then + docker_run "-openeuler" + else + let nr_openeuler_fail++ + fi +} + +docker_build() +{ + docker build -f "$file" -t "${image}$1" . + [ $? != 0 ] && { + [ "$flag" == "1" ] && echo "${image}$1.build.fail: 1" + return 1 + } + [ "$flag" == "1" ] && echo "${image}$1.build.success: 1" + flag=0 +} + +docker_run() +{ + docker run -d --name "${image}" "${image}$1" + wait + docker_stats=$(docker inspect -f '{{.State.Running}}' ${image}) + [ $docker_stats == false ] && { + echo "${image}.run.fail: 1" + return 1 + } + + echo "${image}.run.success: 1" +} + +docker_test() +{ + fit_repo + openeuler_ready + centos_docker +} + +docker_test -- 2.23.0

2 1

[PATCH lkp-tests 10/11] install-iso-pre: add step: submit install-iso-firstboot.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 45567feddd9c..17a821cf3c14 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -218,7 +218,32 @@ submit_install_iso_yaml() submit_install_iso_firstboot_yaml() { - : + log_info "start submit install-iso-firstboot.yaml" + + local firstboot_yaml="${LKP_SRC}/jobs/install-iso-firstboot.yaml" + [ -f "${firstboot_yaml}" ] || die "cannot find ${firstboot_yaml}" + + log_info " + ${LKP_SRC}/sbin/submit + ${firstboot_yaml} + suite=install-iso-firstboot-${ROOTFS_SUFFIX} + os=${iso_os} + os_arch=${iso_arch} + os_version=${iso_version} + os_mount=local + kernel_custom_params=\"use_root_partition=${ROOTFS_SUFFIX} save_root_partition=${ROOTFS_SUFFIX}\" + testbox=$testbox + " + + ${LKP_SRC}/sbin/submit \ + ${firstboot_yaml} \ + suite=install-iso-firstboot-${ROOTFS_SUFFIX} \ + os=${iso_os} \ + os_arch=${iso_arch} \ + os_version=${iso_version} \ + os_mount=local \ + kernel_custom_params="use_root_partition=${ROOTFS_SUFFIX} save_root_partition=${ROOTFS_SUFFIX}" \ + testbox=$testbox || die "submit ${firstboot_yaml} failed" } submit_install_iso_release_yaml() -- 2.23.0

2 4

[PATCH lkp-tests 09/11] install-iso-pre: add step: submit install-iso.yaml
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index d867f1441003..45567feddd9c 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -182,9 +182,38 @@ prepare_submit() EOF } +prepare_install_iso_yaml() +{ + log_info "start prepare install-iso.yaml" + + custom_ipxe="#!ipxe\\\\n" + custom_ipxe="${custom_ipxe}\\\\ndhcp" + custom_ipxe="${custom_ipxe}\\\\ninitrd tftp://${TFTP_SERVER_HOST}/EFI/BOOT/${iso_os}/${iso_arch}/${iso_version}/initrd.img" + custom_ipxe="${custom_ipxe}\\\\nkernel tftp://${TFTP_SERVER_HOST}/EFI/BOOT/${iso_os}/${iso_arch}/${iso_version}/vmlinux ro inst.geoloc=0 console=ttyS0 smmu.bypassdev=0x1000:0x17 smmu.bypassde1000:0x15 viff video=VGA-1:640x480-32@60me initrd=initrd.img ip=dhcp inst.ks=http://${HTTP_SERVER_HOST}:${HTTP_SERVER_PORT}/os/kickstart/ks/${iso_os}_${iso_arch}_${iso_version}.cfg" + custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" + + log_info "custom_ipxe: $custom_ipxe" + sed -i "/custom_ipxe:/ccustom_ipxe: \"${custom_ipxe}\"" ${install_iso_yaml} +} + submit_install_iso_yaml() { - : + log_info "start submit install-iso.yaml" + + local install_iso_yaml="${LKP_SRC}/jobs/install-iso.yaml" + [ -f "${install_iso_yaml}" ] || die "cannot find ${install_iso_yaml}" + + prepare_install_iso_yaml + + log_info " + ${LKP_SRC}/sbin/submit + ${install_iso_yaml} + testbox=$testbox + " + + ${LKP_SRC}/sbin/submit \ + ${install_iso_yaml} \ + testbox=$testbox || die "submit ${install_iso_yaml} failed" } submit_install_iso_firstboot_yaml() -- 2.23.0

2 2

[PATCH lkp-tests 07/11] install-iso-pre: add step: prepare_ks
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 72 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 71 insertions(+), 1 deletion(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 85e2a5fd42ec..e27c9ea80232 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -86,9 +86,79 @@ prepare_lv() create_lv } +generate_ks() +{ + log_info "start generate $ks_file" + + local nic=$(ip route get ${LKP_SERVER} | awk '{print $3; exit}') + + cat > ${ks_file} <<-EOF + # Use network installation + url --url="http://${HTTP_SERVER_HOST}:${HTTP_SERVER_PORT}/${HTTP_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version}/" + + # Network information + network --bootproto=dhcp --device=${nic} --ipv6=auto --activate + network --hostname=localhost.localdomain + + # Specifies a list of disks for the installation program to use. All other disks are ignored. + ignoredisk --only-use=${rootfs_disk#/dev/} + + # if install to special lvm, bootloader should be none. + bootloader --location=none + logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting + + # Root password + rootpw --iscrypted ${ROOTFS_ROOT_PASSWD} + + # Reboot after installation + reboot --eject + + # Use text mode install + text + + # Keyboard layouts + keyboard --vckeymap=us --xlayouts='us' + + # System language + lang en_US.UTF-8 + + # Do not configure the X Window System + skipx + + # System services + services --enabled="chronyd" + + # System timezone + timezone Asia/Shanghai --isUtc + + %packages + @^minimal-environment + + %end + + %anaconda + pwpolicy root --minlen=8 --minquality=1 --notstrict --nochanges --notempty + pwpolicy user --minlen=8 --minquality=1 --notstrict --nochanges --emptyok + pwpolicy luks --minlen=8 --minquality=1 --notstrict --nochanges --notempty + %end + EOF +} + +upload_ks() +{ + log_info "start upload ks to rsync-server: ${RSYNC_SERVER_HOST}:${RSYNC_SERVER_PORT}" + + rsync -vrtopg --port ${RSYNC_SERVER_PORT} ${ks_file} ${RSYNC_SERVER_HOST}::ks/ || die "upload ks to rsync-server failed, please check." +} + prepare_ks() { - : + log_info "start prepare kickstart file" + + local ks_file=/tmp/${iso_os}_${iso_arch}_${iso_version}.ks + + generate_ks + upload_ks } prepare_submit() -- 2.23.0

2 2

[PATCH lkp-tests 06/11] install-iso-pre: add step: prepare_lv
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-pre | 1 + tests/install-iso-pre | 43 +++++++++++++++++++++++++++++++++- 2 files changed, 43 insertions(+), 1 deletion(-) create mode 100644 distro/depends/install-iso-pre diff --git a/distro/depends/install-iso-pre b/distro/depends/install-iso-pre new file mode 100644 index 000000000000..6f88eb919896 --- /dev/null +++ b/distro/depends/install-iso-pre @@ -0,0 +1 @@ +lvm2 diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 4e8cbe22fcd4..85e2a5fd42ec 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -40,9 +40,50 @@ check_params() done } +create_vg() +{ + log_info "start create vg:$VG_NAME" + + local disk + + for disk in "${rootfs_disk[@]}" + do + log_info "start handle disk: $disk" + pvdisplay ${disk} > /dev/null || pvcreate ${disk} || die "create pv failed: $disk" + + if vgdisplay ${VG_NAME} > /dev/null; then + vgextend ${VG_NAME} ${disk} + else + vgcreate ${VG_NAME} ${disk} + fi + done + +} + +create_lv() +{ + log_info "start create lv:$LV_NAME" + + lvcreate -y -L 10G -n ${LV_NAME} ${VG_NAME} || die "create logical volume failed." +} + prepare_lv() { - : + log_info "start prepare logical volume" + + # VG_NAME defines in job.yaml + export VG_NAME + export LV_NAME="${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX}" + + # lv existed + lvdisplay /dev/mapper/${VG_NAME}-${LV_NAME} > /dev/null && { + log_info "iso already install to this testbox, so rootfs maybe already generated ok. please check it." + exit 0 + } + + # lv not existed + vgdisplay ${VG_NAME} > /dev/null || create_vg + create_lv } prepare_ks() -- 2.23.0

2 2

[PATCH lkp-tests 04/11] install-iso-release: add step: release_rootfs
by Yu Chuan 10 Mar '21

10 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-release | 2 ++ tests/install-iso-release | 53 +++++++++++++++++++++++++++++- 2 files changed, 54 insertions(+), 1 deletion(-) create mode 100644 distro/depends/install-iso-release diff --git a/distro/depends/install-iso-release b/distro/depends/install-iso-release new file mode 100644 index 000000000000..2abb6b833291 --- /dev/null +++ b/distro/depends/install-iso-release @@ -0,0 +1,2 @@ +nfs-common +lvm2 diff --git a/tests/install-iso-release b/tests/install-iso-release index 23e8fbff8e4f..8c925ee2bcd8 100755 --- a/tests/install-iso-release +++ b/tests/install-iso-release @@ -7,6 +7,20 @@ set -e . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/log.sh +SKIP_RELEASE_DIRS=( + "srv" + "opt" + "mnt" + "media" + "home" + "lost+found" + "dev" + "proc" + "run" + "sys" + "var/log" +) + check_params() { log_info "start check params" @@ -29,9 +43,46 @@ check_params() done } +prepare_exclude_file() +{ + local exclude_file=$1 + + rm -rf $exclude_file + for i in "${SKIP_RELEASE_DIRS[@]}" + do + echo $i >> $exclude_file + done +} + release_rootfs() { - : + # mount nfs rootfs + local nfs_rootfs_mount_point=/tmp/nfs_rootfs_${ROOTFS_SUFFIX} + + mkdir -p ${nfs_rootfs_mount_point} + mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" + + # activate the lvm + vgchange -ay + + # mount local logical volume + local lv_rootfs_mount_point=/tmp/lv_rootfs_${ROOTFS_SUFFIX} + + mkdir -p ${lv_rootfs_mount_point} + mount /dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} ${lv_rootfs_mount_point} || die "mount lv rootfs failed" + + # release rootfs + local exclude_file=/tmp/exclude_file_${ROOTFS_SUFFIX} + + prepare_exclude_file "$exclude_file" + rsync -az ${lv_rootfs_mount_point}/. ${nfs_rootfs_mount_point} --exclude-from=$exclude_file || die "copy out rootfs failed" + cd ${nfs_rootfs_mount_point} && mkdir -p "${SKIP_RELEASE_DIRS[@]}" + + # umount nfs rootfs + umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" + + # umount local logical volume + umount $lv_rootfs_mount_point || log_warn "umount failed: $lv_rootfs_mount_point" } main() -- 2.23.0

2 2

[PATCH compass-ci] lib/lifecycle.cr: optimize the restart queue
by Wu Zhende 10 Mar '21

10 Mar '21

1.delete machine's history don't need this info. 2.add testbox this field is used to find the corresponding host when a phssical machine is restarted. 3.modify mq queue testbox: vm-2p8g.wuzhende-1 before queue: wuzhende now queue: vm-2p8g.wuzhende Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 3dab6e8..1eb6264 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -110,7 +110,7 @@ class Lifecycle def mq_event_loop puts "deal job events" - q = @mq.ch.queue("job_mq") + q = @mq.ch.queue("job_mq", durable: false) q.subscribe(no_ack: false) do |msg| event = JSON.parse(msg.body_io.to_s) job_state = event["job_state"]? @@ -311,6 +311,8 @@ class Lifecycle return if Time.local < deadline mq_queue = get_machine_reboot_queue(testbox) + machine.as_h.delete("history") + machine.as_h["testbox"] = JSON::Any.new(testbox) @mq.pushlish_confirm(mq_queue, machine.to_json, durable: true) machine["state"] = "rebooting_queue" @@ -320,7 +322,7 @@ class Lifecycle def get_machine_reboot_queue(testbox) if testbox.includes?(".") - testbox =~ /.*\.(.*)-\d+$/ + testbox =~ /(.*)-\d+$/ else testbox =~ /(.*)--.*/ end -- 2.23.0

1 0

[PATCH lkp-tests 03/11] feat: step 3 of iso2rootfs: copy out rootfs in second boot stage
by Yu Chuan 10 Mar '21

10 Mar '21

Framework of install-iso-release. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso-release.yaml | 26 +++++++++++++++++++ tests/install-iso-release | 47 +++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+) create mode 100644 jobs/install-iso-release.yaml create mode 100755 tests/install-iso-release diff --git a/jobs/install-iso-release.yaml b/jobs/install-iso-release.yaml new file mode 100644 index 000000000000..fabb2cd50902 --- /dev/null +++ b/jobs/install-iso-release.yaml @@ -0,0 +1,26 @@ +suite: install-iso-release +category: benchmark +install-iso-release: + ################# + # iso info + ################# + iso_os: + iso_arch: + iso_version: + + ################# + # nfs server info, used for store rootfs + ################# + NFS_SERVER_HOST: + NFS_ROOT_DIR: + + ################# + # result rootfs suffix, format: $(date "+%Y%m%d%H%M%S") + ################# + ROOTFS_SUFFIX: + + ################# + # name of volume group which stores the logical volume of rootfs + ################# + VG_NAME: + diff --git a/tests/install-iso-release b/tests/install-iso-release new file mode 100755 index 000000000000..23e8fbff8e4f --- /dev/null +++ b/tests/install-iso-release @@ -0,0 +1,47 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +set -e + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/log.sh + +check_params() +{ + log_info "start check params" + + local required_vars=( + "iso_os" + "iso_arch" + "iso_version" + + "NFS_SERVER_HOST" + "NFS_ROOT_DIR" + + "ROOTFS_SUFFIX" + "VG_NAME" + ) + + for i in "${required_vars[@]}" + do + [ -n "$(eval echo \$$i)" ] || die "cannot get value of var: $i" + done +} + +release_rootfs() +{ + : +} + +main() +{ + check_params + + release_rootfs + + lvchange --permission r /dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} || + die "change logical volume permission to readonly failed:/dev/mapper/${VG_NAME}-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX}" +} + +main -- 2.23.0

2 2

[PATCH compass-ci] docker/run.sh: change hostname
by Wang Yong 10 Mar '21

10 Mar '21

[before] Build Host : dc-8g [after] Build Host : dc-8g.compass-ci.net this will change Build Host in RPM build Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 2810641..2dfe36d 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm - --hostname $host + --hostname $host.compass-ci.net -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -- 2.23.0

1 0