- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib: update port
by Luan Shengde 16 Mar '21

16 Mar '21

service assigned new port: send-internet-mail: 49000 => 10001 assign-account: 29999 => 10000 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 4 ++-- lib/constants.rb | 2 +- lib/mail-post.rb | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb index 4831545..f551a9d 100644 --- a/lib/assign_account_client.rb +++ b/lib/assign_account_client.rb @@ -21,9 +21,9 @@ names = Set.new %w[ defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] -JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 +JUMPER_PORT = defaults['JUMPER_PORT'] || 10000 SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 10001 LAB = ENV['lab'] # used for other codes calling to assign account for user diff --git a/lib/constants.rb b/lib/constants.rb index 6c30aaf..03b1e40 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -9,7 +9,7 @@ ES_HOST ||= config['ES_HOST'] || '172.17.0.1' ES_PORT ||= config['ES_PORT'] || 9200 SEND_MAIL_HOST ||= config['SEND_MAIL_HOST'] || '172.17.0.1' -SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 10001 SRV_HTTP_RESULT_HOST ||= config['SRV_HTTP_RESULT_HOST'] || ENV['SRV_HTTP_RESULT_HOST'] || '172.17.0.1' SRV_HTTP_OS_HOST ||= config['SRV_HTTP_OS_HOST'] || ENV['SRV_HTTP_OS_HOST'] || '172.17.0.1' diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 4eb5800..84fc3c1 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -76,7 +76,7 @@ def check_send_mail(mail_info) raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? - return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '10001' mail_info = check_email_mapping(mail_info.clone) mail_info = check_email_limit(mail_info.clone) @@ -101,7 +101,7 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' + return if ENV['SEND_MAIL_PORT'].to_s != '10001' return if ENV['HOST_SERVER'] != 'z9' store_email(mail) -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] container/web-backend-nginx: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend-nginx/nginx.conf | 4 ++-- container/web-backend-nginx/start | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/container/web-backend-nginx/nginx.conf b/container/web-backend-nginx/nginx.conf index c9a4ec4..381888a 100644 --- a/container/web-backend-nginx/nginx.conf +++ b/container/web-backend-nginx/nginx.conf @@ -8,7 +8,7 @@ http { default_type application/octet-stream; include /etc/nginx/conf.d/*.conf; server { - listen 11320 ssl; + listen 20003 ssl; server_name api.compass-ci.openeuler.org; ssl_certificate /opt/cert/web-backend.crt; ssl_certificate_key /opt/cert/web-backend.key; @@ -20,7 +20,7 @@ http { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forward-For $proxy_add_x_forwarded_for; # web-backend server - proxy_pass http://172.17.0.1:32767; + proxy_pass http://172.17.0.1:10002; } location ~ ^/(result|pub) { diff --git a/container/web-backend-nginx/start b/container/web-backend-nginx/start index 4da4c8b..5360385 100755 --- a/container/web-backend-nginx/start +++ b/container/web-backend-nginx/start @@ -10,7 +10,7 @@ docker_rm web-backend-nginx cmd=( docker run --name web-backend-nginx - -p 11320:11320 + -p 20003:20003 -d -v /etc/ssl/certs:/opt/cert -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/web-backend: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/start | 2 +- container/web-backend/web-backend | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/web-backend/start b/container/web-backend/start index 6731d27..e680255 100755 --- a/container/web-backend/start +++ b/container/web-backend/start @@ -18,7 +18,7 @@ cmd=( --name web-backend --restart=always -d - -p 32767:32767 + -p 10002:10002 -e ES_PORT=9200 -e SRV_HTTP_RESULT_HOST=$SRV_HTTP_RESULT_HOST diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 74eda85..33a3fae 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -9,7 +9,7 @@ CCI_SRC = ENV['CCI_SRC'] || '/c/compass-ci' require "#{CCI_SRC}/src/lib/web_backend.rb" set :bind, '0.0.0.0' -set :port, 32767 +set :port, 10002 set :show_exceptions, false set :environment, :production -- 2.23.0

1 0

[PATCH lkp-tests] lib/email.sh: change default send email port
by Wei Jihui 16 Mar '21

16 Mar '21

49000 --> 10001 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/email.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/email.sh b/lib/email.sh index a0c99c24..63d5a51e 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -87,5 +87,5 @@ send_email() ${subject}_content - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$email_content" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-10001}/send_mail_text" -d "$email_content" } -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 16 Mar '21

16 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 81 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..bf75b19b7 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,81 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] || { + echo "Ansible failed to install." + exit 1 + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git + if [ $? -eq 0 ]; then + echo "$benchmark is successfully cloned" + else + echo "Please check whether your ansible_repo is correct!" + exit 1 + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook is running failed!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

2 3

[PATCH lkp-tests] stat/ansible_test: stats result for ansible_test
by Wang Chenglong 16 Mar '21

16 Mar '21

The result of ansible_test are counted and recorded in the ES. query it with the 'es-jobs' command. sum.stats.ansible_test.error.Unsupported-pkg-mgr-This-role-is-depends-on-apt-yum-or-pkgng: 1 raw.stats.ansible_test.error.Unsupported-pkg-mgr-This-role-is-depends-on-apt-yum-or-pkgng.message: ["{\"changed\": false, \"msg\": \"Unsupported pkg mgr. This role is depends on apt, yum, or pkgng.\"}"] Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 55 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 44 insertions(+), 11 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..38d6e278c 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,48 @@ #!/usr/bin/env ruby -while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line +require 'json' + +def output_error(error_msg) + return if error_msg.nil? + + if error_msg.is_a? Array + error_msg.each do |i| + error_id = common_error_id i + puts "error.#{error_id}: 1" + puts "error.#{error_id}.message: #{@ansible_failed_info}" + end + else + error_id = common_error_id error_msg + puts "error.#{error_id}: 1" + puts "error.#{error_id}.message: #{@ansible_failed_info}" end end + +def common_error_id(line) + line.gsub!(/\n|\t/, '.') + line =`echo "#{line}"|awk -F'.' '{print $1}'` + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/\^/, '') + line.gsub!(/\.|,/, '') + line.gsub!(/<|>/, '') # error.Request-failed-<urlopen-error-timed-out>: 1 + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/ /, '-') + line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one + line = line.chomp + line +end + +while (line = STDIN.gets) + next unless line =~ /(FAILED!|failed:).*=>(.*)/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.parse @ansible_failed_info + + output_error ansible_failed_json['msg'] + output_error ansible_failed_json['message'] + output_error ansible_failed_json['cmd'] + output_error ansible_failed_json['failures'] +end -- 2.23.0

2 2

[PATCH compass-ci] dev-env: update port
by Luan Shengde 16 Mar '21

16 Mar '21

send-internet-mail has been assigned new port: 49000 -> 10001 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index b740a15..e448dcb 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -10,7 +10,7 @@ server_name=$(hostname | cut -f1 -d.) : ${ES_PORT:=9200} : ${OS_HTTP_HOST:=$server_ip} : ${SEND_MAIL_HOST:=$server_ip} -: ${SEND_MAIL_PORT:=49000} +: ${SEND_MAIL_PORT:=10001} : ${SRV_HTTP_RESULT_HOST:=$server_ip} : ${SRV_HTTP_OS_HOST:=$server_ip} : ${SRV_HTTP_GIT_HOST:=$server_ip} -- 2.23.0

1 0

[PATCH v2 compass-ci] src/lib/job.cr: change rpm_repo to upstream_repo
by Li Ping 16 Mar '21

16 Mar '21

src/lib/job.cr: open the path for uploading rpm packages. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9e18a4c..40aaa62 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -278,7 +278,7 @@ class Job mount_type = "nfs" if mount_type == "cifs" common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") + common_dir = "#{tmp_os}-#{tmp_os_version}" if @hash.has_key?("rpmbuild-pkg") return common_dir end @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 1

[PATCH v2 compass-ci] src/lib/job.cr: open the path for uploading rpm packages.
by Li Ping 16 Mar '21

16 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 9e18a4c..40aaa62 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -278,7 +278,7 @@ class Job mount_type = "nfs" if mount_type == "cifs" common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") + common_dir = "#{tmp_os}-#{tmp_os_version}" if @hash.has_key?("rpmbuild-pkg") return common_dir end @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/compare-results.zh.md: translate to Chinese
by Lin Jiaxin 16 Mar '21

16 Mar '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- doc/manual/compare-results.zh.md | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/doc/manual/compare-results.zh.md b/doc/manual/compare-results.zh.md index 1b43d6b..b96add9 100644 --- a/doc/manual/compare-results.zh.md +++ b/doc/manual/compare-results.zh.md @@ -1,29 +1,26 @@ -# Introduction of compare feature - The compare feature used to analyze the results of different jobs, showing the performance waves and changes - under different influencing factors, for users to analyze performance influencing factor. +# 比较功能介绍 + 比较功能用于分析不同任务的结果，显示不同影响因素下的性能波动和变化，供用户分析性能影响因素。 ## url: https://compass-ci.openeuler.org/compare -## options directions -### filters: +## 选项说明 +### 过滤器: - suite: iperf, netperf, mysql, ... - OS: openeuler 20.03, centos 7.6, ... - os_arch: aarch64, x86 - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... - we can combine the above options arbitrarily to limit compare scope. - choose at least one option as filter. + 我们可以任意组合以上选项来限制比较范围，但至少选择一个选项作为过滤器。 -### dimension - Dimension can select: os, os_version, os_arch, suite, tbox_group. - Within filter, we will compare all different job result by dimension - and keep other test conditions are same. +### 维度 + 可选维度: os, os_version, os_arch, suite, tbox_group. + 在过滤器中，我们将按所选维度比较所有不同的任务结果，并保持其他测试条件一致。 -## example: - filter: suite = iperf - dimension: os_version +## 示例: + 过滤器: suite = iperf + 维度: os_version - result: - os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + 结果: + os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # 其他测试条件一致 20.09 20.03 metric -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/web-backend-nginx: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend-nginx/nginx.conf | 2 +- container/web-backend-nginx/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/web-backend-nginx/nginx.conf b/container/web-backend-nginx/nginx.conf index c9a4ec4..50ced4c 100644 --- a/container/web-backend-nginx/nginx.conf +++ b/container/web-backend-nginx/nginx.conf @@ -8,7 +8,7 @@ http { default_type application/octet-stream; include /etc/nginx/conf.d/*.conf; server { - listen 11320 ssl; + listen 20003 ssl; server_name api.compass-ci.openeuler.org; ssl_certificate /opt/cert/web-backend.crt; ssl_certificate_key /opt/cert/web-backend.key; diff --git a/container/web-backend-nginx/start b/container/web-backend-nginx/start index 4da4c8b..5360385 100755 --- a/container/web-backend-nginx/start +++ b/container/web-backend-nginx/start @@ -10,7 +10,7 @@ docker_rm web-backend-nginx cmd=( docker run --name web-backend-nginx - -p 11320:11320 + -p 20003:20003 -d -v /etc/ssl/certs:/opt/cert -- 2.23.0

1 1

[PATCH compass-ci 1/2] container/web-backend: change service port
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/start | 2 +- container/web-backend/web-backend | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/web-backend/start b/container/web-backend/start index 6731d27..e680255 100755 --- a/container/web-backend/start +++ b/container/web-backend/start @@ -18,7 +18,7 @@ cmd=( --name web-backend --restart=always -d - -p 32767:32767 + -p 10002:10002 -e ES_PORT=9200 -e SRV_HTTP_RESULT_HOST=$SRV_HTTP_RESULT_HOST diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 74eda85..33a3fae 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -9,7 +9,7 @@ CCI_SRC = ENV['CCI_SRC'] || '/c/compass-ci' require "#{CCI_SRC}/src/lib/web_backend.rb" set :bind, '0.0.0.0' -set :port, 32767 +set :port, 10002 set :show_exceptions, false set :environment, :production -- 2.23.0

1 1

[PATCH compass-ci 2/2] answerback-email: update tool with new port
by Luan Shengde 16 Mar '21

16 Mar '21

assign-account service has assigned new port: 29999 => 10000 send-internet-mail service has assigned new port: 49000 => 10001 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index f573f0f..f4b6ef3 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -28,9 +28,9 @@ names = Set.new %w[ defaults = relevant_defaults(names) JUMPER_HOST ||= defaults['JUMPER_HOST'] -JUMPER_PORT ||= defaults['JUMPER_PORT'] || 29999 +JUMPER_PORT ||= defaults['JUMPER_PORT'] || 10000 SEND_MAIL_HOST ||= defaults['SEND_MAIL_HOST'] || 'localhost' -SEND_MAIL_PORT ||= defaults['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_PORT ||= defaults['SEND_MAIL_PORT'] || 10001 LAB ||= defaults['lab'] TOOL_SERVER = 'z9' -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/assign-account: update for new port
by Luan Shengde 16 Mar '21

16 Mar '21

the assign-account servive has been assigned new port: 29999 => 10000 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 2 +- container/assign-account/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index c5d77db..abbab0f 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -10,7 +10,7 @@ require 'yaml' require_relative 'get_account_info' set :bind, '0.0.0.0' -set :port, 29999 +set :port, 10000 post '/assign_account' do begin diff --git a/container/assign-account/start b/container/assign-account/start index 793191a..b926853 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -14,7 +14,7 @@ cmd=( -v /etc:/etc:rw -v /srv/account-info:/opt/account_data/:rw -v /home:/home:rw - -p 29999:29999 + -p 10000:10000 assign-account ) -- 2.23.0

1 0

[PATCH compass-ci] constants: change service default ports
by Cao Xueliang 16 Mar '21

16 Mar '21

Reference: 6354ba631e0fd5f795a1dc28b65e2002bd3de668 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/constants.rb | 6 +++--- src/scheduler/constants.cr | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index 8d8e488..6c30aaf 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -14,9 +14,9 @@ SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_RESULT_HOST ||= config['SRV_HTTP_RESULT_HOST'] || ENV['SRV_HTTP_RESULT_HOST'] || '172.17.0.1' SRV_HTTP_OS_HOST ||= config['SRV_HTTP_OS_HOST'] || ENV['SRV_HTTP_OS_HOST'] || '172.17.0.1' SRV_HTTP_GIT_HOST ||= config['SRV_HTTP_GIT_HOST'] || ENV['SRV_HTTP_GIT_HOST'] || '172.17.0.1' -SRV_HTTP_RESULT_PORT ||= config['SRV_HTTP_RESULT_PORT'] || ENV['SRV_HTTP_RESULT_PORT'] || 11300 -SRV_HTTP_OS_PORT ||= config['SRV_HTTP_OS_PORT'] || ENV['SRV_HTTP_OS_PORT'] || 11304 -SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || 11305 +SRV_HTTP_RESULT_PORT ||= config['SRV_HTTP_RESULT_PORT'] || ENV['SRV_HTTP_RESULT_PORT'] || 20007 +SRV_HTTP_OS_PORT ||= config['SRV_HTTP_OS_PORT'] || ENV['SRV_HTTP_OS_PORT'] || 20009 +SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || 20010 SRV_HTTP_DOMAIN ||= config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index a5e52b8..d18c88d 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -26,7 +26,7 @@ OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1 OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 SRV_HTTP_CCI_HOST = (ENV.has_key?("SRV_HTTP_CCI_HOST") ? ENV["SRV_HTTP_CCI_HOST"] : "172.17.0.1") -SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11307") +SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : 20011).to_i32 CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") LAB_REPO = "lab-z9" -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: change service ports
by Lu Weitao 16 Mar '21

16 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/start-cci | 2 +- container/srv-http/start-git | 2 +- container/srv-http/start-initrd | 2 +- container/srv-http/start-os | 2 +- container/srv-http/start-pub | 2 +- container/srv-http/start-result | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci index 8486476..00d8e4c 100755 --- a/container/srv-http/start-cci +++ b/container/srv-http/start-cci @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-cci - -p 11307:11300 + -p 20011:11300 -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-git b/container/srv-http/start-git index f2292bf..c1181a7 100755 --- a/container/srv-http/start-git +++ b/container/srv-http/start-git @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-git - -p 11305:11300 + -p 20010:11300 -v /srv/git/archlinux:/srv/git/archlinux:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 495cc25..466ec1b 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-initrd - -p 11303:11300 + -p 20008:11300 -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro diff --git a/container/srv-http/start-os b/container/srv-http/start-os index bb344cf..d39e04e 100755 --- a/container/srv-http/start-os +++ b/container/srv-http/start-os @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-os - -p 11304:11300 + -p 20009:11300 -v /srv/os:/srv/os:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-pub b/container/srv-http/start-pub index dc1864c..f42b117 100755 --- a/container/srv-http/start-pub +++ b/container/srv-http/start-pub @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-pub - -p 11302:11300 + -p 20006:11300 -v /srv/pub:/srv/pub:ro -v /etc/localtime:/etc/localtime:ro -d diff --git a/container/srv-http/start-result b/container/srv-http/start-result index f9ecc97..bf0287d 100755 --- a/container/srv-http/start-result +++ b/container/srv-http/start-result @@ -9,7 +9,7 @@ cmd=( docker run --restart=always --name srv-http-result - -p 11300:11300 + -p 20007:11300 -v /srv/result:/srv/result:ro -v /etc/localtime:/etc/localtime:ro -d -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: open the path for uploading rpm packages.
by Li Ping 16 Mar '21

16 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index f8cfa97..40aaa62 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -278,7 +278,7 @@ class Job mount_type = "nfs" if mount_type == "cifs" common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") + common_dir = "#{tmp_os}-#{tmp_os_version}" if @hash.has_key?("rpmbuild-pkg") return common_dir end @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["upstream_repo"].to_s.split("/")[-1]}" + package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 16 Mar '21

16 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 83 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..9ecb4d2c5 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,83 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + echo $ansible_host + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible +} + +install_ansible() +{ + if [ "$os" == "openeuler" ]; then + yum -y reinstall git + yum -y install python3 + pip3 install ansible + elif [ "$os" == "centos" ]; then + yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + yum -y reinstall git + yum -y install ansible + elif [ "$os" == "debian" ]; then + apt-get install -y ansible + fi + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] || { + echo "Ansible failed to install." + exit 1 + } + } +} + +download_ansible_role() +{ + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone git://github.com/$ansible_repo.git + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + exit 1 + fi +} + +ansible_playbook_run() +{ + cat <<-EOF > "$ansible_path"/site.yml + - hosts: $ansible_host + roles: + - role: $benchmark + EOF + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml + + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "Your playbook is running failed!" && exit 1 + fi +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run +} + +main -- 2.23.0

1 1

[PATCH v4 compass-ci 2/2] multi-docker: add reboot function
by Wu Zhende 16 Mar '21

16 Mar '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 34 +++++++++++++++++++++++++++++++++- providers/docker/run.sh | 2 ++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index f4e954e..9d6854f 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -7,6 +7,8 @@ require 'open-uri' require 'json' require 'set' require 'fileutils' + +require '../lib/mq_client' require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' @@ -19,6 +21,8 @@ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 LOG_DIR = '/srv/cci/serial/logs' +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || 'localhost' +MQ_PORT = ENV['MQ_PORT'] || 5672 def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -84,7 +88,8 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, + { 'job_id' => hash['job_id'], + 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path, 'log_dir' => "#{LOG_DIR}/#{hostname}" @@ -145,6 +150,29 @@ def loop_main(hostname, queues) end end +def loop_reboot_docker(hostname) + loop do + begin + reboot_docker(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_docker(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) + queue = mq.queue(hostname, {:durable => true}) + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + puts msg + machine_info = JSON.parse(msg) + job_id = machine_info['job_id'] + system "docker rm -f #{job_id};echo $?" + mq.ack(info) + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -153,6 +181,10 @@ def save_pid(pids) end def multi_docker(hostname, nr_container, queues) + Process.fork do + loop_reboot_docker(hostname) + end + pids = [] nr_container.to_i.times do |i| pid = Process.fork do diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 2dfe36d..80950dd 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -4,6 +4,7 @@ . $LKP_SRC/lib/yaml.sh +: ${job_id:=$$} : ${docker_image:="centos:7"} : ${load_path:="${HOME}/jobs"} : ${hostname:="dc-1g-1"} @@ -23,6 +24,7 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm + --name ${job_id} --hostname $host.compass-ci.net -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 16 Mar '21

16 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH v4 compass-ci] providers/multi-qemu: optimize
by Wu Zhende 16 Mar '21

16 Mar '21

1. optimize function name 2. the host and port are transferred when the MQClient is initalized 3. reboot vm cyclically Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 5979cc9..d3f8406 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -104,8 +104,19 @@ def save_pid(arr) f.close end -def reboot_deadline_machine(hostname) - mq = MQClient.new +def loop_reboot_vm(hostname) + loop do + begin + reboot_vm(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_vm(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) queue = mq.queue(hostname, {:durable => true}) queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| puts msg @@ -118,10 +129,7 @@ end def multiqemu Process.fork do - reboot_deadline_machine(HOSTNAME) - rescue StandardError => e - puts e.backtrace - sleep 5 + loop_reboot_vm(HOSTNAME) end pids = [] -- 2.23.0

1 0

[PATCH compass-ci] container/webhook: change the service port
by Li Yuanchao 15 Mar '21

15 Mar '21

For the port policy. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/webhook/start b/container/webhook/start index 373396a..1d121a4 100755 --- a/container/webhook/start +++ b/container/webhook/start @@ -12,7 +12,7 @@ cmd = %w[ --restart=always --name webhook -d - -p 11301:11301 + -p 20005:20005 node:webhook ] -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: change the local git repo path
by Lin Jiaxin 15 Mar '21

15 Mar '21

As the path of repo mirrors under /srv/git changes, introduce upstream_dir to get new path. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 43932865..350d8fdf 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -6,6 +6,7 @@ # - pkg_ver # - config # upstream_url +# upstream_dir # We'll use the below global env vars, but not treat them as parameters to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ @@ -123,9 +124,9 @@ replace_source() do echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { - url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" + url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#commit=${upstream_commit}" [ -n "$git_tag" ] && { - url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#tag=${git_tag}" + url="${repo_dir}git://${LKP_SERVER}/${upstream_dir}/${upstream_repo}#tag=${git_tag}" } } } -- 2.23.0

1 0

[PATCH compass-ci 1/2] sbin/auto_submit: change the upstream_repo path to submit
by Lin Jiaxin 15 Mar '21

15 Mar '21

Reference: 5d6cea1bd531409f0444abf14088fdecd8b08c2d Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index faebb2f..a27075b 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -33,7 +33,7 @@ class AutoSubmit git_repo = newrefs_info['git_repo'] puts "git_repo : #{git_repo}" url = Array(newrefs_info['url'])[0] - submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{git_repo} upstream_url=#{url}"] + submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{newrefs_info['belong']}/#{git_repo} upstream_url=#{url}"] if newrefs_info['pkgbuild_repo'] pkgbuild_repo = get_pkgbuild_repo(newrefs_info['pkgbuild_repo']) -- 2.23.0

2 2

[PATCH compass-ci] optimize doc
by Luan Shengde 15 Mar '21

15 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-tests-container.zh.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/manual/build-lkp-tests-container.zh.md b/doc/manual/build-lkp-tests-container.zh.md index 10a1611..407ddbf 100644 --- a/doc/manual/build-lkp-tests-container.zh.md +++ b/doc/manual/build-lkp-tests-container.zh.md @@ -60,4 +60,4 @@ submit命令： - 了解更多submit命令的使用方法，请参考：[submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… + 了解更多submit命令的使用方法，请参考：[submit用户手册](https://gitee.com/wu_fengguang/compass-c… -- 2.23.0

1 0