October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] fix: sched: set 2 fields to job: direct_macs, direct_ips
by Ren Wen 20 Oct '20

20 Oct '20

[Why] nodes need communicate with each other when running cluster jobs. after adding 2 fileds, 'direct_ips' and 'direct_macs', can bind ips to macs. 'macs' get from cluster file ('$LKP_SRC/cluster/*'), 'ips' set by scheduler default: 192.168.<211..254>.<host_id> Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a18bdee..1d5b5c4 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -198,6 +198,10 @@ class Sched # collect all job ids job_ids = [] of String + # used to set ips + ip_prefix = "192.168." + host_id = 1 + # steps for each host cluster_config.each do |host, config| tbox_group = host.to_s @@ -219,7 +223,18 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + + # host macs and host ips + direct_macs = config["macs"].as_a + direct_ips = [] of String + net_id = "#{ip_prefix}#{254 - job_id.hash%44}" # 192.168.<211..254> + direct_macs.size.times do + raise "more than 255 macs" if host_id > 255 + direct_ips << "#{net_id}.#{host_id}" + host_id += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

2 2

[PATCH compass-ci] /src/lib/web_backend.rb: fix "/compare_candidates" tbox_group regex bug
by Zhang Yuhang 20 Oct '20

20 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 48cfadb..fda01ca 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -47,7 +47,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end -- 2.23.0

2 2

[PATCH lab-z9] hosts: add new hostfile
by Zhang Yu 20 Oct '20

20 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a101 | 22 ++++++++++++++++++++++ hosts/taishan200-2280-2s48p-256g--a102 | 24 ++++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a101 create mode 100644 hosts/taishan200-2280-2s48p-256g--a102 diff --git a/hosts/taishan200-2280-2s48p-256g--a101 b/hosts/taishan200-2280-2s48p-256g--a101 new file mode 100644 index 0000000..1226913 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a101 @@ -0,0 +1,22 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b89192e1 + - /dev/disk/by-id/scsi-350000399b88bf66d + - /dev/disk/by-id/scsi-350000399b88bef85 +ssd_partitions: + - /dev/disk/by-id/ata-INTEL_SSDSC2KB480G8_BTYF94730FF4480BGN +mac_addr: + - 44:67:47:d7:6c:9c + - 44:67:47:d7:6c:9d + - 44:67:47:d7:6c:9e + - 44:67:47:d7:6c:9f + - 44:67:47:d7:6c:a0 + - 44:67:47:d7:6c:a1 + - 44:67:47:d7:6c:a2 + - 44:67:47:d7:6c:a3 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.6 diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-256g--a102 new file mode 100644 index 0000000..267a962 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a102 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 512G +nr_hdd_partitions: 2 +nr_ssd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88bfbc9 + - /dev/disk/by-id/scsi-350000399b8919395 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH1T9HMLT-00005_S455NY0MB55506 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH1T9HMLT-00005_S455NY0MB55187 + - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000335 + - nvme-nvme.19e5-30333242454a31304743303030333335-48574533365034333031324d3030304e-00000001-part* +mac_addr: + - 44:67:47:c9:ea:30 + - 44:67:47:c9:ea:31 + - 44:67:47:c9:ea:32 + - 44:67:47:c9:ea:33 + - 44:67:47:c9:ea:34 + - 44:67:47:c9:ea:35 + - 44:67:47:c9:ea:36 + - 44:67:47:c9:ea:37 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.7 -- 2.23.0

1 0

[PATCH compass-ci] src/lib: for multitask like multi-qemu
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the scheduler can not find the program multi-qemu-0 and the program multi-qemu-1 Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..e563ddf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,10 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + program_num = program.match(/(.*)(\-\d{1,}$)/) + if program_num != nil + program = program_num.not_nil![1] + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +416,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq!.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq!.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

3 3

[PATCH v2 compass-ci] fix: sched: set 2 fields to job: direct_macs, direct_ips
by Ren Wen 20 Oct '20

20 Oct '20

[Why] nodes need communicate with each other when running cluster jobs. after adding 2 fileds, 'direct_ips' and 'direct_macs', can bind ips to macs. 'macs' get from cluster file ('$LKP_SRC/cluster/*'), 'ips' set by scheduler default. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a18bdee..82d3610 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -198,6 +198,10 @@ class Sched # collect all job ids job_ids = [] of String + # used to set ips + ip_prefix = "192.169." + host_id = 1 + # steps for each host cluster_config.each do |host, config| tbox_group = host.to_s @@ -219,7 +223,17 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + + # host macs and host ips + direct_macs = config["macs"].as_a + direct_ips = [] of String + net_id = "#{ip_prefix}#{job_id.hash%255}" # 192.169.<0..254> + direct_macs.size.times do + direct_ips << "#{net_id}.#{host_id}" + host_id += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

1 1

[PATCH v7 compass-ci 2/6] container/mail-robot: answerback-email.rb
by Luan Shengde 20 Oct '20

20 Oct '20

AssignUuid: read email content check email available check if base url in upstream-repos check if commit url valid check pub_key assign account generate uuid write account/uuid to es send uuid to user Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/answerback-email.rb | 229 +++++++++++++++++++++++ 1 file changed, 229 insertions(+) create mode 100755 container/mail-robot/answerback-email.rb diff --git a/container/mail-robot/answerback-email.rb b/container/mail-robot/answerback-email.rb new file mode 100755 index 0000000..66d13e4 --- /dev/null +++ b/container/mail-robot/answerback-email.rb @@ -0,0 +1,229 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '/c/compass-ci/lib/es_client.rb' + +# Assign uuid/account +class AssignUuid + def initialize(mail_content) + @mail_content = mail_content + @send_mail_server = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + end + + def build_message(sum_infos) + message = <<~EMAIL_MESSAGE + To: #{sum_infos['my_email']} + Subject: Account Ready + + Dear #{sum_infos['my_name']} + + Thank you for joining us. + You can use the following info to submit jobs: + + First: add the info to ~/.config/compass-ci/defaults/{USER}.yaml + + uuid: #{sum_infos['my_uuid']} + lab: z9 + + Second: download lkp-tests and dependencies + git clone https://gitee.com_/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source {HOME}/.bashrc && source {HOME}/.bash_profile + + Third: submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + how to write job yaml + submit jobs/netperf.yaml + + lastly: add your own information and uuid to ~/.config/compass-ci/default/{USER}.yml + my_name: {your name} + my_email: {your email address} + my_uuid: {applied uuid} + + Repalce the fields {...} with your local environment variable. + + regards + compass-ci + EMAIL_MESSAGE + + return message + end + + def build_err_message(email, message) + message = <<~EMAIL_MESSAGE + To: #{email} + Subject: apply account failed + + Dear #{email.split('@')[0]} + + Your application for account failed with following errors + #{message} + + regards + compass-ci + EMAIL_MESSAGE + end + + def send_mail(email_msg) + %x(curl -XPOST "#{@send_mail_server}:#{ENV['S_M_PORT']}/send_mail_text" -d "#{email_msg}") + end + + def url_addr(email) + mail_body = @mail_content.part[0].body.decoded + mail_content = mail_body.gsub(/=/, '').split(/\r|\n|=/).join + url = %x(echo '#{mail_content}' |grep -o -E "https?://[^/]*/[^/]*/[^/]*/[^/]*/[0-9a-zA-Z]{40}").split(/\n/)[0] + + if url.nil? + message = 'No commit url found' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise message + end + + return url + end + + def check_email_available(email) + url = url_addr(email).chomp + + base_url = url.gsub(%r{/[^/]*/[0-9a-zA-Z]{40}$}, '') + + upstream_dir = '/c/upstream-repos' + check_url_in_upstream_repos(upstream_dir, base_url, email) + check_url_commit(url, base_url, email) + + return url + end + + def check_url_in_upstream_repos(upstream_dir, base_url, email) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + if match_out.empty? + message = 'The url is not in upstream-repo list' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise StandardError, message, base_url + end + end + + def check_url_commit(url, base_url, email) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + check_gitee_commit(url, base_url, email) if hub_name.eql? 'gitee.com' + check_non_gitee_commit(url, email) unless hub_name.eql? 'gitee.com' + end + + def check_non_gitee_commit(url, email) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index email + + if email_index.nil? + message = 'Your commit url is not a valid commit url.' + email_msg = build_err_message(email, message) + send_mail(email_msg) + raise StandardError, message, url + end + end + + def check_gitee_commit(url, base_url, email) + repo_info = { + 'repos_dir' => '/repos_dir', + 'repo_dir' => url.split('/')[-3], + 'repo_url' => [base_url, 'git'].join('.'), + 'repo_path' => File.join('/repos_dir', url.split('/')[-3]), + 'commit_id' => url.split('/')[-1] + } + check_gitee_repo_commit(url, email, repo_info) + end + + def check_gitee_repo_commit(url, email, repo_info) + Dir.chdir repo_info['repos_dir'] + %x(/usr/bin/git clone #{repo_info['repo_url']}) + email_index = %x(/usr/bin/git -C #{repo_info['repo_dir']} show #{repo_info['commit_id']}).index email + + FileUtils.rm_rf repo_info['repo_dir'] + + if email_index.nil? + message = 'Your commit url is not a valid commit url.' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise StandardError, message, url + end + end + + def pub_key_value(email) + pub_key = @mail_content.part[1].body.decoded.split(/\r|\n/).join if @mail_content.part[1].filename == 'id_rsa.pub' + + unless pub_key + message = 'No pub_key found' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise message + end + + return pub_key + end + + def account_info(pub_key, email) + account_info_str = %x(curl -XGET "#{ENV['J_HOST']}:#{ENV['J_PORT']}/assign_account" -d "pub_key: #{pub_key}") + account_info = JSON.parse account_info_str + + if account_info['account'].empty? + message = 'No more available jumper account.' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise StandardError, message, account_info if account_info['account'].empty? + end + + return account_info + end + + def build_sum_infos + email = @mail_content.from[0] + my_name = @mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + commit_url = check_email_available(email) + pub_key = pub_key_value(email) + acct_infos = account_info(pub_key, email) + account_uuid = %x(uuidgen).chomp + sum_infos = { + 'my_email' => email, + 'my_name' => my_name, + 'my_commit_url' => commit_url, + 'my_login_name' => acct_infos['account'], + 'my_uuid' => account_uuid, + } + store_account_info(sum_infos) + sum_infos['acct_infos'] = acct_infos + sum_infos + end + + def send_uuid(sum_infos) + email_msg = build_message(sum_infos) + + send_mail(email_msg) + end + + def send_account + sum_infos = build_sum_infos + + send_uuid(sum_infos) + end + + def store_account_info(sum_infos) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(sum_infos['my_email'], sum_infos) + end +end -- 2.23.0

2 4

[PATCH lkp-tests 1/2] jobs: add test-tools.yaml
by Li Ping 20 Oct '20

20 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- jobs/test-tools.yaml | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 jobs/test-tools.yaml diff --git a/jobs/test-tools.yaml b/jobs/test-tools.yaml new file mode 100644 index 00000000..153fefd0 --- /dev/null +++ b/jobs/test-tools.yaml @@ -0,0 +1,10 @@ +suite: test-tools +testcase: test-tools +category: functional + +os_mount: cifs +os_version: 20.03-dailybuild + +test-tools: + testsuite: + case: -- 2.23.0

2 1

[PATCH lkp-tests] bin: mkdir /etc/pki/ca-trust/source/anchors
by Hu Xuejiao 20 Oct '20

20 Oct '20

When run job-task, appeat warning: cannot create /etc/pki/ca-trust/source/anchors/docker_registry_proxy.crt: Directory nonexistent Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- bin/lkp-setup-rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index d1b17a9c..09930990 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -139,6 +139,7 @@ install_docker_proxy() Environment="HTTPS_PROXY=http://$DOCKER_PROXY_HOST:$DOCKER_PROXY_PORT/" EOF + mkdir -p /etc/pki/ca-trust/source/anchors curl http://$DOCKER_PROXY_HOST:$DOCKER_PROXY_PORT/ca.crt > \ /etc/pki/ca-trust/source/anchors/docker_registry_proxy.crt || return update-ca-trust extract -- 2.23.0

1 0

[PATCH lkp-tests] bin: update ~ to /root
by Hu Xuejiao 20 Oct '20

20 Oct '20

When run job-task, appear warning: /lkp/lkp/src/bin/lkp-setup-rootfs: 154: cannot create ~/.gemrc: Directory nonexistent Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- bin/lkp-setup-rootfs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 09f7e7eb..d1b17a9c 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -151,7 +151,7 @@ EOF modify_gem_source() { - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc + echo ':sources: ["http://rubygems.org"]' >> /root/.gemrc return 0 } -- 2.23.0

1 0

[PATCH compass-ci] fix: sched: set 2 fields to job: direct_macs, direct_ips
by Ren Wen 20 Oct '20

20 Oct '20

cluster nodes need communicate with each other, so set 2 fields: 1) direct_macs: get from cluster file ($LKP_SRC/cluster/*) 2) direct_ips: set by scheduler default (they're local ips) It will add ips to corresponding devices before nodes executing jobs. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a18bdee..62b7b9d 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -198,6 +198,11 @@ class Sched # collect all job ids job_ids = [] of String + # set net id, and host id + # ip = net_id + set_id + net_id = "192.168.211." + host_id = 1 + # steps for each host cluster_config.each do |host, config| tbox_group = host.to_s @@ -219,7 +224,16 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + + # host macs and host ips + direct_macs = config["macs"].as_a + direct_ips = [] of String + direct_macs.size.times do + direct_ips << "#{net_id}#{host_id}" + host_id += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

2 1