- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci 2/2] doc: add job/rootfs_disk.md
by Yu Chuan 19 Feb '21

19 Feb '21

Add description of rootfs_disk field in job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/rootfs_disk.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 doc/job/rootfs_disk.md diff --git a/doc/job/rootfs_disk.md b/doc/job/rootfs_disk.md new file mode 100644 index 000000000000..18d165ca1e08 --- /dev/null +++ b/doc/job/rootfs_disk.md @@ -0,0 +1,27 @@ +# Summary +========= + +`rootfs_disk` defines a list of disks, these disks will be combined into one volume group(os), +this volume group will have multiple logical volume, storing multiple rootfs, each of these +logical volume corresponds to a rootfs, such as follow: + /dev/mapper/ + . + ├── os-openeuler_aarch64_20.03 + ├── os-openeuler_aarch64_20.03_{timestamp} + ├── os-openeuler_aarch64_20.03_sp1 + ├── os-openeuler_aarch64_20.03_sp1_{timestamp} + └── ... + +Addition: +1. `/dev/mapper/os-{os}_{os_arch}_{os_version}_{timestamp}` + - is a readonly logical volume + - used for backup and rollback + +2. `/dev/mapper/os-{os}_{os_arch}_{os_version}` + - is a read write logical volume + - use for boot and run job + - defaultly, this logical volume will be wiped before every job run + +3. every logical volume default size: 10G + +4. more info can refer os_mount.md -- 2.23.0

2 2

[PATCH compass-ci 2/4] container/mail-robot: add handle forwarded my_name/my_email(s)
by Luan Shengde 19 Feb '21

19 Feb '21

[why] for forwarded email from security admins, the email content will contain my_name/my_email(s) waiting to be assigned an account. a forwarded email may contain multi my_name/my_email(s), just loop them. add store lab to ES Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 49 ++++++++++++++++++++--- 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index e8288d7..bc340ad 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -54,6 +54,8 @@ class ApplyAccount @mail_content = mail_content @es_host = @send_mail_host @es_port = ES_PORT + # email address file for security admins. + @sec_admins = YAML.load_file '/etc/compass-ci/conf/sec_admins.yaml' @my_info = {} end @@ -63,11 +65,31 @@ class ApplyAccount # in order to successfully send email for failed parsing # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] - parse_mail_content - apply_my_account - store_account_info - send_mail('') + # for the forwarded email, we will get the email/name from the mail_content + # all emails from security admins will be treated as forwarded emails. + # the mail_content may like: + # --- + # my_name_1: my_email_1 + # my_name_2: my_email_2 + # --- + # the forwarded email allowed to contain multi emails/names + # in this case, we will loop them + if @sec_admins.include? @my_info['my_email'] + users = forward_users + users.each do |user| + @my_info.clear + next if user.match?(/---/) + + @my_info['my_name'], @my_info['my_email'] = user.split(/:\s+/) + @my_info['my_ssh_pubkey'] = [] + applying_account + sleep 10 + end + else + parse_mail_content + applying_account + end rescue StandardError => e puts e.message puts e.backtrace @@ -75,6 +97,17 @@ class ApplyAccount send_mail(e.message) end + def forward_users + forward_email_content = ParseApplyAccountEmail.new(@mail_content) + forward_email_content.extract_users + end + + def applying_account + apply_my_account + store_account_info + send_mail('') + end + def parse_mail_content parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) @@ -93,7 +126,9 @@ class ApplyAccount apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? apply_info.update my_account_es apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + if my_ssh_pubkey_new + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + end @my_info.update apply_info apply_info['is_update_account'] = true apply_info @@ -102,6 +137,9 @@ class ApplyAccount def apply_my_account my_account_es = read_my_account_es apply_info = {} + + # lab will also be stored to ES. + @my_info['lab'] = ENV['lab'] if my_account_es build_apply_info(apply_info, my_account_es) else @@ -109,7 +147,6 @@ class ApplyAccount @my_info['my_token'] = my_token apply_info.update @my_info end - apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end -- 2.23.0

2 7

[PATCH compass-ci 1/4] container/mail-robot: optimize regular express
by Luan Shengde 19 Feb '21

19 Feb '21

[why] the mail robot will enable forwarded emails from security admins. the mail subject may 'apply account', 'Fw: apply account', .etc. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/mail-robot.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb index a809eae..c3bc943 100755 --- a/container/mail-robot/mail-robot.rb +++ b/container/mail-robot/mail-robot.rb @@ -32,7 +32,7 @@ def handle_new_email(mail_file, mail_drafts) end def apply_account(mail_content) - return unless mail_content.subject == 'apply account' + return unless mail_content.subject.match?(/apply account/i) assign_uuid = ApplyAccount.new(mail_content) assign_uuid.check_to_send_account -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 18 Feb '21

18 Feb '21

[How] ansible_stats_openeuler < ${job_output} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 95 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 85 insertions(+), 10 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..bfb0c2f11 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,90 @@ #!/usr/bin/env ruby +require 'json' + while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line + line = line.gsub(/'/, ' ') + ansible_stats_error = `echo '#{line}' | grep -E 'FAILED! |failed: .*}$' | awk -F'=>' '{$1="";print $0}'` + next if ansible_stats_error.empty? + + a = JSON.load(ansible_stats_error) + unless a['msg'].nil? + case a['msg'].chomp + + # When running some plug-in, the file was not found + when /No file was found when using (.+)\./ + puts "files.error.no-file-was-found-when-using.#{$1}: 1" + puts "files.error.no-file-was-found-when-using.#{$1}.message: #{ansible_stats_error}" + + # No such command + when /No such file or directory: b (.+) :/ + puts "command.error.no-such-file-or-directory.#{$1}: 1" + puts "command.error.no-such-file-or-directory.#{a['cmd']}.message: #{ansible_stats_error}" + + # The URL of the RPM is invalid + when /Failure downloading (.+), / + puts "download.error.http-error-404 [#{$1}]: 1" + puts "download.error.http-error-404 [#{$1}]: #{ansible_stats_error}" + + # Failed to download the yum repodata + when /Failed to download metadata for repo (.+) :/ + puts "download.error.failed-to-download-metadata.#{$1}: 1" + puts "download.error.failed-to-download-metadata.#{$1}.message: #{ansible_stats_error}" + + when /Failed to download packages: Cannot download (.+):/ + puts "download.error.failed-to-download-package.#{$1}: 1" + puts "download.error.failed-to-download-package.#{$1}.message: #{ansible_stats_error}" + + # The task includes an option with an undefined variable + when /: (.+) is undefined/ + puts "syntax.error.an-undefined-variable.#{$1}: 1" + puts "syntax.error.an-undefined-variable.#{$1}.message: #{ansible_stats_error}" + + # The command runing error + when /non-zero return code/ + puts "command.error[#{a['cmd']}]: 1" + puts "command.error [#{a['cmd']}].message: #{ansible_stats_error}" + + # Some service could not find the requested host + when /Could not find the requested service (.+): host/ + puts "request.error.could-not-find-service-host.#{$1}: 1" + puts "request.error.could-not-find-service-host.#{$1}.messgae: #{ansible_stats_error}" + + # Unable to start some services + when /Unable to start service (.+):/ + puts "service.error.unable-to-start-service.#{$1}: 1" + puts "service.error.unable-to-start-service.#{$1}.message: #{ansible_stats_error}" + + # Some role in not sipported this system + when /(.+) is not supported/ + puts "system.error.not-supported.#{$1.gsub(/ /, '-')}: 1" + puts "system.error.not-supported.message: #{ansible_stats_error}" + + # failed to create temporary content file: The read operation timed out. + when /The read operation timed out/ + puts 'connection.error.read-timed-out: 1' + puts "connection.error.read-timed-out.message: #{ansible_stats_error}" + end + end + + unless a['failures'].nil? + a['failures'].each do |items| + case items.chomp + # Failed to install some of the specified packages. + when /No package (.+) available/ + puts "yum.error.no-package-match.#{$1}: 1" + puts "yum.error.no-package-match.#{$1}.message: #{line}" + end + end + end + + next if a['message'].nil? + + case a['message'].chomp + + # Could not find or access ansible yaml + when /Could not find or access (.+) / + puts "yaml.error.not-find-yml.#{$1}: 1" + puts "yaml.error.not-find-yml.#{$1}.message: #{ansible_stats_error}" end end -- 2.23.0

2 1

[PATCH v2 compass-ci] git_mirror.rb: fix that the priority with priority queue may be negative
by Li Yuanchao 18 Feb '21

18 Feb '21

Priority queue use the priority to decide which one pop first, and small priority one would be pop first. And the more active repo should be pop first, so priority of repo would minus its new ref times to make its priority small. But some times, the result can be negative, this make the repo delayed a lot. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 940d5e2..39b1619 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,7 +203,8 @@ class MirrorMain fork_key = @priority_queue.delete_min_return_key do_push(fork_key) - @priority_queue.push fork_key, (@priority - @fork_stat[fork_key][:priority]) + priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 + @priority_queue.push fork_key, priority_set @priority += 1 end -- 2.23.0

2 2

[PATCH compass-ci] rsync-server: add support upload function for ks file
by Yu Chuan 18 Feb '21

18 Feb '21

[Why] We need to achieve automatic installation through 'pxe + kickstart + dnsmasq + httpd'. So we need to rsync the ks file by rsync-server. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/rsync-server/rsyncd.conf | 6 ++++++ container/rsync-server/start | 1 + 2 files changed, 7 insertions(+) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index cdba89cf20c8..ef00acb1bc0c 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -10,3 +10,9 @@ path = /srv/result/upload/ read only = no max connections = 200 refuse options = delete + +[ks] +path = /srv/os/kickstart/ks/ +read only = no +max connections = 200 +refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 956013268f89..ecd783a22dd2 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -11,6 +11,7 @@ cmd=( --restart=always --name=rsync_server -v /srv/result/upload:/srv/result/upload + -v /srv/os/kickstart/ks:/srv/os/kickstart/ks -v /etc/localtime:/etc/localtime:ro -p 11387:11387 rsync_server:latest -- 2.23.0

2 6

[PATCH v2 lkp-tests] lib/job.rb: delete read "/etc/compass-ci/defaults/*.yaml"
by Cao Xueliang 18 Feb '21

18 Feb '21

The compass-ci/scheduler will read the yamls, so, delete it in lkp-tests Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/job.rb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index eacb7ec95..37c538753 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -418,8 +418,7 @@ class Job def load_self_config self_config_path = "#{ENV['HOME']}/.config/compass-ci" - Dir.glob(['/etc/compass-ci/defaults/*.yaml', - "#{self_config_path}/defaults/*.yaml"]).each do |file| + Dir.glob(["#{self_config_path}/defaults/*.yaml"]).each do |file| load_one_defaults(file, @job) end -- 2.23.0

2 2

[PATCH lkp-tests] daemon: add nginx-sever
by Wei Jihui 18 Feb '21

18 Feb '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/nginx-server | 173 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100755 daemon/nginx-server diff --git a/daemon/nginx-server b/daemon/nginx-server new file mode 100755 index 00000000..a7dd1033 --- /dev/null +++ b/daemon/nginx-server @@ -0,0 +1,173 @@ +#!/bin/sh +# - nics +# - nics_ips +# - core_nums +# - protocol +# - alive_type + +. $LKP_SRC/lib/common.sh + +# To manually check CPU usage when clients sent http/https requests, please use htop tool. +# htop tool download url: http://rpmfind.net/linux/epel/8/Everything/aarch64/Packages/h/htop-2.2.0-6.… +# For send http request, can use httpress tool. +# httpress tool download url: http://rpmfind.net/linux/epel/7/aarch64/Packages/h/httpress-1.1.0-2.el7.aar… + +: "${protocol:=https}" # http or https +: "${alive_type:=short}" # short or long + +nics=(${nics//,/ }) +nics_ips=(${nics_ips//,/ }) +[ -n "$BENCHMARK_ROOT" ] || BENCHMARK_ROOT=/lkp/benchmarks + +check_nginx_install() +{ + if [ -x "$nginx_path" ]; then + pkill -f "$nginx_path" + else + echo "nginx is not installed" >&2 && exit 1 + fi + + if [ ! -f "$conf_dir/$protocol/basic.conf" ]; then + echo "Could not find nginx conf" >&2 && exit 1 + fi +} + +prepare() +{ + setenforce 0 + ulimit -SHn 409600 + export LD_PRELOAD=/usr/lib64/libtcmalloc.so.4 + + # require by $LKP_SRC/bin/set_nic_irq_affinity + pkill -9 irqbalance +} + +bind_core_net_irp() +{ + # binding cores are used to process NIC interrupts, each corresponds to a NIC. + # the interrupt cores in each CPU are bound only to the NIC that belong to the CPU. + # the CPU cores are bound by node. + + ip addr add "$nic_ip"/24 dev "$nic" + ip link set "$nic" up + + ethtool -L $nic combined $bind_core_nums + $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic +} + +update_nginx_conf() +{ + update_key=$1 + update_value="$(eval echo '$'$update_key)" + sed -i "s|{{$update_key}}|$update_value|" "$conf_path" +} + +start_nginx() +{ + local ssl_certificate_path="$conf_dir/$protocol/server_2048.crt" + local ssl_certificate_key_path="$conf_dir/$protocol/server_2048.key" + local conf_path="$conf_dir/$protocol.$alive_type.$core_range.conf" + cp -f "$conf_dir/$protocol/basic.conf" "$conf_path" + source "$conf_dir/$protocol/$alive_type/$core_range" + + update_nginx_conf html_index + update_nginx_conf listen_port + update_nginx_conf worker_processes + update_nginx_conf worker_cpu_affinity + [ "$protocol" == 'https' ] && { + update_nginx_conf ssl_certificate + update_nginx_conf ssl_certificate_key + } + + "$nginx_path" -c "$conf_path" -p "$BENCHMARK_ROOT/nginx-server/" +} + +assign_core_ranges() +{ + if [ "$protocol" == 'http' ]; then + case "$core_nums" in + "4") + if [ "$alive_type" == 'long' ]; then + core_ranges='3' + else + core_ranges='2-3' + fi + ;; + "8") + if [ "$alive_type" == 'long' ]; then + core_ranges='6-7' + else + core_ranges='2-3 6-7' + fi + ;; + "48") + if [ "$alive_type" == 'long' ]; then + core_ranges='18-23 42-47' + else + core_ranges='12-17 18-23 36-41 42-47' + fi + ;; + "96") + if [ "$alive_type" == 'long' ]; then + core_ranges='18-23 42-47 66-71 90-95' + else + core_ranges='0-15 16-31 48-63 64-79' + fi + ;; + *) echo "The enter core num:$core_nums is incorrect" && exit 1 + ;; + esac + else + case "$core_nums" in + "4") core_ranges='3' + ;; + "8") + if [ "$alive_type" == 'long' ]; then + core_ranges='6-7' + else + core_ranges='7' + fi + ;; + "48") + if [ "$alive_type" == 'long' ]; then + core_ranges='20-23 44-47' + else + core_ranges='23 47' + fi + ;; + "96") + if [ "$alive_type" == 'long' ]; then + core_ranges='20-23 44-47 68-71 92-95' + else + core_ranges='46-47 94-95' + fi + ;; + *) echo "The enter core num:$core_nums is incorrect" && exit 1 + ;; + esac + fi +} + +main() +{ + local core_ranges + local nginx_path="$BENCHMARK_ROOT/nginx-server/sbin/nginx" + local conf_dir="$BENCHMARK_ROOT/nginx-server/nginx_conf" + + check_nginx_install + prepare + assign_core_ranges + + index=0 + for core_range in $core_ranges + do + local nic=${nics[$index]} + local nic_ip=${nics_ips[$index]} + local bind_core_nums=$(cpu_list_num $core_range) + bind_core_net_irp + start_nginx + index=$((index+1)) + done +} + +main -- 2.23.0

1 0

[PATCH compass-ci] sbin/compare: fix No Method Error
by Lu Weitao 18 Feb '21

18 Feb '21

[Why] use: compare group_id=fio-basic-openeuler-acr-1301864-1,fio-basic-openeuler-acr-1301864-2,fio-basic-openeuler-acr-1301864-3 -d pp.acr3.acr error message: Traceback (most recent call last): 2: from /home/wangyong/compass-ci/sbin/compare:72:in `<main>' 1: from /home/wangyong/compass-ci/lib/compare.rb:66:in `compare_group' /home/lwt/compass-ci/lib/compare_matrixes.rb:400:in `compare_group_matrices': undefined method `[]' for nil:NilClass (NoMethodError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 4 ++++ lib/matrix2.rb | 2 ++ 2 files changed, 6 insertions(+) diff --git a/lib/compare.rb b/lib/compare.rb index 21e24df..7c182c4 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -63,6 +63,10 @@ def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') groups_matrices, suites_hash, latest_jobs_hash = create_groups_matrices_list(conditions, dims) + unless groups_matrices + warn 'Empty group matrices!' + exit + end compare_group_matrices(groups_matrices, suites_hash, latest_jobs_hash, options) end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ab92986..5e0ad21 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -128,6 +128,8 @@ def combine_group_query_data(job_list, dims) end get_groups_matrix(groups, group_key, value, suites_hash, latest_jobs_hash) end + return nil if groups.empty? + return groups, suites_hash, latest_jobs_hash end -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: fix that the priority with priority queue may be negative
by Li Yuanchao 18 Feb '21

18 Feb '21

Priority queue use the priority to decide which one pop first, and small priority first. And the more active repo should be pop first, so priority of repo would minus its new ref times to make its priority small. But some times, the result can be negative, this make the repo delayed a lot. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 940d5e2..39b1619 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,7 +203,8 @@ class MirrorMain fork_key = @priority_queue.delete_min_return_key do_push(fork_key) - @priority_queue.push fork_key, (@priority - @fork_stat[fork_key][:priority]) + priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 + @priority_queue.push fork_key, priority_set @priority += 1 end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] sparrow/dev-env: add the local-testbox-env.yaml
by Cao Xueliang 18 Feb '21

18 Feb '21

The scheduler will read local-testbox-env.yaml, then add the content to the job yaml. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8812839..31a12db 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -59,6 +59,23 @@ ES_PORT: $ES_PORT SRV_HTTP_PORT: $SRV_HTTP_PORT EOF +cat > /etc/compass-ci/defaults/local-testbox-env.yaml <<-EOF +LKP_SERVER: $server_ip +GIT_SERVER: $GIT_SERVER +INITRD_HTTP_HOST: $initrd_http_host +INITRD_HTTP_PORT: $initrd_http_port +GITCACHE_HOST: $gitcache_host +GITCACHE_PORT: $gitcache_port +SEND_MAIL_HOST: $server_ip +SEND_MAIL_PORT: $SEND_MAIL_PORT +ASSIST_RESULT_HOST: $server_ip +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +RESULT_WEBDAV_HOST: $server_ip +RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT +SRV_HTTP_HOST: $server_ip +SRV_HTTP_PORT: $SRV_HTTP_PORT +EOF + cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] scheduler: add testbox env in scheduler
by Cao Xueliang 18 Feb '21

18 Feb '21

We add the testbox env use the submit script in lkp-tests, but the submit don't need to do that, so, we add the testbox env in scheduler. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 2 ++ src/scheduler/testbox_env.cr | 17 +++++++++++++++++ 2 files changed, 19 insertions(+) create mode 100644 src/scheduler/testbox_env.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index b724b07..8f0dbab 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "scheduler/testbox_env.cr" require "../scheduler/elasticsearch_client" require "./json_logger" @@ -52,6 +53,7 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @hash.merge!(testbox_env) @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new diff --git a/src/scheduler/testbox_env.cr b/src/scheduler/testbox_env.cr new file mode 100644 index 0000000..a378e2a --- /dev/null +++ b/src/scheduler/testbox_env.cr @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + def testbox_env(flag = "local") + hash = Hash(String, JSON::Any).new + + yaml_any = File.open("/etc/compass-ci/defaults/#{flag}-testbox-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + hash.merge!(Hash(String, JSON::Any).from_json(yaml_any.to_json)) + + hash + end +end -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 18 Feb '21

18 Feb '21

[How] ansible_stats_openeuler < ${job_output} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 100 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 90 insertions(+), 10 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..f31771e5c 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,95 @@ #!/usr/bin/env ruby +require 'json' + while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line + line = line.gsub(/'/, ' ') + ansible_stats_error = `echo '#{line}' |grep -E 'FAILED! |failed: .*}$' | awk -F'=>' '{$1="";print $0}'` + next if ansible_stats_error.empty? + + a = JSON.load(ansible_stats_error) + unless a['msg'].nil? + + # When running some plug-in, the file was not found + if a['msg'].match(/No file was found when using (.+)\./) + puts "files.error.no-file-was-found-when-using.#{$1}: 1" + puts "files.error.no-file-was-found-when-using.#{$1}.message: #{ansible_stats_error}" + end + + # No such command + if a['msg'].match(/No such file or directory: b (.+) :/) + puts "command.error.no-such-file-or-directory.#{$1}: 1" + puts "command.error.no-such-file-or-directory.#{a['cmd']}.message: #{ansible_stats_error}" + end + + # The URL of the RPM is invalid + if a['msg'].match(/Failure downloading (.+), /) + puts "download.error.http-error-404 [#{$1}]: 1" + puts "download.error.http-error-404 [#{$1}]: #{ansible_stats_error}" + end + + # Failed to download the yum repodata + if a['msg'].match(/Failed to download metadata for repo (.+) :/) + puts "download.error.failed-to-download-metadata.#{$1}: 1" + puts "download.error.failed-to-download-metadata.#{$1}.message: #{ansible_stats_error}" + end + + if a['msg'].match(/Failed to download packages: Cannot download (.+):/) + puts "download.error.failed-to-download-package.#{$1}: 1" + puts "download.error.failed-to-download-package.#{$1}.message: #{ansible_stats_error}" + end + + # The task includes an option with an undefined variable. + if a['msg'].match(/: (.+) is undefined/) + puts "syntax.error.an-undefined-variable.#{$1}: 1" + puts "syntax.error.an-undefined-variable.#{$1}.message: #{ansible_stats_error}" + end + + # The command runing error. + if a['msg'].match(/non-zero return code/) + puts "command.error[#{a['cmd']}]: 1" + puts "command.error [#{a['cmd']}].message: #{ansible_stats_error}" + end + + # Some service could not fand the requested host. + if a['msg'].match(/Could not find the requested service (.+): host/) + puts "request.error.could-not-find-service-host.#{$1}: 1" + puts "request.error.could-not-find-service-host.#{$1}.messgae: #{ansible_stats_error}" + end + + # Unable to start some services + if a['msg'].match(/Unable to start service (.+):/) + puts "service.error.unable-to-star-service.#{$1}: 1" + puts "service.error.unable-to-star-service.#{$1}.message: #{ansible_stats_error}" + end + + # Some role in not sipported this system + if a['msg'].match(/(.+) is not supported/) + puts "system.error.not-supported.#{$1.gsub(/ /, '-')}: 1" + puts "system.error.not-supported.message: #{ansible_stats_error}" + end + + # failed to create temporary content file: The read operation timed out. + if a['msg'].match(/The read operation timed out/) + puts 'connection.error.read-timed-out: 1' + puts "connection.error.read-timed-out.message: #{ansible_stats_error}" + end + end + + unless a['failures'].nil? + a['failures'].each do |items| + # Failed to install some of the specified packages. + items.match(/No package (.+) available/) + puts "yum.error.no-package-match.#{$1}: 1" + puts "yum.error.no-package-match.#{$1}.message: #{line}" + end + end + + next if a['message'].nil? + + # Could not find or access ansible yaml. + if a['message'].to_s.match(/Could not find or access (.+) /) + puts "yaml.error.not-find-yml.#{$1}: 1" + puts "yaml.error.not-find-yml.#{$1}.message: #{ansible_stats_error}" end end -- 2.23.0

3 4

[PATCH lkp-tests] setup/fs: support much values of hdd_partitions parameter
by Cui Yili 10 Feb '21

10 Feb '21

hdd_partitions: /dev/vda /dev/vdb as above, setup/fs only deal with /dev/vda. Signed-off-by: Cui Yili <2268260388(a)qq.com> --- setup/fs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/fs b/setup/fs index 23eaa3b85..824c86614 100755 --- a/setup/fs +++ b/setup/fs @@ -159,7 +159,7 @@ mount_fs() { if is_btrfs_raid_levels; then bdevs="${partitions%% *}" else - bdevs="$partitions" + bdevs="$hdd_partitions" fi fs_options -- 2.23.0

2 1

[PATCH v3 lkp-tests 2/2] tests/cci-makepkg: download submodules from remote end
by Liu Shaofei 10 Feb '21

10 Feb '21

[why] When building software package by cci-makepkg, the source codes repository contains submodules. By default, the submodules cannot be downloaded from Z9/Crystal during building packages. [how] By appending --norevisedomain param, disable submodules to be downloaded from Z9/Crystal. Just download submodules from remote end. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/cci-makepkg | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index ecbf72134..9c2ddc257 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -134,7 +134,7 @@ update_softlink() build_source_pkg() { local mark="true" - local pkg_args="-A --check --skippgpcheck" + local pkg_args="-A --check --skippgpcheck --norevisedomain" [ "${DISTRO}" == "archlinux" ] && { local mark="pacman" local pkg_args="${pkg_args} -s --needed --noconfirm" @@ -142,7 +142,11 @@ build_source_pkg() local makepkg="$LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf" echo "${makepkg}" - PACMAN="${mark}" BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + PACMAN="${mark}" \ + BUILDDIR="$TMP" \ + CARCH="$arch" \ + PKGEXT=".cgz" \ + CGZDEST="$sync_dest/$cgz_name" $makepkg [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] sbin/makepkg: fix whether to change domain of submodules
by Liu Shaofei 10 Feb '21

10 Feb '21

the param "--norevisedomain" indicates that all of submodules will not be changed domian name, and downloaded from remote end. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/makepkg | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 7a92ea71b..6b5836251 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -89,6 +89,7 @@ SIGNPKG='' SPLITPKG=0 SOURCEONLY=0 VERIFYSOURCE=0 +NOREVISEDOMAIN=0 # Forces the pkgver of the current PKGBUILD. Used by the fakeroot call # when dealing with svn/cvs/etc PKGBUILDs. @@ -587,6 +588,16 @@ download_git() { fi } +# modify submodules address to local server address. +# (( NOREVISEDOMAIN )) is true, not replace submodules domain, and download it from remote end. +modify_gitmodules() { + (( NOREVISEDOMAIN )) && return + + if [[ -f ".gitmodules" ]] && [[ -n ${GIT_SERVER} ]]; then + git config --system url."git://${GIT_SERVER}/".insteadOf https:// + fi +} + extract_git() { local netfile=$1 @@ -623,10 +634,7 @@ extract_git() { cd_safe "${dir##*/}" - # modify submodules address to local server address. - if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then - echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig - fi + modify_gitmodules local ref=origin/HEAD if [[ -n $fragment ]]; then @@ -3421,6 +3429,7 @@ usage() { printf -- "$(gettext " --skipinteg Do not perform any verification checks on source files")\n" printf -- "$(gettext " --skippgpcheck Do not verify source files with PGP signatures")\n" printf -- "$(gettext " --verifysource Download source files (if needed) and perform integrity checks")\n" + printf -- "$(gettext " --norevisedomain Do not change domain name when downloading submodules repository")\n" echo printf -- "$(gettext "These options can be passed to %s:")\n" "pacman" echo @@ -3462,7 +3471,7 @@ OPT_LONG=('allsource' 'check' 'clean' 'cleanbuild' 'config:' 'force' 'geninteg' 'help' 'holdver' 'ignorearch' 'install' 'key:' 'log' 'noarchive' 'nobuild' 'nocolor' 'nocheck' 'nodeps' 'noextract' 'noprepare' 'nosign' 'pkg:' 'repackage' 'rmdeps' 'sign' 'skipchecksums' 'skipinteg' 'skippgpcheck' 'source' 'syncdeps' - 'verifysource' 'version') + 'verifysource' 'version' 'norevisedomain') # Pacman Options OPT_LONG+=('asdeps' 'noconfirm' 'needed' 'noprogressbar') @@ -3514,6 +3523,7 @@ while true; do -s|--syncdeps) DEP_BIN=1 ;; -S|--source) SOURCEONLY=1 ;; --verifysource) VERIFYSOURCE=1 ;; + --norevisedomain) NOREVISEDOMAIN=1 ;; -h|--help) usage; exit 0 ;; # E_OK -V|--version) version; exit 0 ;; # E_OK -- 2.23.0

1 0

[PATCH compass-ci] container/defconfig.rb: optimize skip rebuild image judge
by Wei Jihui 10 Feb '21

10 Feb '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/defconfig.rb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index aa7d2fa..602b498 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -49,8 +49,7 @@ def get_available_memory end def docker_skip_rebuild(tag) - return if ENV['action'].nil? - return unless ENV['action'].include? 'run-only' + return if ENV['action'] != 'run-only' exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" end -- 2.23.0

1 0

[PATCH v2 lkp-tests] setup/fs: support much values of hdd_partitions parameter
by Cui Yili 10 Feb '21

10 Feb '21

error: hdd_partitions: /dev/vda /dev/vdb as above, setup/fs only deal with /dev/vda. /dev/vdb did not recognize. Signed-off-by: Cui Yili <2268260388(a)qq.com> --- setup/fs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/fs b/setup/fs index 23eaa3b85..824c86614 100755 --- a/setup/fs +++ b/setup/fs @@ -159,7 +159,7 @@ mount_fs() { if is_btrfs_raid_levels; then bdevs="${partitions%% *}" else - bdevs="$partitions" + bdevs="$hdd_partitions" fi fs_options -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] sbin/makepkg: download submodules from local server by default
by Liu Shaofei 10 Feb '21

10 Feb '21

[why] When building software package by cci-makepkg, the source codes repository contains submodules. By default, these submodules are downloaded from the local server, but the local server does not have submodules download address. So the submodules repository fails to be downloaded to test machine during the build. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/makepkg | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 7a92ea71b..b592a2ccb 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -587,6 +587,17 @@ download_git() { fi } +# modify submodules address to local server address. +# [ "$DISABLE_DOMAIN_CHANGE" == "false" ] => replace submodules domain, and download it from local server +# [ "$DISABLE_DOMAIN_CHANGE" == "true" ] => not replace submodules domain, and download it from remote end. +modify_gitmodules() { + [ "$DISABLE_DOMAIN_CHANGE" == "false" ] || return + + if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then + echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig + fi +} + extract_git() { local netfile=$1 @@ -623,10 +634,7 @@ extract_git() { cd_safe "${dir##*/}" - # modify submodules address to local server address. - if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then - echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig - fi + modify_gitmodules local ref=origin/HEAD if [[ -n $fragment ]]; then @@ -3668,6 +3676,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +DISABLE_DOMAIN_CHANGE=${DISABLE_DOMAIN_CHANGE:-false} if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then -- 2.23.0

2 3

[PATCH v2 lkp-tests 2/2] tests/cci-makepkg: download submodules from remote end
by Liu Shaofei 10 Feb '21

10 Feb '21

DISABLE_DOMAIN_CHANGE parameter value is true, "sbin/makepkg" file will not replace domain name of submodules, the submodules will be downloaded from remote end. Signed-off-by: Liu Shaofei <370072077(a)qq.com> i# Your branch is ahead of 'origin/master' by 1 commit. --- tests/cci-makepkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index ecbf72134..8c744d807 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -142,7 +142,7 @@ build_source_pkg() local makepkg="$LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf" echo "${makepkg}" - PACMAN="${mark}" BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + PACMAN="${mark}" DISABLE_DOMAIN_CHANGE=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

2 2

[PATCH compass-ci] src/scheduler/kernel_params.cr: no rootovl for local mount
by Xu Xijian 09 Feb '21

09 Feb '21

If the os_mount is "local", there is no need of overlay and it should be writable, so in the kernel params, delete "rootovl" for it and change "ro" to "rw" for it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index ed64bea..6fbe405 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -3,7 +3,9 @@ class Job private def kernel_common_params - return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" + common_params = "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job ip=dhcp" + return "#{common_params} rw" if "#{self.os_mount}" == "local" + return "#{common_params} rootovl ro" end private def kernel_custom_params -- 2.23.0

1 0

[PATCH compass-ci] container/defconfig.rb: fix undefined method `includes' for nil:NilClass
by Xu Xijian 09 Feb '21

09 Feb '21

[Why] When "ENV['action']" is nil, it will output error in this line in defconfig.rb: return unless ENV['action'].include? 'run-only' [errlog] Traceback (most recent call last): 1: from ./build:15:in `<main>' /home/xxj/compass-ci/container/defconfig.rb:52:in `docker_skip_rebuild': undefined method `include?' for nil:NilClass (NoMethodError) Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/defconfig.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 7dc9f33..aa7d2fa 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -49,6 +49,7 @@ def get_available_memory end def docker_skip_rebuild(tag) + return if ENV['action'].nil? return unless ENV['action'].include? 'run-only' exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: delete redundant wait
by Liu Yinsi 09 Feb '21

09 Feb '21

[why] if `&` is followed by `wait` is invalid to parallel run, in function do_one_build(), already has `wait`, so delete it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 1 - 1 file changed, 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 4fcbac7..e7309fd 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -70,7 +70,6 @@ do [ "$container_name" == 'ssh-r' ] && continue build_depends $dir & - wait start_depends $dir done -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/4-docker/buildall: delete redundant wait
by Liu Yinsi 09 Feb '21

09 Feb '21

[why] if `&` is followed by `wait` is invalid to parallel run, in function do_one_build(), already has `wait`and `&`, so delete it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a4dab4b..96c7385 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -67,8 +67,7 @@ tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir & - wait + build_depends $dir start_depends $dir done -- 2.23.0

1 0

[PATCH compass-ci] qemu/kvm.sh: kvm support virtual disk
by Cui Yili 09 Feb '21

09 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- providers/qemu/kvm.sh | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 68b9a62..4be33b9 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -97,6 +97,26 @@ set_helper() [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper } +add_disk() +{ + [ -d "$mount_points" ] || mount_points=$(pwd) + + [ -n "$hdd_partitions" ] && { + local unix_time=$(date +%s) + + local index=0 + for disk in $hdd_partitions + do + qcow2_file="${mount_points}/${unix_time}${disk##*/}.qcow2" + drive="file=${qcow2_file},media=disk,format=qcow2,index=${index}" + ((index++)) + + qemu-img create -fq qcow2 "${qcow2_file}" 512G + kvm+=(-drive ${drive}) + done + } +} + set_nic() { nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" @@ -134,7 +154,7 @@ print_message() log_info initrds: $initrds log_info append: $append [ "$DEBUG" == "true" ] || log_info less $log_file - + sleep 5 } @@ -228,6 +248,7 @@ set_options print_message public_option +add_disk individual_option run_qemu -- 2.23.0

1 0

2024

2023

2022

2021

2020

Compass-ci