- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] distro/depends/rpmbuild-pkg: add depend pkg
by Wei Jihui 19 Feb '21

19 Feb '21

[Error] yum-builddep: command not found [How] %yum provides yum-builddep dnf-plugins-core-4.0.11-5.oe1.noarch : Core Plugins for DNF Repo : @System Matched from: Filename : /usr/bin/yum-builddep Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/rpmbuild-pkg | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/rpmbuild-pkg b/distro/depends/rpmbuild-pkg index 6f00a6a2..50fd9b75 100644 --- a/distro/depends/rpmbuild-pkg +++ b/distro/depends/rpmbuild-pkg @@ -1,2 +1,3 @@ build-essential rpmdevtools +dnf-plugins-core -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] doc: add job/rootfs_disk.md
by Yu Chuan 19 Feb '21

19 Feb '21

Add description of rootfs_disk field in job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/rootfs_disk.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 doc/job/rootfs_disk.md diff --git a/doc/job/rootfs_disk.md b/doc/job/rootfs_disk.md new file mode 100644 index 000000000000..18d165ca1e08 --- /dev/null +++ b/doc/job/rootfs_disk.md @@ -0,0 +1,27 @@ +# Summary +========= + +`rootfs_disk` defines a list of disks, these disks will be combined into one volume group(os), +this volume group will have multiple logical volume, storing multiple rootfs, each of these +logical volume corresponds to a rootfs, such as follow: + /dev/mapper/ + . + ├── os-openeuler_aarch64_20.03 + ├── os-openeuler_aarch64_20.03_{timestamp} + ├── os-openeuler_aarch64_20.03_sp1 + ├── os-openeuler_aarch64_20.03_sp1_{timestamp} + └── ... + +Addition: +1. `/dev/mapper/os-{os}_{os_arch}_{os_version}_{timestamp}` + - is a readonly logical volume + - used for backup and rollback + +2. `/dev/mapper/os-{os}_{os_arch}_{os_version}` + - is a read write logical volume + - use for boot and run job + - defaultly, this logical volume will be wiped before every job run + +3. every logical volume default size: 10G + +4. more info can refer os_mount.md -- 2.23.0

2 2

[PATCH compass-ci 2/4] container/mail-robot: add handle forwarded my_name/my_email(s)
by Luan Shengde 19 Feb '21

19 Feb '21

[why] for forwarded email from security admins, the email content will contain my_name/my_email(s) waiting to be assigned an account. a forwarded email may contain multi my_name/my_email(s), just loop them. add store lab to ES Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 49 ++++++++++++++++++++--- 1 file changed, 43 insertions(+), 6 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index e8288d7..bc340ad 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -54,6 +54,8 @@ class ApplyAccount @mail_content = mail_content @es_host = @send_mail_host @es_port = ES_PORT + # email address file for security admins. + @sec_admins = YAML.load_file '/etc/compass-ci/conf/sec_admins.yaml' @my_info = {} end @@ -63,11 +65,31 @@ class ApplyAccount # in order to successfully send email for failed parsing # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] - parse_mail_content - apply_my_account - store_account_info - send_mail('') + # for the forwarded email, we will get the email/name from the mail_content + # all emails from security admins will be treated as forwarded emails. + # the mail_content may like: + # --- + # my_name_1: my_email_1 + # my_name_2: my_email_2 + # --- + # the forwarded email allowed to contain multi emails/names + # in this case, we will loop them + if @sec_admins.include? @my_info['my_email'] + users = forward_users + users.each do |user| + @my_info.clear + next if user.match?(/---/) + + @my_info['my_name'], @my_info['my_email'] = user.split(/:\s+/) + @my_info['my_ssh_pubkey'] = [] + applying_account + sleep 10 + end + else + parse_mail_content + applying_account + end rescue StandardError => e puts e.message puts e.backtrace @@ -75,6 +97,17 @@ class ApplyAccount send_mail(e.message) end + def forward_users + forward_email_content = ParseApplyAccountEmail.new(@mail_content) + forward_email_content.extract_users + end + + def applying_account + apply_my_account + store_account_info + send_mail('') + end + def parse_mail_content parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) @@ -93,7 +126,9 @@ class ApplyAccount apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? apply_info.update my_account_es apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + if my_ssh_pubkey_new + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + end @my_info.update apply_info apply_info['is_update_account'] = true apply_info @@ -102,6 +137,9 @@ class ApplyAccount def apply_my_account my_account_es = read_my_account_es apply_info = {} + + # lab will also be stored to ES. + @my_info['lab'] = ENV['lab'] if my_account_es build_apply_info(apply_info, my_account_es) else @@ -109,7 +147,6 @@ class ApplyAccount @my_info['my_token'] = my_token apply_info.update @my_info end - apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end -- 2.23.0

2 7

[PATCH compass-ci 1/4] container/mail-robot: optimize regular express
by Luan Shengde 19 Feb '21

19 Feb '21

[why] the mail robot will enable forwarded emails from security admins. the mail subject may 'apply account', 'Fw: apply account', .etc. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/mail-robot.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb index a809eae..c3bc943 100755 --- a/container/mail-robot/mail-robot.rb +++ b/container/mail-robot/mail-robot.rb @@ -32,7 +32,7 @@ def handle_new_email(mail_file, mail_drafts) end def apply_account(mail_content) - return unless mail_content.subject == 'apply account' + return unless mail_content.subject.match?(/apply account/i) assign_uuid = ApplyAccount.new(mail_content) assign_uuid.check_to_send_account -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 18 Feb '21

18 Feb '21

[How] ansible_stats_openeuler < ${job_output} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 95 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 85 insertions(+), 10 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..bfb0c2f11 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,90 @@ #!/usr/bin/env ruby +require 'json' + while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line + line = line.gsub(/'/, ' ') + ansible_stats_error = `echo '#{line}' | grep -E 'FAILED! |failed: .*}$' | awk -F'=>' '{$1="";print $0}'` + next if ansible_stats_error.empty? + + a = JSON.load(ansible_stats_error) + unless a['msg'].nil? + case a['msg'].chomp + + # When running some plug-in, the file was not found + when /No file was found when using (.+)\./ + puts "files.error.no-file-was-found-when-using.#{$1}: 1" + puts "files.error.no-file-was-found-when-using.#{$1}.message: #{ansible_stats_error}" + + # No such command + when /No such file or directory: b (.+) :/ + puts "command.error.no-such-file-or-directory.#{$1}: 1" + puts "command.error.no-such-file-or-directory.#{a['cmd']}.message: #{ansible_stats_error}" + + # The URL of the RPM is invalid + when /Failure downloading (.+), / + puts "download.error.http-error-404 [#{$1}]: 1" + puts "download.error.http-error-404 [#{$1}]: #{ansible_stats_error}" + + # Failed to download the yum repodata + when /Failed to download metadata for repo (.+) :/ + puts "download.error.failed-to-download-metadata.#{$1}: 1" + puts "download.error.failed-to-download-metadata.#{$1}.message: #{ansible_stats_error}" + + when /Failed to download packages: Cannot download (.+):/ + puts "download.error.failed-to-download-package.#{$1}: 1" + puts "download.error.failed-to-download-package.#{$1}.message: #{ansible_stats_error}" + + # The task includes an option with an undefined variable + when /: (.+) is undefined/ + puts "syntax.error.an-undefined-variable.#{$1}: 1" + puts "syntax.error.an-undefined-variable.#{$1}.message: #{ansible_stats_error}" + + # The command runing error + when /non-zero return code/ + puts "command.error[#{a['cmd']}]: 1" + puts "command.error [#{a['cmd']}].message: #{ansible_stats_error}" + + # Some service could not find the requested host + when /Could not find the requested service (.+): host/ + puts "request.error.could-not-find-service-host.#{$1}: 1" + puts "request.error.could-not-find-service-host.#{$1}.messgae: #{ansible_stats_error}" + + # Unable to start some services + when /Unable to start service (.+):/ + puts "service.error.unable-to-start-service.#{$1}: 1" + puts "service.error.unable-to-start-service.#{$1}.message: #{ansible_stats_error}" + + # Some role in not sipported this system + when /(.+) is not supported/ + puts "system.error.not-supported.#{$1.gsub(/ /, '-')}: 1" + puts "system.error.not-supported.message: #{ansible_stats_error}" + + # failed to create temporary content file: The read operation timed out. + when /The read operation timed out/ + puts 'connection.error.read-timed-out: 1' + puts "connection.error.read-timed-out.message: #{ansible_stats_error}" + end + end + + unless a['failures'].nil? + a['failures'].each do |items| + case items.chomp + # Failed to install some of the specified packages. + when /No package (.+) available/ + puts "yum.error.no-package-match.#{$1}: 1" + puts "yum.error.no-package-match.#{$1}.message: #{line}" + end + end + end + + next if a['message'].nil? + + case a['message'].chomp + + # Could not find or access ansible yaml + when /Could not find or access (.+) / + puts "yaml.error.not-find-yml.#{$1}: 1" + puts "yaml.error.not-find-yml.#{$1}.message: #{ansible_stats_error}" end end -- 2.23.0

2 1

[PATCH v2 compass-ci] git_mirror.rb: fix that the priority with priority queue may be negative
by Li Yuanchao 18 Feb '21

18 Feb '21

Priority queue use the priority to decide which one pop first, and small priority one would be pop first. And the more active repo should be pop first, so priority of repo would minus its new ref times to make its priority small. But some times, the result can be negative, this make the repo delayed a lot. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 940d5e2..39b1619 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,7 +203,8 @@ class MirrorMain fork_key = @priority_queue.delete_min_return_key do_push(fork_key) - @priority_queue.push fork_key, (@priority - @fork_stat[fork_key][:priority]) + priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 + @priority_queue.push fork_key, priority_set @priority += 1 end -- 2.23.0

2 2

[PATCH compass-ci] rsync-server: add support upload function for ks file
by Yu Chuan 18 Feb '21

18 Feb '21

[Why] We need to achieve automatic installation through 'pxe + kickstart + dnsmasq + httpd'. So we need to rsync the ks file by rsync-server. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/rsync-server/rsyncd.conf | 6 ++++++ container/rsync-server/start | 1 + 2 files changed, 7 insertions(+) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index cdba89cf20c8..ef00acb1bc0c 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -10,3 +10,9 @@ path = /srv/result/upload/ read only = no max connections = 200 refuse options = delete + +[ks] +path = /srv/os/kickstart/ks/ +read only = no +max connections = 200 +refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 956013268f89..ecd783a22dd2 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -11,6 +11,7 @@ cmd=( --restart=always --name=rsync_server -v /srv/result/upload:/srv/result/upload + -v /srv/os/kickstart/ks:/srv/os/kickstart/ks -v /etc/localtime:/etc/localtime:ro -p 11387:11387 rsync_server:latest -- 2.23.0

2 6

[PATCH v2 lkp-tests] lib/job.rb: delete read "/etc/compass-ci/defaults/*.yaml"
by Cao Xueliang 18 Feb '21

18 Feb '21

The compass-ci/scheduler will read the yamls, so, delete it in lkp-tests Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/job.rb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index eacb7ec95..37c538753 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -418,8 +418,7 @@ class Job def load_self_config self_config_path = "#{ENV['HOME']}/.config/compass-ci" - Dir.glob(['/etc/compass-ci/defaults/*.yaml', - "#{self_config_path}/defaults/*.yaml"]).each do |file| + Dir.glob(["#{self_config_path}/defaults/*.yaml"]).each do |file| load_one_defaults(file, @job) end -- 2.23.0

2 2

[PATCH lkp-tests] daemon: add nginx-sever
by Wei Jihui 18 Feb '21

18 Feb '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/nginx-server | 173 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 173 insertions(+) create mode 100755 daemon/nginx-server diff --git a/daemon/nginx-server b/daemon/nginx-server new file mode 100755 index 00000000..a7dd1033 --- /dev/null +++ b/daemon/nginx-server @@ -0,0 +1,173 @@ +#!/bin/sh +# - nics +# - nics_ips +# - core_nums +# - protocol +# - alive_type + +. $LKP_SRC/lib/common.sh + +# To manually check CPU usage when clients sent http/https requests, please use htop tool. +# htop tool download url: http://rpmfind.net/linux/epel/8/Everything/aarch64/Packages/h/htop-2.2.0-6.… +# For send http request, can use httpress tool. +# httpress tool download url: http://rpmfind.net/linux/epel/7/aarch64/Packages/h/httpress-1.1.0-2.el7.aar… + +: "${protocol:=https}" # http or https +: "${alive_type:=short}" # short or long + +nics=(${nics//,/ }) +nics_ips=(${nics_ips//,/ }) +[ -n "$BENCHMARK_ROOT" ] || BENCHMARK_ROOT=/lkp/benchmarks + +check_nginx_install() +{ + if [ -x "$nginx_path" ]; then + pkill -f "$nginx_path" + else + echo "nginx is not installed" >&2 && exit 1 + fi + + if [ ! -f "$conf_dir/$protocol/basic.conf" ]; then + echo "Could not find nginx conf" >&2 && exit 1 + fi +} + +prepare() +{ + setenforce 0 + ulimit -SHn 409600 + export LD_PRELOAD=/usr/lib64/libtcmalloc.so.4 + + # require by $LKP_SRC/bin/set_nic_irq_affinity + pkill -9 irqbalance +} + +bind_core_net_irp() +{ + # binding cores are used to process NIC interrupts, each corresponds to a NIC. + # the interrupt cores in each CPU are bound only to the NIC that belong to the CPU. + # the CPU cores are bound by node. + + ip addr add "$nic_ip"/24 dev "$nic" + ip link set "$nic" up + + ethtool -L $nic combined $bind_core_nums + $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic +} + +update_nginx_conf() +{ + update_key=$1 + update_value="$(eval echo '$'$update_key)" + sed -i "s|{{$update_key}}|$update_value|" "$conf_path" +} + +start_nginx() +{ + local ssl_certificate_path="$conf_dir/$protocol/server_2048.crt" + local ssl_certificate_key_path="$conf_dir/$protocol/server_2048.key" + local conf_path="$conf_dir/$protocol.$alive_type.$core_range.conf" + cp -f "$conf_dir/$protocol/basic.conf" "$conf_path" + source "$conf_dir/$protocol/$alive_type/$core_range" + + update_nginx_conf html_index + update_nginx_conf listen_port + update_nginx_conf worker_processes + update_nginx_conf worker_cpu_affinity + [ "$protocol" == 'https' ] && { + update_nginx_conf ssl_certificate + update_nginx_conf ssl_certificate_key + } + + "$nginx_path" -c "$conf_path" -p "$BENCHMARK_ROOT/nginx-server/" +} + +assign_core_ranges() +{ + if [ "$protocol" == 'http' ]; then + case "$core_nums" in + "4") + if [ "$alive_type" == 'long' ]; then + core_ranges='3' + else + core_ranges='2-3' + fi + ;; + "8") + if [ "$alive_type" == 'long' ]; then + core_ranges='6-7' + else + core_ranges='2-3 6-7' + fi + ;; + "48") + if [ "$alive_type" == 'long' ]; then + core_ranges='18-23 42-47' + else + core_ranges='12-17 18-23 36-41 42-47' + fi + ;; + "96") + if [ "$alive_type" == 'long' ]; then + core_ranges='18-23 42-47 66-71 90-95' + else + core_ranges='0-15 16-31 48-63 64-79' + fi + ;; + *) echo "The enter core num:$core_nums is incorrect" && exit 1 + ;; + esac + else + case "$core_nums" in + "4") core_ranges='3' + ;; + "8") + if [ "$alive_type" == 'long' ]; then + core_ranges='6-7' + else + core_ranges='7' + fi + ;; + "48") + if [ "$alive_type" == 'long' ]; then + core_ranges='20-23 44-47' + else + core_ranges='23 47' + fi + ;; + "96") + if [ "$alive_type" == 'long' ]; then + core_ranges='20-23 44-47 68-71 92-95' + else + core_ranges='46-47 94-95' + fi + ;; + *) echo "The enter core num:$core_nums is incorrect" && exit 1 + ;; + esac + fi +} + +main() +{ + local core_ranges + local nginx_path="$BENCHMARK_ROOT/nginx-server/sbin/nginx" + local conf_dir="$BENCHMARK_ROOT/nginx-server/nginx_conf" + + check_nginx_install + prepare + assign_core_ranges + + index=0 + for core_range in $core_ranges + do + local nic=${nics[$index]} + local nic_ip=${nics_ips[$index]} + local bind_core_nums=$(cpu_list_num $core_range) + bind_core_net_irp + start_nginx + index=$((index+1)) + done +} + +main -- 2.23.0

1 0

[PATCH compass-ci] sbin/compare: fix No Method Error
by Lu Weitao 18 Feb '21

18 Feb '21

[Why] use: compare group_id=fio-basic-openeuler-acr-1301864-1,fio-basic-openeuler-acr-1301864-2,fio-basic-openeuler-acr-1301864-3 -d pp.acr3.acr error message: Traceback (most recent call last): 2: from /home/wangyong/compass-ci/sbin/compare:72:in `<main>' 1: from /home/wangyong/compass-ci/lib/compare.rb:66:in `compare_group' /home/lwt/compass-ci/lib/compare_matrixes.rb:400:in `compare_group_matrices': undefined method `[]' for nil:NilClass (NoMethodError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 4 ++++ lib/matrix2.rb | 2 ++ 2 files changed, 6 insertions(+) diff --git a/lib/compare.rb b/lib/compare.rb index 21e24df..7c182c4 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -63,6 +63,10 @@ def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') groups_matrices, suites_hash, latest_jobs_hash = create_groups_matrices_list(conditions, dims) + unless groups_matrices + warn 'Empty group matrices!' + exit + end compare_group_matrices(groups_matrices, suites_hash, latest_jobs_hash, options) end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ab92986..5e0ad21 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -128,6 +128,8 @@ def combine_group_query_data(job_list, dims) end get_groups_matrix(groups, group_key, value, suites_hash, latest_jobs_hash) end + return nil if groups.empty? + return groups, suites_hash, latest_jobs_hash end -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: fix that the priority with priority queue may be negative
by Li Yuanchao 18 Feb '21

18 Feb '21

Priority queue use the priority to decide which one pop first, and small priority first. And the more active repo should be pop first, so priority of repo would minus its new ref times to make its priority small. But some times, the result can be negative, this make the repo delayed a lot. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 940d5e2..39b1619 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,7 +203,8 @@ class MirrorMain fork_key = @priority_queue.delete_min_return_key do_push(fork_key) - @priority_queue.push fork_key, (@priority - @fork_stat[fork_key][:priority]) + priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 + @priority_queue.push fork_key, priority_set @priority += 1 end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] sparrow/dev-env: add the local-testbox-env.yaml
by Cao Xueliang 18 Feb '21

18 Feb '21

The scheduler will read local-testbox-env.yaml, then add the content to the job yaml. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8812839..31a12db 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -59,6 +59,23 @@ ES_PORT: $ES_PORT SRV_HTTP_PORT: $SRV_HTTP_PORT EOF +cat > /etc/compass-ci/defaults/local-testbox-env.yaml <<-EOF +LKP_SERVER: $server_ip +GIT_SERVER: $GIT_SERVER +INITRD_HTTP_HOST: $initrd_http_host +INITRD_HTTP_PORT: $initrd_http_port +GITCACHE_HOST: $gitcache_host +GITCACHE_PORT: $gitcache_port +SEND_MAIL_HOST: $server_ip +SEND_MAIL_PORT: $SEND_MAIL_PORT +ASSIST_RESULT_HOST: $server_ip +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +RESULT_WEBDAV_HOST: $server_ip +RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT +SRV_HTTP_HOST: $server_ip +SRV_HTTP_PORT: $SRV_HTTP_PORT +EOF + cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] scheduler: add testbox env in scheduler
by Cao Xueliang 18 Feb '21

18 Feb '21

We add the testbox env use the submit script in lkp-tests, but the submit don't need to do that, so, we add the testbox env in scheduler. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 2 ++ src/scheduler/testbox_env.cr | 17 +++++++++++++++++ 2 files changed, 19 insertions(+) create mode 100644 src/scheduler/testbox_env.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index b724b07..8f0dbab 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "scheduler/testbox_env.cr" require "../scheduler/elasticsearch_client" require "./json_logger" @@ -52,6 +53,7 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @hash.merge!(testbox_env) @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new diff --git a/src/scheduler/testbox_env.cr b/src/scheduler/testbox_env.cr new file mode 100644 index 0000000..a378e2a --- /dev/null +++ b/src/scheduler/testbox_env.cr @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + def testbox_env(flag = "local") + hash = Hash(String, JSON::Any).new + + yaml_any = File.open("/etc/compass-ci/defaults/#{flag}-testbox-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + hash.merge!(Hash(String, JSON::Any).from_json(yaml_any.to_json)) + + hash + end +end -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 18 Feb '21

18 Feb '21

[How] ansible_stats_openeuler < ${job_output} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 100 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 90 insertions(+), 10 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..f31771e5c 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,95 @@ #!/usr/bin/env ruby +require 'json' + while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line + line = line.gsub(/'/, ' ') + ansible_stats_error = `echo '#{line}' |grep -E 'FAILED! |failed: .*}$' | awk -F'=>' '{$1="";print $0}'` + next if ansible_stats_error.empty? + + a = JSON.load(ansible_stats_error) + unless a['msg'].nil? + + # When running some plug-in, the file was not found + if a['msg'].match(/No file was found when using (.+)\./) + puts "files.error.no-file-was-found-when-using.#{$1}: 1" + puts "files.error.no-file-was-found-when-using.#{$1}.message: #{ansible_stats_error}" + end + + # No such command + if a['msg'].match(/No such file or directory: b (.+) :/) + puts "command.error.no-such-file-or-directory.#{$1}: 1" + puts "command.error.no-such-file-or-directory.#{a['cmd']}.message: #{ansible_stats_error}" + end + + # The URL of the RPM is invalid + if a['msg'].match(/Failure downloading (.+), /) + puts "download.error.http-error-404 [#{$1}]: 1" + puts "download.error.http-error-404 [#{$1}]: #{ansible_stats_error}" + end + + # Failed to download the yum repodata + if a['msg'].match(/Failed to download metadata for repo (.+) :/) + puts "download.error.failed-to-download-metadata.#{$1}: 1" + puts "download.error.failed-to-download-metadata.#{$1}.message: #{ansible_stats_error}" + end + + if a['msg'].match(/Failed to download packages: Cannot download (.+):/) + puts "download.error.failed-to-download-package.#{$1}: 1" + puts "download.error.failed-to-download-package.#{$1}.message: #{ansible_stats_error}" + end + + # The task includes an option with an undefined variable. + if a['msg'].match(/: (.+) is undefined/) + puts "syntax.error.an-undefined-variable.#{$1}: 1" + puts "syntax.error.an-undefined-variable.#{$1}.message: #{ansible_stats_error}" + end + + # The command runing error. + if a['msg'].match(/non-zero return code/) + puts "command.error[#{a['cmd']}]: 1" + puts "command.error [#{a['cmd']}].message: #{ansible_stats_error}" + end + + # Some service could not fand the requested host. + if a['msg'].match(/Could not find the requested service (.+): host/) + puts "request.error.could-not-find-service-host.#{$1}: 1" + puts "request.error.could-not-find-service-host.#{$1}.messgae: #{ansible_stats_error}" + end + + # Unable to start some services + if a['msg'].match(/Unable to start service (.+):/) + puts "service.error.unable-to-star-service.#{$1}: 1" + puts "service.error.unable-to-star-service.#{$1}.message: #{ansible_stats_error}" + end + + # Some role in not sipported this system + if a['msg'].match(/(.+) is not supported/) + puts "system.error.not-supported.#{$1.gsub(/ /, '-')}: 1" + puts "system.error.not-supported.message: #{ansible_stats_error}" + end + + # failed to create temporary content file: The read operation timed out. + if a['msg'].match(/The read operation timed out/) + puts 'connection.error.read-timed-out: 1' + puts "connection.error.read-timed-out.message: #{ansible_stats_error}" + end + end + + unless a['failures'].nil? + a['failures'].each do |items| + # Failed to install some of the specified packages. + items.match(/No package (.+) available/) + puts "yum.error.no-package-match.#{$1}: 1" + puts "yum.error.no-package-match.#{$1}.message: #{line}" + end + end + + next if a['message'].nil? + + # Could not find or access ansible yaml. + if a['message'].to_s.match(/Could not find or access (.+) /) + puts "yaml.error.not-find-yml.#{$1}: 1" + puts "yaml.error.not-find-yml.#{$1}.message: #{ansible_stats_error}" end end -- 2.23.0

3 4

[PATCH lkp-tests] setup/fs: support much values of hdd_partitions parameter
by Cui Yili 10 Feb '21

10 Feb '21

hdd_partitions: /dev/vda /dev/vdb as above, setup/fs only deal with /dev/vda. Signed-off-by: Cui Yili <2268260388(a)qq.com> --- setup/fs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/fs b/setup/fs index 23eaa3b85..824c86614 100755 --- a/setup/fs +++ b/setup/fs @@ -159,7 +159,7 @@ mount_fs() { if is_btrfs_raid_levels; then bdevs="${partitions%% *}" else - bdevs="$partitions" + bdevs="$hdd_partitions" fi fs_options -- 2.23.0

2 1

[PATCH v3 lkp-tests 2/2] tests/cci-makepkg: download submodules from remote end
by Liu Shaofei 10 Feb '21

10 Feb '21

[why] When building software package by cci-makepkg, the source codes repository contains submodules. By default, the submodules cannot be downloaded from Z9/Crystal during building packages. [how] By appending --norevisedomain param, disable submodules to be downloaded from Z9/Crystal. Just download submodules from remote end. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/cci-makepkg | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index ecbf72134..9c2ddc257 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -134,7 +134,7 @@ update_softlink() build_source_pkg() { local mark="true" - local pkg_args="-A --check --skippgpcheck" + local pkg_args="-A --check --skippgpcheck --norevisedomain" [ "${DISTRO}" == "archlinux" ] && { local mark="pacman" local pkg_args="${pkg_args} -s --needed --noconfirm" @@ -142,7 +142,11 @@ build_source_pkg() local makepkg="$LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf" echo "${makepkg}" - PACMAN="${mark}" BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + PACMAN="${mark}" \ + BUILDDIR="$TMP" \ + CARCH="$arch" \ + PKGEXT=".cgz" \ + CGZDEST="$sync_dest/$cgz_name" $makepkg [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] sbin/makepkg: fix whether to change domain of submodules
by Liu Shaofei 10 Feb '21

10 Feb '21

the param "--norevisedomain" indicates that all of submodules will not be changed domian name, and downloaded from remote end. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/makepkg | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 7a92ea71b..6b5836251 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -89,6 +89,7 @@ SIGNPKG='' SPLITPKG=0 SOURCEONLY=0 VERIFYSOURCE=0 +NOREVISEDOMAIN=0 # Forces the pkgver of the current PKGBUILD. Used by the fakeroot call # when dealing with svn/cvs/etc PKGBUILDs. @@ -587,6 +588,16 @@ download_git() { fi } +# modify submodules address to local server address. +# (( NOREVISEDOMAIN )) is true, not replace submodules domain, and download it from remote end. +modify_gitmodules() { + (( NOREVISEDOMAIN )) && return + + if [[ -f ".gitmodules" ]] && [[ -n ${GIT_SERVER} ]]; then + git config --system url."git://${GIT_SERVER}/".insteadOf https:// + fi +} + extract_git() { local netfile=$1 @@ -623,10 +634,7 @@ extract_git() { cd_safe "${dir##*/}" - # modify submodules address to local server address. - if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then - echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig - fi + modify_gitmodules local ref=origin/HEAD if [[ -n $fragment ]]; then @@ -3421,6 +3429,7 @@ usage() { printf -- "$(gettext " --skipinteg Do not perform any verification checks on source files")\n" printf -- "$(gettext " --skippgpcheck Do not verify source files with PGP signatures")\n" printf -- "$(gettext " --verifysource Download source files (if needed) and perform integrity checks")\n" + printf -- "$(gettext " --norevisedomain Do not change domain name when downloading submodules repository")\n" echo printf -- "$(gettext "These options can be passed to %s:")\n" "pacman" echo @@ -3462,7 +3471,7 @@ OPT_LONG=('allsource' 'check' 'clean' 'cleanbuild' 'config:' 'force' 'geninteg' 'help' 'holdver' 'ignorearch' 'install' 'key:' 'log' 'noarchive' 'nobuild' 'nocolor' 'nocheck' 'nodeps' 'noextract' 'noprepare' 'nosign' 'pkg:' 'repackage' 'rmdeps' 'sign' 'skipchecksums' 'skipinteg' 'skippgpcheck' 'source' 'syncdeps' - 'verifysource' 'version') + 'verifysource' 'version' 'norevisedomain') # Pacman Options OPT_LONG+=('asdeps' 'noconfirm' 'needed' 'noprogressbar') @@ -3514,6 +3523,7 @@ while true; do -s|--syncdeps) DEP_BIN=1 ;; -S|--source) SOURCEONLY=1 ;; --verifysource) VERIFYSOURCE=1 ;; + --norevisedomain) NOREVISEDOMAIN=1 ;; -h|--help) usage; exit 0 ;; # E_OK -V|--version) version; exit 0 ;; # E_OK -- 2.23.0

1 0

[PATCH compass-ci] container/defconfig.rb: optimize skip rebuild image judge
by Wei Jihui 10 Feb '21

10 Feb '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/defconfig.rb | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index aa7d2fa..602b498 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -49,8 +49,7 @@ def get_available_memory end def docker_skip_rebuild(tag) - return if ENV['action'].nil? - return unless ENV['action'].include? 'run-only' + return if ENV['action'] != 'run-only' exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" end -- 2.23.0

1 0

[PATCH v2 lkp-tests] setup/fs: support much values of hdd_partitions parameter
by Cui Yili 10 Feb '21

10 Feb '21

error: hdd_partitions: /dev/vda /dev/vdb as above, setup/fs only deal with /dev/vda. /dev/vdb did not recognize. Signed-off-by: Cui Yili <2268260388(a)qq.com> --- setup/fs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/fs b/setup/fs index 23eaa3b85..824c86614 100755 --- a/setup/fs +++ b/setup/fs @@ -159,7 +159,7 @@ mount_fs() { if is_btrfs_raid_levels; then bdevs="${partitions%% *}" else - bdevs="$partitions" + bdevs="$hdd_partitions" fi fs_options -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] sbin/makepkg: download submodules from local server by default
by Liu Shaofei 10 Feb '21

10 Feb '21

[why] When building software package by cci-makepkg, the source codes repository contains submodules. By default, these submodules are downloaded from the local server, but the local server does not have submodules download address. So the submodules repository fails to be downloaded to test machine during the build. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/makepkg | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 7a92ea71b..b592a2ccb 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -587,6 +587,17 @@ download_git() { fi } +# modify submodules address to local server address. +# [ "$DISABLE_DOMAIN_CHANGE" == "false" ] => replace submodules domain, and download it from local server +# [ "$DISABLE_DOMAIN_CHANGE" == "true" ] => not replace submodules domain, and download it from remote end. +modify_gitmodules() { + [ "$DISABLE_DOMAIN_CHANGE" == "false" ] || return + + if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then + echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig + fi +} + extract_git() { local netfile=$1 @@ -623,10 +634,7 @@ extract_git() { cd_safe "${dir##*/}" - # modify submodules address to local server address. - if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then - echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig - fi + modify_gitmodules local ref=origin/HEAD if [[ -n $fragment ]]; then @@ -3668,6 +3676,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +DISABLE_DOMAIN_CHANGE=${DISABLE_DOMAIN_CHANGE:-false} if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then -- 2.23.0

2 3

[PATCH v2 lkp-tests 2/2] tests/cci-makepkg: download submodules from remote end
by Liu Shaofei 10 Feb '21

10 Feb '21

DISABLE_DOMAIN_CHANGE parameter value is true, "sbin/makepkg" file will not replace domain name of submodules, the submodules will be downloaded from remote end. Signed-off-by: Liu Shaofei <370072077(a)qq.com> i# Your branch is ahead of 'origin/master' by 1 commit. --- tests/cci-makepkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index ecbf72134..8c744d807 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -142,7 +142,7 @@ build_source_pkg() local makepkg="$LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf" echo "${makepkg}" - PACMAN="${mark}" BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + PACMAN="${mark}" DISABLE_DOMAIN_CHANGE=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

2 2

[PATCH compass-ci] src/scheduler/kernel_params.cr: no rootovl for local mount
by Xu Xijian 09 Feb '21

09 Feb '21

If the os_mount is "local", there is no need of overlay and it should be writable, so in the kernel params, delete "rootovl" for it and change "ro" to "rw" for it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index ed64bea..6fbe405 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -3,7 +3,9 @@ class Job private def kernel_common_params - return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" + common_params = "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job ip=dhcp" + return "#{common_params} rw" if "#{self.os_mount}" == "local" + return "#{common_params} rootovl ro" end private def kernel_custom_params -- 2.23.0

1 0

[PATCH compass-ci] container/defconfig.rb: fix undefined method `includes' for nil:NilClass
by Xu Xijian 09 Feb '21

09 Feb '21

[Why] When "ENV['action']" is nil, it will output error in this line in defconfig.rb: return unless ENV['action'].include? 'run-only' [errlog] Traceback (most recent call last): 1: from ./build:15:in `<main>' /home/xxj/compass-ci/container/defconfig.rb:52:in `docker_skip_rebuild': undefined method `include?' for nil:NilClass (NoMethodError) Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/defconfig.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 7dc9f33..aa7d2fa 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -49,6 +49,7 @@ def get_available_memory end def docker_skip_rebuild(tag) + return if ENV['action'].nil? return unless ENV['action'].include? 'run-only' exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: delete redundant wait
by Liu Yinsi 09 Feb '21

09 Feb '21

[why] if `&` is followed by `wait` is invalid to parallel run, in function do_one_build(), already has `wait`, so delete it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 1 - 1 file changed, 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 4fcbac7..e7309fd 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -70,7 +70,6 @@ do [ "$container_name" == 'ssh-r' ] && continue build_depends $dir & - wait start_depends $dir done -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/4-docker/buildall: delete redundant wait
by Liu Yinsi 09 Feb '21

09 Feb '21

[why] if `&` is followed by `wait` is invalid to parallel run, in function do_one_build(), already has `wait`and `&`, so delete it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a4dab4b..96c7385 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -67,8 +67,7 @@ tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir & - wait + build_depends $dir start_depends $dir done -- 2.23.0

1 0

[PATCH compass-ci] qemu/kvm.sh: kvm support virtual disk
by Cui Yili 09 Feb '21

09 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- providers/qemu/kvm.sh | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 68b9a62..4be33b9 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -97,6 +97,26 @@ set_helper() [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper } +add_disk() +{ + [ -d "$mount_points" ] || mount_points=$(pwd) + + [ -n "$hdd_partitions" ] && { + local unix_time=$(date +%s) + + local index=0 + for disk in $hdd_partitions + do + qcow2_file="${mount_points}/${unix_time}${disk##*/}.qcow2" + drive="file=${qcow2_file},media=disk,format=qcow2,index=${index}" + ((index++)) + + qemu-img create -fq qcow2 "${qcow2_file}" 512G + kvm+=(-drive ${drive}) + done + } +} + set_nic() { nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" @@ -134,7 +154,7 @@ print_message() log_info initrds: $initrds log_info append: $append [ "$DEBUG" == "true" ] || log_info less $log_file - + sleep 5 } @@ -228,6 +248,7 @@ set_options print_message public_option +add_disk individual_option run_qemu -- 2.23.0

1 0

[PATCH compass-ci] doc/support-os-os_arch-os_version.md: add 20.03-SP1
by Xu Xijian 09 Feb '21

09 Feb '21

1. add 20.03-SP1 version to this doc, it's already supported 2. adjust some lines to be aligned Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/support-os-os_arch-os_version.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md index 5d30486..f91e91b 100644 --- a/doc/support-os-os_arch-os_version.md +++ b/doc/support-os-os_arch-os_version.md @@ -1,22 +1,24 @@ ``` os os_arch os_version supported-os_mount openeuler aarch64 1.0 cifs/nfs, - 20.03 cifs/nfs, ,initramfs - 20.09 cifs/nfs, ,initramfs + 20.03 cifs/nfs, ,initramfs + 20.03-SP1 cifs/nfs, + 20.09 cifs/nfs, ,initramfs mainline-dailybuild cifs/nfs, 20.03-dailybuild cifs/nfs, 20.09-dailybuild cifs/nfs, 20.03-sp1-dailybuild cifs/nfs, x86 20.03 cifs/nfs, + 20.03-SP1 cifs/nfs, 20.09 cifs/nfs, centos aarch64 7 container, - 7.6.1810 cifs/nfs, ,initramfs - 7.8.2003 cifs/nfs, ,initramfs + 7.6.1810 cifs/nfs, ,initramfs + 7.8.2003 cifs/nfs, ,initramfs 8.1.1911 cifs/nfs, 8 container, x86 7.6.1810 cifs/nfs, 7.8.2003 cifs/nfs, -debian aarch64 sid cifs/nfs, ,initramfs +debian aarch64 sid cifs/nfs, ,initramfs x86 sid cifs/nfs, archlinux aarch64 2020-11-12 cifs/nfs, ``` -- 2.23.0

1 0

[PATCH compass-ci] serial_log: change permission
by Wu Zhende 09 Feb '21

09 Feb '21

The automatically generated file may be unreadable due to umask. Change the permission to the required permission when creating. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 1 + providers/multi-qemu | 1 + 2 files changed, 2 insertions(+) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 56c9754..28ed29b 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -96,6 +96,7 @@ end def record_start_log(log_file, hash: {}) start_time = Time.new File.open(log_file, 'w') do |f| + f.chmod(0664) # fluentd refresh time is 1s # let fluentd to monitor this file first sleep(2) diff --git a/providers/multi-qemu b/providers/multi-qemu index c729c8e..1f945e6 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -53,6 +53,7 @@ def record_runtime_log(hostname, start_time: Time.new, is_start: true) log_file = "#{LOG_DIR}/#{hostname}" if is_start File.open(log_file, 'w') do |f| + f.chmod(0664) # fluentd refresh time is 1s # let fluentd to monitor this file first sleep(2) -- 2.23.0

2 1

[PATCH compass-ci] sparrow: start ssh-r container when the host reboot
by Cao Xueliang 09 Feb '21

09 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/7-systemd/cci-network | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/7-systemd/cci-network b/sparrow/7-systemd/cci-network index 0438935..d01295c 100755 --- a/sparrow/7-systemd/cci-network +++ b/sparrow/7-systemd/cci-network @@ -15,3 +15,5 @@ $CCI_SRC/sparrow/2-network/cifs # --restart=always option is not absolutely reliable # container need to start its dependences firstly. $CCI_SRC/sparrow/4-docker/buildall run-only +# need start ssh-r when the host reboot +$CCI_SRC/container/ssh-r/start -- 2.23.0

1 1

[PATCH v4 compass-ci 3/3] sparrow/4-docker/buildall: delete unnecessary code
by Liu Yinsi 09 Feb '21

09 Feb '21

[why] avoid to rebuild container, this issue already solved by calling docker_skip_rebuild() in each build script, so delete related code. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index e6ba448..4fcbac7 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -69,11 +69,8 @@ do container_name=$(basename $dir) [ "$container_name" == 'ssh-r' ] && continue - docker images | grep -wq "$container_name" - if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then - build_depends $dir & - wait - fi + build_depends $dir & + wait start_depends $dir done -- 2.23.0

2 2

[PATCH compass-ci] sparrow/4-docker/buildall: build container ssh-r, but skip run
by Liu Yinsi 09 Feb '21

09 Feb '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 4fcbac7..a4dab4b 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,6 +56,7 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" + [ "$container_name" == 'ssh-r' ] && continue [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run @@ -66,9 +67,6 @@ tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - container_name=$(basename $dir) - [ "$container_name" == 'ssh-r' ] && continue - build_depends $dir & wait start_depends $dir -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package/common: fix invalid sed command
by Liu Yinsi 09 Feb '21

09 Feb '21

before: sed /^Environment=/a\TimeoutSec=0' after: sed -i /^Environment=/a\TimeoutSec=0' Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index fca591c..0324f58 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -48,7 +48,7 @@ cat > /etc/docker/daemon.json <<EOF } EOF -sed '/^Environment=/a\TimeoutSec=0' /usr/lib/systemd/system/docker.service +sed -i '/^Environment=/a\TimeoutSec=0' /usr/lib/systemd/system/docker.service systemctl daemon-reload systemctl restart docker -- 2.23.0

1 0

[PATCH v2 lkp-tests] hosts/*: add default hdd_partitions
by Wang Yong 09 Feb '21

09 Feb '21

add default hdd_partitions for vm Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- hosts/vm-1p1g | 1 + hosts/vm-2p16g | 1 + hosts/vm-2p1g | 1 + hosts/vm-2p32g | 1 + hosts/vm-2p4g | 1 + hosts/vm-2p8g | 1 + 6 files changed, 6 insertions(+) diff --git a/hosts/vm-1p1g b/hosts/vm-1p1g index c64cc3601..1b26b2fc6 100644 --- a/hosts/vm-1p1g +++ b/hosts/vm-1p1g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 1 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p16g b/hosts/vm-2p16g index 428450720..21ec35343 100644 --- a/hosts/vm-2p16g +++ b/hosts/vm-2p16g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 16G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g b/hosts/vm-2p1g index c78103361..0052caf1a 100644 --- a/hosts/vm-2p1g +++ b/hosts/vm-2p1g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p32g b/hosts/vm-2p32g index 14455f202..ab876e8f9 100644 --- a/hosts/vm-2p32g +++ b/hosts/vm-2p32g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 32G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g b/hosts/vm-2p4g index 7e9570da1..3282e3793 100644 --- a/hosts/vm-2p4g +++ b/hosts/vm-2p4g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g b/hosts/vm-2p8g index e5d5f846b..d92082523 100644 --- a/hosts/vm-2p8g +++ b/hosts/vm-2p8g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: /dev/vda -- 2.23.0

1 0

[PATCH lkp-tests] hosts/*: add default hdd_partitions
by Wang Yong 09 Feb '21

09 Feb '21

add default hdd_partitions for vm Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- hosts/vm-1p1g | 1 + hosts/vm-1p1g-pxe | 1 + hosts/vm-2p16g | 1 + hosts/vm-2p1g | 1 + hosts/vm-2p1g-pxe | 1 + hosts/vm-2p32g | 1 + hosts/vm-2p4g | 1 + hosts/vm-2p4g-pxe | 1 + hosts/vm-2p8g | 1 + hosts/vm-2p8g-pxe | 1 + hosts/vm-hi1620-1p1g | 1 + hosts/vm-hi1620-2p1g | 1 + hosts/vm-hi1620-2p4g | 1 + hosts/vm-hi1620-2p8g | 1 + hosts/vm-pxe-hi1620-1p1g | 1 + hosts/vm-pxe-hi1620-2p1g | 1 + hosts/vm-pxe-hi1620-2p4g | 1 + hosts/vm-pxe-hi1620-2p8g | 1 + hosts/vm-snb | 1 + hosts/vm-snb-i386 | 1 + 20 files changed, 20 insertions(+) diff --git a/hosts/vm-1p1g b/hosts/vm-1p1g index c64cc3601..1b26b2fc6 100644 --- a/hosts/vm-1p1g +++ b/hosts/vm-1p1g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 1 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-1p1g-pxe b/hosts/vm-1p1g-pxe index 233f63935..5c0e081cc 100644 --- a/hosts/vm-1p1g-pxe +++ b/hosts/vm-1p1g-pxe @@ -3,3 +3,4 @@ template: pxe nr_node: 1 nr_cpu: 1 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p16g b/hosts/vm-2p16g index 428450720..21ec35343 100644 --- a/hosts/vm-2p16g +++ b/hosts/vm-2p16g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 16G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g b/hosts/vm-2p1g index c78103361..0052caf1a 100644 --- a/hosts/vm-2p1g +++ b/hosts/vm-2p1g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g-pxe b/hosts/vm-2p1g-pxe index 6129d20e0..60658d380 100644 --- a/hosts/vm-2p1g-pxe +++ b/hosts/vm-2p1g-pxe @@ -3,3 +3,4 @@ template: pxe nr_node: 1 nr_cpu: 2 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p32g b/hosts/vm-2p32g index 14455f202..ab876e8f9 100644 --- a/hosts/vm-2p32g +++ b/hosts/vm-2p32g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 32G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g b/hosts/vm-2p4g index 7e9570da1..3282e3793 100644 --- a/hosts/vm-2p4g +++ b/hosts/vm-2p4g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g-pxe b/hosts/vm-2p4g-pxe index 8ba4e58ac..638f803f8 100644 --- a/hosts/vm-2p4g-pxe +++ b/hosts/vm-2p4g-pxe @@ -3,3 +3,4 @@ template: pxe nr_node: 1 nr_cpu: 2 memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g b/hosts/vm-2p8g index e5d5f846b..d92082523 100644 --- a/hosts/vm-2p8g +++ b/hosts/vm-2p8g @@ -3,3 +3,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g-pxe b/hosts/vm-2p8g-pxe index ae83d6f00..caa6a88f7 100644 --- a/hosts/vm-2p8g-pxe +++ b/hosts/vm-2p8g-pxe @@ -3,3 +3,4 @@ template: pxe nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: /dev/vda diff --git a/hosts/vm-hi1620-1p1g b/hosts/vm-hi1620-1p1g index bf2a7b39f..eae6e1793 100644 --- a/hosts/vm-hi1620-1p1g +++ b/hosts/vm-hi1620-1p1g @@ -4,3 +4,4 @@ nr_cpu: 1 nr_node: 1 memory: 1G arch: aarch64 +hdd_partitions: /dev/vda diff --git a/hosts/vm-hi1620-2p1g b/hosts/vm-hi1620-2p1g index 1c4786acc..b3bd9254d 100644 --- a/hosts/vm-hi1620-2p1g +++ b/hosts/vm-hi1620-2p1g @@ -4,3 +4,4 @@ nr_cpu: 2 nr_node: 1 memory: 1G arch: aarch64 +hdd_partitions: /dev/vda diff --git a/hosts/vm-hi1620-2p4g b/hosts/vm-hi1620-2p4g index f2a5b5ea4..bc0f7796e 100644 --- a/hosts/vm-hi1620-2p4g +++ b/hosts/vm-hi1620-2p4g @@ -4,3 +4,4 @@ nr_cpu: 2 nr_node: 1 memory: 4G arch: aarch64 +hdd_partitions: /dev/vda diff --git a/hosts/vm-hi1620-2p8g b/hosts/vm-hi1620-2p8g index 17805246f..2c5bf624d 100644 --- a/hosts/vm-hi1620-2p8g +++ b/hosts/vm-hi1620-2p8g @@ -4,3 +4,4 @@ nr_cpu: 2 nr_node: 1 memory: 8G arch: aarch64 +hdd_partitions: /dev/vda diff --git a/hosts/vm-pxe-hi1620-1p1g b/hosts/vm-pxe-hi1620-1p1g index 9fe6ab6e2..c9b76942c 100644 --- a/hosts/vm-pxe-hi1620-1p1g +++ b/hosts/vm-pxe-hi1620-1p1g @@ -3,3 +3,4 @@ template: pxe nr_cpu: 1 memory: 1G arch: aarch64 +hdd_partitions: /dev/vda diff --git a/hosts/vm-pxe-hi1620-2p1g b/hosts/vm-pxe-hi1620-2p1g index 597c64ef8..7e4281e5c 100644 --- a/hosts/vm-pxe-hi1620-2p1g +++ b/hosts/vm-pxe-hi1620-2p1g @@ -3,3 +3,4 @@ template: pxe nr_cpu: 2 memory: 1G arch: aarch64 +hdd_partitions: /dev/vda diff --git a/hosts/vm-pxe-hi1620-2p4g b/hosts/vm-pxe-hi1620-2p4g index 60112abd7..568839610 100644 --- a/hosts/vm-pxe-hi1620-2p4g +++ b/hosts/vm-pxe-hi1620-2p4g @@ -3,3 +3,4 @@ template: pxe nr_cpu: 2 memory: 4G arch: aarch64 +hdd_partitions: /dev/vda diff --git a/hosts/vm-pxe-hi1620-2p8g b/hosts/vm-pxe-hi1620-2p8g index 801e7b655..ea2c765a3 100644 --- a/hosts/vm-pxe-hi1620-2p8g +++ b/hosts/vm-pxe-hi1620-2p8g @@ -3,3 +3,4 @@ template: pxe nr_cpu: 2 memory: 8G arch: aarch64 +hdd_partitions: /dev/vda diff --git a/hosts/vm-snb b/hosts/vm-snb index 7fc279796..aecd726fb 100644 --- a/hosts/vm-snb +++ b/hosts/vm-snb @@ -1,3 +1,4 @@ model: qemu-system-x86_64 -enable-kvm -cpu SandyBridge nr_cpu: 2 memory: 8G +hdd_partitions: /dev/vda diff --git a/hosts/vm-snb-i386 b/hosts/vm-snb-i386 index bdbcd03e0..744047bdd 100644 --- a/hosts/vm-snb-i386 +++ b/hosts/vm-snb-i386 @@ -1,3 +1,4 @@ model: qemu-system-i386 -enable-kvm -cpu SandyBridge nr_cpu: 2 memory: 8G +hdd_partitions: /dev/vda -- 2.23.0

2 2

[PATCH lkp-tests] tests/cci-makepkg: disable submoudles download from server
by Liu Shaofei 09 Feb '21

09 Feb '21

[why] When building software package by cci-makepkg, the source codes repository contains submodules. By default, these submodules are downloaded from the local server, but the local server does not have submodules download address. So the submodules repository fails to be downloaded to test machine during the build. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/makepkg | 15 +++++++++++---- tests/cci-makepkg | 2 +- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 7a92ea71b..639454d3b 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -587,6 +587,15 @@ download_git() { fi } +# modify submodules address to local server address. +modify_gitmodules() { + [ "$DISABLE_SUBMODULES" == "true" ] || return + + if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then + echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig + fi +} + extract_git() { local netfile=$1 @@ -623,10 +632,7 @@ extract_git() { cd_safe "${dir##*/}" - # modify submodules address to local server address. - if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then - echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig - fi + modify_gitmodules local ref=origin/HEAD if [[ -n $fragment ]]; then @@ -3668,6 +3674,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +DISABLE_SUBMODULES=${DISABLE_SUBMODULES:-false} if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then diff --git a/tests/cci-makepkg b/tests/cci-makepkg index ecbf72134..59f18d9a7 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -142,7 +142,7 @@ build_source_pkg() local makepkg="$LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf" echo "${makepkg}" - PACMAN="${mark}" BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + PACMAN="${mark}" DISABLE_SUBMODULES=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

2 4

[PATCH v3 compass-ci 1/3] container/dracut-initrd: merge 3 apt-get install cmds
by Xu Xijian 09 Feb '21

09 Feb '21

1. merge three apt-get install cmds into one 2. add -qq for apt-get install, -qq means no output except for errors, so we can remove ">/dev/null 2>&1". Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 75148a1..f7119ec 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -5,9 +5,9 @@ export DEBIAN_FRONTEND=noninteractive apt-get update -apt-get install -y --fix-missing --no-install-recommends apt-utils >/dev/null 2>&1 -apt-get install -y --fix-missing nfs-common netbase cifs-utils kmod rsync -apt-get install -y --fix-missing dracut dracut-network dracut-config-generic +apt-get install -y --fix-missing --no-install-recommends -qq apt-utils \ +nfs-common netbase cifs-utils kmod rsync dracut dracut-network \ +dracut-config-generic apt-get clean rm -rf /var/lib/apt/lists/* -- 2.23.0

2 1

[PATCH v3 compass-ci 2/3] container/dracut-initrd: fix: awk: command not found
by Xu Xijian 09 Feb '21

09 Feb '21

[Why] When start an os, it will output awk: command not found in dracut stage. See errlog, this happens in 40network module, so let's install awk. [errlog] [ 7.417877] dracut-initqueue[368]: /lib/net-lib.sh: line 125: awk: command not found Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index f7119ec..8f03fa8 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -16,3 +16,4 @@ rm -rf /var/lib/apt/lists/* cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh +sed -i "/install() {/ainst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -- 2.23.0

3 3

[PATCH v4 compass-ci 2/3] container: call docker_skip_rebuild()
by Liu Yinsi 09 Feb '21

09 Feb '21

prevent rebuild container when execute `sparrow/4-docker/buildall run-only` Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/add-software/build | 4 ++++ container/alpine/build | 4 ++++ container/archlinux/build | 4 ++++ container/assign-account/build | 4 ++++ container/assist-result/build | 4 ++++ container/assistant/build | 4 ++++ container/auto-submit/build | 4 ++++ container/conserver/build | 4 ++++ container/crystal-base/build | 4 ++++ container/crystal-compiler/build | 4 ++++ container/crystal-shards/build | 3 +++ container/debian/build | 4 ++++ container/delimiter/build | 2 ++ container/dnsmasq/build | 4 ++++ container/dracut-initrd/build | 4 ++++ container/es/build | 2 ++ container/extract-stats/build | 4 ++++ container/fetch-mail/build | 4 ++++ container/fluentd-base/build | 4 ++++ container/git-daemon/build | 4 ++++ container/git-mirror/build | 4 ++++ container/gitcache/build | 4 ++++ container/initrd-cifs/build | 4 ++++ container/initrd-http/build | 4 ++++ container/kibana/build | 4 ++++ container/lifecycle/build | 2 ++ container/lkp-initrd/build | 4 ++++ container/logging-es/build | 2 ++ container/mail-robot/build | 4 ++++ container/manjaro/build | 4 ++++ container/master-fluentd/build | 4 ++++ container/minio/build | 4 ++++ container/monitoring/build | 2 ++ container/netdata-slave/build | 4 ++++ container/netdata/build | 4 ++++ container/ntp-server/build | 4 ++++ container/openresty-proxy-cache/build | 4 ++++ container/os-cifs/build | 4 ++++ container/os-http/build | 4 ++++ container/os-nfs/build | 4 ++++ container/qcow2rootfs/build | 4 ++++ container/qemu-efi/build | 4 ++++ container/rabbitmq/build | 4 ++++ container/redis/build | 4 ++++ container/register-accounts/build | 4 ++++ container/registry/build | 4 ++++ container/remote-git/build | 4 ++++ container/result-cifs/build | 4 ++++ container/result-webdav/build | 4 ++++ container/rsync-server/build | 4 ++++ container/scheduler-dev/build | 4 ++++ container/scheduler/build | 2 ++ container/send-internet-mail/build | 4 ++++ container/send-mail/build | 4 ++++ container/shellcheck/build | 4 ++++ container/srv-http/build | 4 ++++ container/ssh-r/build | 4 ++++ container/sub-fluentd/build | 4 ++++ container/submit/build | 4 ++++ container/taskqueue/build | 2 ++ container/upload-libvirt-xml/build | 4 ++++ container/web-backend-nginx/build | 4 ++++ container/web-backend/build | 4 ++++ container/webhook/build | 4 ++++ 64 files changed, 241 insertions(+) diff --git a/container/add-software/build b/container/add-software/build index 234c8b9..bc3c95a 100755 --- a/container/add-software/build +++ b/container/add-software/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "addsoftware:auto" + docker build -t addsoftware:auto . diff --git a/container/alpine/build b/container/alpine/build index bd277c3..3168799 100755 --- a/container/alpine/build +++ b/container/alpine/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:testbed" + docker build -t alpine:testbed . diff --git a/container/archlinux/build b/container/archlinux/build index 9749489..6db5a68 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "archlinux:testbed" + docker build --build-arg ARCH=$(arch) -t archlinux:testbed . diff --git a/container/assign-account/build b/container/assign-account/build index 3984341..480c410 100755 --- a/container/assign-account/build +++ b/container/assign-account/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "assign-account:latest" + docker build -t assign-account:latest . diff --git a/container/assist-result/build b/container/assist-result/build index 130127b..600b80a 100755 --- a/container/assist-result/build +++ b/container/assist-result/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "assist-result" + docker build -t assist-result . diff --git a/container/assistant/build b/container/assistant/build index 7778c9f..89a8a97 100755 --- a/container/assistant/build +++ b/container/assistant/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:assistant" + docker build -t debian:assistant . diff --git a/container/auto-submit/build b/container/auto-submit/build index 96383e3..48ff111 100755 --- a/container/auto-submit/build +++ b/container/auto-submit/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:auto-submit" + docker build -t alpine:auto-submit . diff --git a/container/conserver/build b/container/conserver/build index 0700c5e..a7e3867 100755 --- a/container/conserver/build +++ b/container/conserver/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "conserver_server" + ./generate_conserver.rb || exit 1 docker build -t conserver_server . diff --git a/container/crystal-base/build b/container/crystal-base/build index 4df65a5..642e190 100755 --- a/container/crystal-base/build +++ b/container/crystal-base/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:crystal-base" + if [ -n "$SQUID_HOST" ] && [ -n "$SQUID_PORT" ]; then proxy="http://$SQUID_HOST:$SQUID_PORT/" else diff --git a/container/crystal-compiler/build b/container/crystal-compiler/build index cee43eb..570fe98 100755 --- a/container/crystal-compiler/build +++ b/container/crystal-compiler/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:crystal-compiler" + docker build -t alpine:crystal-compiler . diff --git a/container/crystal-shards/build b/container/crystal-shards/build index 784f571..7c2a712 100755 --- a/container/crystal-shards/build +++ b/container/crystal-shards/build @@ -6,6 +6,9 @@ # If job.yaml does not have these two parameters, git will use the local proxy cache, # GITCACHE_HOST defaults to the local ip, GITCACHE_PORT default port is 5000 . ../defconfig.sh + +docker_skip_rebuild "alpine:crystal-shards" + load_cci_defaults docker build --build-arg GITCACHE_HOST=$GITCACHE_HOST --build-arg GITCACHE_PORT=$GITCACHE_PORT -t alpine:crystal-shards . diff --git a/container/debian/build b/container/debian/build index 6bff579..eac9af2 100755 --- a/container/debian/build +++ b/container/debian/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:testbed" + docker build -t debian:testbed . diff --git a/container/delimiter/build b/container/delimiter/build index f7f53e7..606149d 100755 --- a/container/delimiter/build +++ b/container/delimiter/build @@ -4,6 +4,8 @@ . ../defconfig.sh +docker_skip_rebuild "delimiter" + load_cci_defaults docker build -t delimiter . diff --git a/container/dnsmasq/build b/container/dnsmasq/build index 8a22575..b5738bc 100755 --- a/container/dnsmasq/build +++ b/container/dnsmasq/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "dnsmasq:alpine" + [[ -e /tftpboot/boot.ipxe ]] || { cp tftpboot/boot.ipxe /tftpboot/boot.ipxe } diff --git a/container/dracut-initrd/build b/container/dracut-initrd/build index 181b896..f02cea9 100755 --- a/container/dracut-initrd/build +++ b/container/dracut-initrd/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:dracut" + docker build -t debian:dracut . diff --git a/container/es/build b/container/es/build index 17a225c..c7d7115 100755 --- a/container/es/build +++ b/container/es/build @@ -5,6 +5,8 @@ require_relative '../defconfig.rb' +docker_skip_rebuild "es643b:alpine311" + available_memory = get_available_memory system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." diff --git a/container/extract-stats/build b/container/extract-stats/build index 1aa9982..c315f0c 100755 --- a/container/extract-stats/build +++ b/container/extract-stats/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "extract-stats" + CCI_SRC=$CCI_SRC bash $CCI_SRC/container/compile . diff --git a/container/fetch-mail/build b/container/fetch-mail/build index e72bed8..9b1a501 100755 --- a/container/fetch-mail/build +++ b/container/fetch-mail/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "fetch-mail:latest" + docker build -t fetch-mail:latest . diff --git a/container/fluentd-base/build b/container/fluentd-base/build index 32e0a38..72b85bf 100755 --- a/container/fluentd-base/build +++ b/container/fluentd-base/build @@ -3,4 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "fluentd-base:alpine" + system 'docker build -t fluentd-base:alpine .' diff --git a/container/git-daemon/build b/container/git-daemon/build index d81e4b9..5ba9a52 100755 --- a/container/git-daemon/build +++ b/container/git-daemon/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "apline311:git-daemon" + docker build -t apline311:git-daemon . diff --git a/container/git-mirror/build b/container/git-mirror/build index 66967e4..c302ac7 100755 --- a/container/git-mirror/build +++ b/container/git-mirror/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine38:git-mirror" + docker build -t alpine38:git-mirror . diff --git a/container/gitcache/build b/container/gitcache/build index 5135f7c..d7f3523 100755 --- a/container/gitcache/build +++ b/container/gitcache/build @@ -3,4 +3,8 @@ # Copyright (C) 2019-2020 Eric D.little51 # SPDX-License-Identifier: GPL-3.0 +. ../defconfig.sh + +docker_skip_rebuild "gitcache" + docker build -t gitcache . diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build index 4b93647..19d5125 100755 --- a/container/initrd-cifs/build +++ b/container/initrd-cifs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "initrd-cifs" + docker build -t initrd-cifs . diff --git a/container/initrd-http/build b/container/initrd-http/build index 1b039dc..22babcd 100755 --- a/container/initrd-http/build +++ b/container/initrd-http/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "initrd-http" + docker build -t initrd-http . diff --git a/container/kibana/build b/container/kibana/build index 52d5a2a..9a1fb6d 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -3,6 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "kibana:7.6.2" + BASE_IMAGE_DICT = { 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', 'x86_64' => 'kibana:7.6.2' }.freeze diff --git a/container/lifecycle/build b/container/lifecycle/build index bfb7490..00a9c84 100755 --- a/container/lifecycle/build +++ b/container/lifecycle/build @@ -11,6 +11,8 @@ defaults = relevant_defaults(names) LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11312' +docker_skip_rebuild "lifecycle-#{LIFECYCLE_PORT}" + CCI_SRC = ENV['CCI_SRC'] system "bash #{CCI_SRC}/container/compile ." system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." diff --git a/container/lkp-initrd/build b/container/lkp-initrd/build index 2383323..cfba617 100755 --- a/container/lkp-initrd/build +++ b/container/lkp-initrd/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:lkp" + docker build -t alpine:lkp . diff --git a/container/logging-es/build b/container/logging-es/build index 6465ace..3be841a 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -5,6 +5,8 @@ require_relative '../defconfig.rb' +docker_skip_rebuild "logging-es:7.6.2" + BASE_IMAGE_DICT = { 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', 'x86_64' => 'elasticsearch:7.6.2' diff --git a/container/mail-robot/build b/container/mail-robot/build index c564add..4483e89 100755 --- a/container/mail-robot/build +++ b/container/mail-robot/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "mail-robot:latest" + docker build -t mail-robot:latest . diff --git a/container/manjaro/build b/container/manjaro/build index 114e550..6aeb3c6 100755 --- a/container/manjaro/build +++ b/container/manjaro/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "manjaro:testbed" + declare -A BASE_IMAGE_DICT BASE_IMAGE_DICT=( diff --git a/container/master-fluentd/build b/container/master-fluentd/build index 7d0291d..987cfcd 100755 --- a/container/master-fluentd/build +++ b/container/master-fluentd/build @@ -3,4 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "master-fluentd:alpine" + system 'docker build -t master-fluentd:alpine .' diff --git a/container/minio/build b/container/minio/build index d1ccd06..b0909e9 100755 --- a/container/minio/build +++ b/container/minio/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "jessestuart/minio" + docker pull jessestuart/minio diff --git a/container/monitoring/build b/container/monitoring/build index bcb0c37..121ede0 100755 --- a/container/monitoring/build +++ b/container/monitoring/build @@ -11,6 +11,8 @@ defaults = relevant_defaults(names) MONITOR_PORT = (defaults['MONITOR_PORT'] || '11310') +docker_skip_rebuild "monitoring-#{MONITOR_PORT}" + CCI_SRC = ENV['CCI_SRC'] system "bash #{CCI_SRC}/container/compile ." system "docker build -t monitoring-#{MONITOR_PORT} ." diff --git a/container/netdata-slave/build b/container/netdata-slave/build index 1a84550..cb53364 100755 --- a/container/netdata-slave/build +++ b/container/netdata-slave/build @@ -2,4 +2,8 @@ # Origin: https://hub.docker.com/r/netdata/netdata # SPDX-License-Identifier: GPL-3.0+ +. ../defconfig.sh + +docker_skip_rebuild "netdata/netdata-slave" + docker build -t netdata/netdata-slave . diff --git a/container/netdata/build b/container/netdata/build index 809b445..6aa9a17 100755 --- a/container/netdata/build +++ b/container/netdata/build @@ -2,4 +2,8 @@ # Origin: https://hub.docker.com/r/netdata/netdata # SPDX-License-Identifier: GPL-3.0+ +. ../defconfig.sh + +docker_skip_rebuild "netdata/netdatalkp" + docker build -t netdata/netdatalkp . diff --git a/container/ntp-server/build b/container/ntp-server/build index 7826fb2..d52f9dd 100755 --- a/container/ntp-server/build +++ b/container/ntp-server/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "ntp-server" + docker build -t ntp-server . diff --git a/container/openresty-proxy-cache/build b/container/openresty-proxy-cache/build index 507c4fe..7665209 100755 --- a/container/openresty-proxy-cache/build +++ b/container/openresty-proxy-cache/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "openresty:proxy_cache" + [ -d "openresty" ] && rm -rf openresty git clone https://gitee.com/cuiyili/openresty.git || exit 1 diff --git a/container/os-cifs/build b/container/os-cifs/build index 77be8e9..de44c44 100755 --- a/container/os-cifs/build +++ b/container/os-cifs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "os-cifs" + docker build -t os-cifs . diff --git a/container/os-http/build b/container/os-http/build index c1cb008..367748e 100755 --- a/container/os-http/build +++ b/container/os-http/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "os-http" + docker build -t os-http . diff --git a/container/os-nfs/build b/container/os-nfs/build index eea5a7d..e0fd476 100755 --- a/container/os-nfs/build +++ b/container/os-nfs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "os-nfs" + docker build -t os-nfs . diff --git a/container/qcow2rootfs/build b/container/qcow2rootfs/build index 565de24..7f31bb4 100755 --- a/container/qcow2rootfs/build +++ b/container/qcow2rootfs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "qcow2rootfs:auto" + docker build -t qcow2rootfs:auto . diff --git a/container/qemu-efi/build b/container/qemu-efi/build index cac9426..4f1d488 100755 --- a/container/qemu-efi/build +++ b/container/qemu-efi/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:qemu-efi" + [ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 [ $(arch) == "x86_64" ] && uefi=ovmf diff --git a/container/rabbitmq/build b/container/rabbitmq/build index c0bc179..930c112 100755 --- a/container/rabbitmq/build +++ b/container/rabbitmq/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "rabbitmq:3-management" + docker pull rabbitmq:3-management diff --git a/container/redis/build b/container/redis/build index e5230b0..490f99d 100755 --- a/container/redis/build +++ b/container/redis/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "redis507n:alpine311" + docker build -t redis507n:alpine311 . diff --git a/container/register-accounts/build b/container/register-accounts/build index e1dcfa4..2364b34 100755 --- a/container/register-accounts/build +++ b/container/register-accounts/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "register-accounts" + docker build -t register-accounts . diff --git a/container/registry/build b/container/registry/build index ca8680a..2edc2c1 100755 --- a/container/registry/build +++ b/container/registry/build @@ -2,4 +2,8 @@ # Origin: https://docs.docker.com/registry # SPDX-License-Identifier: Apache license +. ../defconfig.sh + +docker_skip_rebuild "registry" + docker pull registry diff --git a/container/remote-git/build b/container/remote-git/build index dad1a9b..727c7f9 100755 --- a/container/remote-git/build +++ b/container/remote-git/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:remote-git" + docker build -t debian:remote-git . diff --git a/container/result-cifs/build b/container/result-cifs/build index d97b6dd..fd9d2df 100755 --- a/container/result-cifs/build +++ b/container/result-cifs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "result-cifs" + docker build -t result-cifs . diff --git a/container/result-webdav/build b/container/result-webdav/build index 7d70ae6..b5be9e9 100755 --- a/container/result-webdav/build +++ b/container/result-webdav/build @@ -4,4 +4,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "result-webdav" + docker build -t result-webdav . diff --git a/container/rsync-server/build b/container/rsync-server/build index 72bfc47..5d36265 100755 --- a/container/rsync-server/build +++ b/container/rsync-server/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "rsync_server" + docker build -t rsync_server . diff --git a/container/scheduler-dev/build b/container/scheduler-dev/build index 1e4c57e..56d27c0 100755 --- a/container/scheduler-dev/build +++ b/container/scheduler-dev/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:scheduler-dev" + docker build -t alpine:scheduler-dev . diff --git a/container/scheduler/build b/container/scheduler/build index 367aa4e..412988a 100755 --- a/container/scheduler/build +++ b/container/scheduler/build @@ -11,6 +11,8 @@ defaults = relevant_defaults(names) SCHED_PORT = (defaults['SCHED_PORT'] || '3000') +docker_skip_rebuild "sch-ruby-a:v0.00d-#{SCHED_PORT}" + CCI_SRC = ENV['CCI_SRC'] system "bash #{CCI_SRC}/container/compile ." system "cp #{CCI_SRC}/sbin/create-job-cpio.sh ." diff --git a/container/send-internet-mail/build b/container/send-internet-mail/build index 078adaf..07ca143 100755 --- a/container/send-internet-mail/build +++ b/container/send-internet-mail/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "send-internet-mail:latest" + docker build -t send-internet-mail:latest . diff --git a/container/send-mail/build b/container/send-mail/build index 9159641..8ae9788 100755 --- a/container/send-mail/build +++ b/container/send-mail/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "send-mail:latest" + docker build -t send-mail:latest . diff --git a/container/shellcheck/build b/container/shellcheck/build index 854bae6..f295e51 100755 --- a/container/shellcheck/build +++ b/container/shellcheck/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "shellcheck" + docker build -t shellcheck . diff --git a/container/srv-http/build b/container/srv-http/build index 3a500ea..eb0dcbc 100755 --- a/container/srv-http/build +++ b/container/srv-http/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "srv-http" + docker build -t srv-http . diff --git a/container/ssh-r/build b/container/ssh-r/build index 4bec543..2ad1bd3 100755 --- a/container/ssh-r/build +++ b/container/ssh-r/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "ssh-r:0.001" + if [ ! -f "known_auth_keys" ]; then echo "-----> ERROR: no testbox rsa public key file 'known_auth_keys' at currrent DIR" exit 1 diff --git a/container/sub-fluentd/build b/container/sub-fluentd/build index 797cd0d..1ec2246 100755 --- a/container/sub-fluentd/build +++ b/container/sub-fluentd/build @@ -3,4 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "sub-fluentd:alpine" + system 'docker build -t sub-fluentd:alpine .' diff --git a/container/submit/build b/container/submit/build index fba4281..dff884a 100755 --- a/container/submit/build +++ b/container/submit/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "submit:latest" + docker build -t submit:latest . diff --git a/container/taskqueue/build b/container/taskqueue/build index 3a5d4c9..4f69f85 100755 --- a/container/taskqueue/build +++ b/container/taskqueue/build @@ -10,6 +10,8 @@ load_cci_defaults service_port=${TASKQUEUE_PORT:-3060} +docker_skip_rebuild "taskqueue-alpine:v0.001-${service_port}" + if [[ ! -f "taskqueue" ]]; then $CCI_SRC/container/compile . else diff --git a/container/upload-libvirt-xml/build b/container/upload-libvirt-xml/build index d39838b..0a22173 100755 --- a/container/upload-libvirt-xml/build +++ b/container/upload-libvirt-xml/build @@ -4,6 +4,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "upload-libvirt-xml" + # The libvirt xml upload has nothing to do with the job. It renames the uploaded file and does not currently have permission to upload. # it is a transitional service. Two different functions use the same port service, diff --git a/container/web-backend-nginx/build b/container/web-backend-nginx/build index 4c9d3fe..d282570 100755 --- a/container/web-backend-nginx/build +++ b/container/web-backend-nginx/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "web-backend-nginx" + docker build -t web-backend-nginx . diff --git a/container/web-backend/build b/container/web-backend/build index 671fc97..c7f7d8a 100755 --- a/container/web-backend/build +++ b/container/web-backend/build @@ -1,4 +1,8 @@ #!/bin/sh # SPDX-License-Identifier: MulanPSL-2.0+ +. ../defconfig.sh + +docker_skip_rebuild "debian:web-backend" + docker build -t debian:web-backend . diff --git a/container/webhook/build b/container/webhook/build index bb5c0a7..a3475c4 100755 --- a/container/webhook/build +++ b/container/webhook/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "node:webhook" + docker build -t node:webhook . -- 2.23.0

2 1

[PATCH lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 09 Feb '21

09 Feb '21

[How] ansible_stats_openeuler < ${job_output} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 102 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 92 insertions(+), 10 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..556d4bcdd 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,97 @@ #!/usr/bin/env ruby +require 'json' + while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line + line = line.gsub(/'/, ' ') + ansible_stats_error = `echo '#{line}' |grep -E 'FAILED! |failed: .*}$' | awk -F'=>' '{$1="";print $0}'` + next if ansible_stats_error.empty? + + a = JSON.load(ansible_stats_error) + +# a['msg'].each_line do + unless a['msg'].nil? + + # When running some plug-in, the file was not found + if a['msg'].match(/No file was found when using (.+)\./) + puts "files.error.no-file-was-found-when-using.#{$1}: 1" + puts "files.error.no-file-was-found-when-using.#{$1}.message: #{ansible_stats_error}" + end + + # No such command + if a['msg'].match(/No such file or directory: b (.+) :/) + puts "command.error.no-such-file-or-directory.#{$1}: 1" + puts "command.error.no-such-file-or-directory.#{a['cmd']}.message: #{ansible_stats_error}" + end + + # The URL of the RPM is invalid + if a['msg'].match(/Failure downloading (.+), /) + puts "download.error.http-error-404 [#{$1}]: 1" + puts "download.error.http-error-404 [#{$1}]: #{ansible_stats_error}" + end + + # Failed to download the yum repodata + if a['msg'].match(/Failed to download metadata for repo (.+) :/) + puts "download.error.failed-to-download-metadata.#{$1}: 1" + puts "download.error.failed-to-download-metadata.#{$1}.message: #{ansible_stats_error}" + end + + if a['msg'].match(/Failed to download packages: Cannot download (.+):/) + puts "download.error.failed-to-download-package.#{$1}: 1" + puts "download.error.failed-to-download-package.#{$1}.message: #{ansible_stats_error}" + end + + # The task includes an option with an undefined variable. + if a['msg'].match(/: (.+) is undefined/) + puts "syntax.error.an-undefined-variable.#{$1}: 1" + puts "syntax.error.an-undefined-variable.#{$1}.message: #{ansible_stats_error}" + end + + # The command runing error. + if a['msg'].match(/non-zero return code/) + puts "command.error[#{a['cmd']}]: 1" + puts "command.error [#{a['cmd']}].message: #{ansible_stats_error}" + end + + # Some service could not fand the requested host. + if a['msg'].match(/Could not find the requested service (.+): host/) + puts "request.error.could-not-find-service-host.#{$1}: 1" + puts "request.error.could-not-find-service-host.#{$1}.messgae: #{ansible_stats_error}" + end + + # Unable to start some services + if a['msg'].match(/Unable to start service (.+):/) + puts "service.error.unable-to-star-service.#{$1}: 1" + puts "service.error.unable-to-star-service.#{$1}.message: #{ansible_stats_error}" + end + + # Some role in not sipported this system + if a['msg'].match(/(.+) is not supported/) + puts "system.error.not-supported.#{$1.gsub(/ /, '-')}: 1" + puts "system.error.not-supported.message: #{ansible_stats_error}" + end + + # failed to create temporary content file: The read operation timed out. + if a['msg'].match(/The read operation timed out/) + puts 'connection.error.read-timed-out: 1' + puts "read.error.read-timed-out.message: #{ansible_stats_error}" + end + end + + unless a['failures'].nil? + a['failures'].each do |items| + # Failed to install some of the specified packages. + items.match(/No package (.+) available/) + puts "yum.error.no-package-match.#{$1}: 1" + puts "yum.error.no-package-match.#{$1}.message: #{line}" + end + end + + next if a['message'].nil? + + # Could not find or access ansible yaml. + if a['message'].to_s.match(/Could not find or access (.+) /) + puts "yaml.error.not-find-yml.#{$1}: 1" + puts "yaml.error.not-find-yml.#{$1}.message: #{ansible_stats_error}" end end -- 2.23.0

1 1

[PATCH lab-z9] idle/*/multi-qemu.yaml: add disk
by Cui Yili 09 Feb '21

09 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- .../multi-qemu.yaml | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml index 513b631..8182a97 100644 --- a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml @@ -1,10 +1,9 @@ suite: multi-qemu category: functional -os: openeuler -os_version: 20.03 -os_arch: aarch64 -os_mount: initramfs +disk: 1HDD +fs: ext4 + simplify-ci: multi-qemu-0: nr_vm: 20 @@ -16,3 +15,8 @@ multi-qemu-2: nr_vm: 2 tbox_group: vm-1p1g sleep: 100d + +os: openeuler +os_version: 20.03 +os_arch: aarch64 +os_mount: initramfs -- 2.23.0

1 0

[PATCH lkp-tests] hosts: add dc-16g
by Cao Xueliang 09 Feb '21

09 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- hosts/dc-16g | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 hosts/dc-16g diff --git a/hosts/dc-16g b/hosts/dc-16g new file mode 100644 index 000000000..1f16373e6 --- /dev/null +++ b/hosts/dc-16g @@ -0,0 +1,3 @@ +nr_cpu: 16 +memory: 16g + -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/es-find: support count keyword
by Lu Weitao 09 Feb '21

09 Feb '21

[Example] es-find tbox_group=vm-2p8g -c/--count suite,job_state { "iperf": { "finished": 50, "failed": 40, "incomplete": 4, "boot": 3 }, "iperf-walk-os-test": { "finished": 167, "failed": 22 }, "borrow": { "finished": 47, "incomplete": 17, "failed": 2 }, ... } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/sbin/es-find b/sbin/es-find index ed54955..2c3b3dc 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -6,6 +6,7 @@ # Usage: # es-find key=val ... # es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." +# es-find tbox_group=vm-2p8g -c suite,job_state # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -74,6 +75,7 @@ end options = { nr_jobs: 10, desc_keyword: 'start_time' } range = nil +count = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -97,6 +99,10 @@ opt_parser = OptionParser.new do |opts| range = r end + opts.on('-c', '--count KEYWORD', 'count keyword: suite|job_state|testbox|id|...') do |c| + count = c + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -108,6 +114,11 @@ items = parse_argv items[:range] = prase_range(range) if range es = ESQuery.new(ES_HOST, ES_PORT) -query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) +if count + count_keywords = count.split(',') + query_result = es.query_fields(count_keywords, items) +else + query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) +end json_string = JSON.pretty_generate(query_result) puts json_string -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: fix PKGBUILD url 404
by Cao Xueliang 09 Feb '21

09 Feb '21

[error] http://api.compass-ci.openeuler.org:11300/archlinux/aur-v/vifm-git 404 [how] Need the add git to the url. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5e1103c..d43785b 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -36,7 +36,7 @@ class MailBisectResult prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' bisect_report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" - pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/#{@pkgbuild_repo}\n" + pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/git/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY -- 2.23.0

1 0

[PATCH lkp-tests] setup/simply-ci: fix failed to git clone
by Liu Yinsi 09 Feb '21

09 Feb '21

[error] Cloning into '/c/compass-ci'... fatal: unable to look up (port 9418) (Name or service not known) [how] before: git clone git://$GIT_SERVER/... after: git clone https://$GIT_SERVER/... Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..076c357d9 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -14,7 +14,7 @@ git_ci() EOF fi - git clone git://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git /c/compass-ci + git clone https://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git /c/compass-ci } dev_env() -- 2.23.0

1 0

[PATCH compass-ci] docker/docker.rb: use curl instead of wget
by Lin Jiaxin 09 Feb '21

09 Feb '21

wget -q will turn off wget's output, even error. curl -sS can disable progress meter but still show error messages. So use curl -sS instead of wget -q. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- providers/docker/docker.rb | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 56c9754..f4e954e 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -49,8 +49,9 @@ def parse_response(url) return hash end -def wget_cmd(path, url, name) - system "wget -q -P #{path} #{url} && gzip -dc #{path}/#{name} | cpio -id -D #{path}" + +def curl_cmd(path, url, name) + system "curl -sS --create-dirs -o #{path}/#{name} #{url} && gzip -dc #{path}/#{name} | cpio -id -D #{path}" end def build_load_path(hostname) @@ -75,8 +76,8 @@ def load_initrds(load_path, hash) arch = RUBY_PLATFORM.split('-')[0] job_url = hash['job'] lkp_url = hash['lkp'] - wget_cmd(load_path, job_url, 'job.cgz') - wget_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") + curl_cmd(load_path, job_url, 'job.cgz') + curl_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") end def start_container(hostname, load_path, hash) -- 2.23.0

2 1

[PATCH compass-ci] sbin/auto_submit: add support for different branch different job
by Li Yuanchao 08 Feb '21

08 Feb '21

Now can support some branch use specific submit commands and others use common submit commands for the same repo. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 78 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 62 insertions(+), 16 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 410445d..6d29543 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -30,21 +30,6 @@ class AutoSubmit return pkgbuild_repo end - def submit(newrefs_info, submit_argv) - newrefs_info['new_refs']['heads'].each do |branch, commit_id| - inactive_time = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git log --pretty=format:"%cr" -1 #{commit_id}) - next if inactive_time =~ /(month|year)/ - - real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") - real_argvs.push("upstream_commit=#{commit_id}") - tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) - real_argvs.push("upstream_tag=#{tag}") unless tag.empty? - - system(real_argvs.join(' ')) - end - end - def get_argvs(newrefs_info) git_repo = newrefs_info['git_repo'] puts "git_repo : #{git_repo}" @@ -60,16 +45,77 @@ class AutoSubmit submit_argv end + def submit_one_job(submit_argv, git_repo, branch, commit_id) + inactive_time = %x(git -C /srv/git/#{git_repo}.git log --pretty=format:"%cr" -1 #{commit_id}) + return if inactive_time =~ /(month|year)/ + + real_argvs = Array.new(submit_argv) + real_argvs.push("upstream_branch=#{branch}") + real_argvs.push("upstream_commit=#{commit_id}") + tag = %x(git -C /srv/git/#{git_repo}.git tag --points-at #{commit_id}) + real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + + system(real_argvs.join(' ')) + end + + def get_branch_commands(submit_commands) + submit_commands.each do |element| + return element['branch_commands'] if element['branch_commands'] + end + return nil + end + + def submit_specific_branch(submit_argv, newrefs_info, branch_commands) + return newrefs_info['new_refs']['heads'] if branch_commands.nil? + + non_specific_commits = {} + newrefs_info['new_refs']['heads'].each do |branch, commit_id| + branch_name = branch.delete_prefix('refs/heads/') + if branch_commands[branch_name].nil? + non_specific_commits[branch] = commit_id + next + end + argvs = Array.new(submit_argv) + argvs.push(branch_commands[branch_name]) + submit_one_job(argvs, newrefs_info['git_repo'], branch_name, commit_id) + end + return non_specific_commits + end + + def submit_non_specific(non_specific_commits, git_repo, submit_argv) + non_specific_commits.each do |branch, commit_id| + branch = branch.delete_prefix('refs/heads/') + submit_one_job(submit_argv, git_repo, branch, commit_id) + end + end + + # Add support of different branch different job + # newrefs_info['submit'] is like: + # ["submit command 1", + # "submit command 2", + # ..., + # { "branch_commands" => { "branch_1" => "submit command 1", + # ..., + # "branch_n" => "submit command n" + # } + # } + # ] def submit_job(newrefs_info) submit_argv = get_argvs(newrefs_info) return unless submit_argv return unless newrefs_info['submit'] + branch_commands = get_branch_commands(newrefs_info['submit']) + non_specific_commits = submit_specific_branch(submit_argv, newrefs_info, branch_commands) + return if non_specific_commits.empty? + newrefs_info['submit'].each do |argv_config| + next if argv_config.is_a? Hash + argvs = Array.new(submit_argv) argvs.push(argv_config) - submit(newrefs_info, argvs) + submit_non_specific(non_specific_commits, newrefs_info['git_repo'], argvs) end end -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] container/defconfig: add function docker_skip_rebuild
by Liu Yinsi 08 Feb '21

08 Feb '21

When execute `sparrow/4-docker/buildall run-only`, if docker image already exists, just exit build script, not rebuild container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 6 ++++++ container/defconfig.sh | 8 ++++++++ 2 files changed, 14 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index f9b0824..7dc9f33 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -47,3 +47,9 @@ def get_available_memory # take the middle value according to the system memory size. [1024, 30720, Math.sqrt(memtotal) * 1024].sort[1].to_i end + +def docker_skip_rebuild(tag) + return unless ENV['action'].include? 'run-only' + + exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" +end diff --git a/container/defconfig.sh b/container/defconfig.sh index 54e65b5..7c575ef 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -41,3 +41,11 @@ push_image() docker tag "$src_tag" "$dst_tag" docker push "$dst_tag" } + +docker_skip_rebuild() +{ + tag=$1 + [ "$action" != "run-only" ] && return + docker image inspect $tag > /dev/null 2>&1 + [ "$?" == "0" ] && exit 1 +} -- 2.23.0

1 0

[PATCH v4 compass-ci] container/register-accounts: fix failed to register account
by Liu Yinsi 08 Feb '21

08 Feb '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/register-accounts# ./start register-accounts register-accounts /c/compass-ci/container/register-accounts # ./register-accounts awk: /etc/compass-ci/defaults/*.yaml: No such file or directory "option: {\"my_email\"=>\"\", \"my_name\"=>\"\", \"lab\"=>\"\"}" Traceback (most recent call last): 10: from ./register-accounts:8:in `<main>' 9: from ./register-accounts:8:in `require_relative' 8: from /c/compass-ci/sbin/build-my-info.rb:57:in `<top (required)>' 7: from /c/compass-ci/sbin/build-my-info.rb:48:in `build_my_info' 6: from /c/compass-ci/lib/build_my_info_client.rb:30:in `search_my_info' 5: from /c/compass-ci/lib/es_query.rb:58:in `query_by_id' 4: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-api-7.10.1/lib/elasticsearch/api/actions/search.rb:103:in `search' 3: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/client.rb:182:in `perform_request' 2: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request' 1: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/base.rb:347:in `perform_request' /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error': [400] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: {\\n \\"bool\\" : {\\n \\"must\\" : [\\n {\\n \\"term\\" : {\\n \\"_id\\" : {\\n \\"value\\" : \\"\\",\\n \\"boost\\" : 1.0\\n }\\n }\\n }\\n ],\\n \\"adjust_pure_negative\\" : true,\\n \\"boost\\" : 1.0\\n }\\n}","index_uuid":"MH_AlDBvTYa9sQhwsESXgQ","index":"accounts"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"accounts","node":"I1usO2axQ9-XZhzcX3oHxw","reason":{"type":"query_shard_exception","reason":"failed to create query: {\\n \\"bool\\" : {\\n \\"must\\" : [\\n {\\n \\"term\\" : {\\n \\"_id\\" : {\\n \\"value\\" : \\"\\",\\n \\"boost\\" : 1.0\\n }\\n }\\n }\\n ],\\n \\"adjust_pure_negative\\" : true,\\n \\"boost\\" : 1.0\\n }\\n}","index_uuid":"MH_AlDBvTYa9sQhwsESXgQ","index":"accounts","caused_by":{"type":"illegal_argument_exception","reason":"Ids can't be empty"}}}]},"status":400} (Elasticsearch::Transport::Transport::Errors::BadRequest) refer code: /c/compass-ci/sbin/build-my-info.rb: ... option = { 'my_email' => `git config --global user.email`.chomp, 'my_name' => `git config --global user.name`.chomp, 'lab' => `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp } ... build_my_info(option) ... [why] when call build-my-info.rb, we just want call function build_my_info(), no need to execute option, in caller script already define option values [how] use `if $PROGRAM_NAME == __FILE__`, when sbin/build-my-info.rb is not executed independently but invoked by other files, avoid to execute some part code. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/build-my-info.rb | 72 ++++++++++++++++++++++--------------------- 1 file changed, 37 insertions(+), 35 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index dbe093d..c7a7798 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -7,49 +7,51 @@ require_relative '../lib/es_client' require_relative '../lib/build_my_info_client' require 'optparse' -option = { - 'my_email' => `git config --global user.email`.chomp, - 'my_name' => `git config --global user.name`.chomp, - 'lab' => `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def build_my_info(option) + build_my_info = BuildMyInfo.new(option['my_email']) + info_es = build_my_info.search_my_info + info_es.update option + info_es['my_token'] = %x(uuidgen).chomp if info_es['my_token'].nil? -options = OptionParser.new do |opts| - opts.banner = "Usage: build-my-info [-e email] [-n name] [-l lab] [-t]\n" + build_my_info.config_my_info(info_es) +end - opts.separator '' - opts.separator 'options:' +if $PROGRAM_NAME == __FILE__ + option = { + 'my_email' => `git config --global user.email`.chomp, + 'my_name' => `git config --global user.name`.chomp, + 'lab' => `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp + } - opts.on('-e email', 'my_email') do |email| - option['my_email'] = email - end + options = OptionParser.new do |opts| + opts.banner = "Usage: build-my-info [-e email] [-n name] [-l lab] [-t]\n" - opts.on('-n name', 'my_name') do |name| - option['my_name'] = name - end + opts.separator '' + opts.separator 'options:' - opts.on('-l lab', 'lab') do |lab| - option['lab'] = lab - end + opts.on('-e email', 'my_email') do |email| + option['my_email'] = email + end - opts.on('-t', 'my_token') do - option['my_token'] = %x(uuidgen).chomp - end + opts.on('-n name', 'my_name') do |name| + option['my_name'] = name + end - opts.on_tail('-h', 'show this message') do - puts opts - exit - end -end + opts.on('-l lab', 'lab') do |lab| + option['lab'] = lab + end -options.parse! + opts.on('-t', 'my_token') do + option['my_token'] = %x(uuidgen).chomp + end -def build_my_info(option) - build_my_info = BuildMyInfo.new(option['my_email']) - info_es = build_my_info.search_my_info - info_es.update option - info_es['my_token'] = %x(uuidgen).chomp if info_es['my_token'].nil? + opts.on_tail('-h', 'show this message') do + puts opts + exit + end + end - build_my_info.config_my_info(info_es) -end + options.parse! -build_my_info(option) + build_my_info(option) +end -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies.sh: delete redundant code
by Wang Yong 08 Feb '21

08 Feb '21

write_shell_profile function will source profile, no need source twice Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- sbin/install-dependencies.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index ed8aeb927..b687dc0d6 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -65,7 +65,6 @@ set_env() write_host write_shell_profile "export LKP_SRC=$PWD" write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" - source $(shell_profile) } set_env -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/install-dependencies.sh: change install way
by Wang Yong 08 Feb '21

08 Feb '21

[before] install packages from $DISTRO detect [after] install packages from Package Manager it's not convenient when install packages in a new $DISTRO Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- sbin/install-dependencies.sh | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index b5b070f01..ed8aeb927 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -5,28 +5,36 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh -# choose install function base on DISTRIBUTION +# choose install function base on common Package Manager linux_dep() { - source /etc/os-release - case $ID in - ubuntu|debian) + get_package_manager + + case "$installer" in + apt-get) export DEBIAN_FRONTEND=noninteractive - sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + sudo "$installer" install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + ;; + dnf|yum) + sudo "$installer" install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux ;; - openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - sudo dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - else - sudo yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - fi + pacman) + sudo "$installer" -Sy --noconfirm --needed ruby rubygems gcc make git lftp util-linux ;; *) - echo "$ID not support! please install dependencies manually." && exit 1 + echo "Unknown Package Manager! please install dependencies manually." && exit 1 ;; esac } +get_package_manager() +{ + has_cmd "yum" && installer="yum" + has_cmd "dnf" && installer="dnf" && return + has_cmd "apt-get" && installer="apt-get" && return + has_cmd "pacman" && installer="pacman" && return +} + mac_dep() { brew install ruby -- 2.23.0

2 2