- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests/build-pkg: fix to generate the directory of *.cgz file
by Liu Shaofei 31 Dec '20

31 Dec '20

[why] Since the file upload path saved by ES is inconsistent with the file path generated by the build-pkg script, the *.cgz file upload failed. [error] ==> /tmp/stdout <== renamed '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz.tmp' -> '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz' ==> Leaving fakeroot environment. ==> Finished making: kubernetes 1.18.5-1 (Wed Dec 30 16:50:17 CST 2020) create soft link: latest.cgz -> HEAD.cgz ==> /tmp/stderr <== curl: (22) The requested URL returned error: 403 Forbidden Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 801649cf4..e21397ee6 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -88,6 +88,9 @@ request_pkg() create_cgz_name() { + pkgname="${pkgname}" + [ -n "${pkgbase}" ] && pkgname="${pkgbase}" + cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" pkg_args="-A --check --skippgpcheck" -- 2.23.0

3 5

[PATCH compass-ci] doc: add runtime field in borrow-*.yml and update borrow-machine.*.md
by Zhang Yale 31 Dec '20

31 Dec '20

User can specify the borrow time in "runtime" field Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 7 ++++--- doc/manual/borrow-machine.zh.md | 9 +++++---- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index bed9fcc..b598bec 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -203,6 +203,7 @@ Ensure that you have performed the following operations according to the [apply- suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 3f500e2..5288377 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -206,7 +206,8 @@ hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + # ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v3 lkp-tests] optimize the deploy-cci script
by Hu Xuejiao 31 Dec '20

31 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 52 ++++++++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 55835544..8e6acbac 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -2,30 +2,38 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# redirect remote-repository address to "/srv/git/gitee.com" -if [ -n "$GIT_SERVER" ]; then - cat >> /etc/gitconfig <<-EOF - [url "git://$GIT_SERVER/gitee.com"] - insteadOf=https://gitee.com - EOF -fi +export_env() +{ + export initrd_http_host=$INITRD_HTTP_HOST + export initrd_http_port=$INITRD_HTTP_PORT + export gitcache_host=$GITCACHE_HOST + export gitcache_port=$GITCACHE_PORT +} -export initrd_http_host=$INITRD_HTTP_HOST -export initrd_http_port=$INITRD_HTTP_PORT -export gitcache_host=$GITCACHE_HOST -export gitcache_port=$GITCACHE_PORT +deploy() +{ + # redirect remote-repository address to "/srv/git/gitee.com" + if [ -n "$GIT_SERVER" ]; then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF + fi -mkdir -p /c -cd /c || exit -git clone https://gitee.com/wu_fengguang/compass-ci.git + mkdir -p /c && cd /c + git clone https://gitee.com/wu_fengguang/compass-ci.git -cd /c/compass-ci || exit -[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + cd /c/compass-ci || exit + [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" -# Need mount tmpfs to fix compass-ci/container/os-nfs/start error -# "exportfs: /exports/result does not support NFS export" -mount -t tmpfs none /srv + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error + # "exportfs: /exports/result does not support NFS export" + mount -t tmpfs none /srv -# test the script for one-click deployment -cd /c/compass-ci/sparrow || exit -./install-tiny + # test the script for one-click deployment + cd /c/compass-ci/sparrow || exit + ./install-tiny +} + +export_env +deploy -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] pkg: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/httpress/PKGBUILD | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 pkg/httpress/PKGBUILD diff --git a/pkg/httpress/PKGBUILD b/pkg/httpress/PKGBUILD new file mode 100644 index 00000000..8891d6c7 --- /dev/null +++ b/pkg/httpress/PKGBUILD @@ -0,0 +1,17 @@ +pkgname=httpress +pkgver=1 +pkgrel=1 +url='http://ftp.us2.freshrpms.net/linux/RPM/epel/7/aarch64/Packages/h/httpress-1…' +arch=('aarch64') +license=('BSD') +source=("https://download.fedoraproject.org/pub/epel/7/SRPMS/Packages/h/httpress-1.1…") +md5sums=('SKIP') + +package() +{ + # The rpm from source maybe not be available + # httpress-1.1.0-13.fc31.aarch64.rpm has been validated to be available + rpmpkg_path="${pkgdir}/opt/rpms/" + mkdir -p "${rpmpkg_path}" + cp "${srcdir}/"*.rpm "${rpmpkg_path}" +} -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] distro/depends: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/httpress | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/httpress diff --git a/distro/depends/httpress b/distro/depends/httpress new file mode 100644 index 00000000..73fc57c6 --- /dev/null +++ b/distro/depends/httpress @@ -0,0 +1,2 @@ +numactl +ethtool -- 2.23.0

1 0

[PATCH compass-ci] lib/es_query.rb: parse es_result to a Hash result
by Lu Kaiyi 31 Dec '20

31 Dec '20

[why] make es result more easy access. [example] input: result_hash = {} es_result = [{"key"=>"build-pkg", "doc_count"=>354526, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>5708}, {"key"=>"finished", "doc_count"=>1033}, {"key"=>"incomplete", "doc_count"=>204}, {"key"=>"submit", "doc_count"=>136}, ... output: result_hash = {"build-pkg"=> {"failed"=>5708, "finished"=>1033, "incomplete"=>204, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1785, "failed"=>675, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 29d50e9..dcd735b 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -129,7 +129,9 @@ class ESQuery result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] return nil if result.empty? - result + result_hash = {} + parse_fields(result, result_hash) + result_hash end end @@ -217,3 +219,46 @@ def build_aggs_from_fields(fields, aggs_hash) build_aggs_from_fields(fields, aggs_hash['aggs']["all_#{field}"]) end + +# input: +# result_hash = {} +# es_result = +# [{"key"=>"build-pkg", +# "doc_count"=>354526, +# "all_job_state"=> +# {"doc_count_error_upper_bound"=>0, +# "sum_other_doc_count"=>0, +# "buckets"=> +# [{"key"=>"failed", "doc_count"=>5708}, +# {"key"=>"finished", "doc_count"=>1033}, +# {"key"=>"incomplete", "doc_count"=>204}, +# {"key"=>"submit", "doc_count"=>136}, +# ... +# +# output: +# result_hash = +# {"build-pkg"=> +# {"failed"=>5708, +# "finished"=>1033, +# "incomplete"=>204, +# "submit"=>136, +# "OOM"=>11, +# "post_run"=>2}, +# "cci-depends"=> +# {"finished"=>1785, +# "failed"=>675, +# ... +def parse_fields(es_result, result_hash) + es_result.each do |result| + key = result['key'] + sub_field = result.keys.detect { |field| field.start_with?('all_') } + + if sub_field + result_hash[key] = {} + all_field = result[sub_field]['buckets'] + parse_fields(all_field, result_hash[key]) + else + result_hash[key] = result['doc_count'] + end + end +end -- 2.23.0

2 1

[PATCH compass-ci] matrix2.rb: filter groups to make compare faster
by Li Yuanchao 31 Dec '20

31 Dec '20

When do compare, after group jobs, there are many one_size group, which can not find another job to make a group. Delete such groups to save time. But, if there is no size > 1 group, 3 one_size groups will be kept to avoid 'No Data'. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/matrix2.rb | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 14e9558..0e80858 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -123,18 +123,37 @@ end def combine_group_query_data(job_list, dims) suites_hash = {} groups = auto_group(job_list, dims) + have_multi_member = multi_member?(groups) + one_size_count = 0 groups.each do |group_key, value| - suite_list = [] - value.each do |dimension_key, jobs| - groups[group_key][dimension_key], suites = create_matrix(jobs) - suite_list.concat(suites) + if value.size < 2 + one_size_count += 1 + if have_multi_member || one_size_count > 3 + groups.delete(group_key) + next + end end - suites_hash[group_key] = suite_list + get_groups_matrix(groups, group_key, value, suites_hash) end - return groups, suites_hash end +def get_groups_matrix(groups, group_key, value, suites_hash) + suite_list = [] + value.each do |dimension_key, jobs| + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) + end + suites_hash[group_key] = suite_list +end + +def multi_member?(groups) + groups.each do |_key, value| + return true if value.size > 1 + end + return false +end + # input: # 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() # 2. params from user's template include: -- 2.23.0

2 3

[PATCH v4 compass-ci] providers/qemu: refactor kvm script
by Liu Yinsi 31 Dec '20

31 Dec '20

encapsulating into functions. test: /srv/result/host-info/2020-12-30/vm-2p8g/openeuler-20.09-aarch64/z9.275381 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 160 +++++++++++++++++++++++++----------------- 1 file changed, 97 insertions(+), 63 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fd921dc..0845099 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,78 +7,112 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + [ ! -f "$log_file" ] || { + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + } +} + +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm +} -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} +parse_ipxe_script() +{ + append= + initrds= + while read a b c + do + case "$a" in + '#') + ;; + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + *) + ;; + esac + done < $ipxe_script +} -append= -initrds= -while read a b c -do - case "$a" in - '#') - ;; - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - *) - ;; - esac -done < $ipxe_script +check_option_value() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || exit +} -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +get_initrd() +{ + initrd=initrd + cat $initrds > $initrd } -[ -n "$initrds" ] || { - exit +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 } -initrd=initrd -cat $initrds > $initrd +run_qemu() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) + + "${kvm[@]}" --append "${append}" +} -echo $SCHED_PORT -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file +check_logfile +write_logfile -sleep 5 +qemu_command +parse_ipxe_script +check_option_value +get_initrd -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) +print_message -"${kvm[@]}" --append "${append}" +run_qemu -- 2.23.0

2 2

[PATCH v2 lkp-tests] tests/build-pkg: fix to generate the directory of *.cgz file
by Liu Shaofei 31 Dec '20

31 Dec '20

[why] Since the file upload path saved by ES is inconsistent with the file path generated by the build-pkg script, the *.cgz file upload failed. [error] ==> /tmp/stdout <== renamed '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz.tmp' -> '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz' ==> Leaving fakeroot environment. ==> Finished making: kubernetes 1.18.5-1 (Wed Dec 30 16:50:17 CST 2020) create soft link: latest.cgz -> HEAD.cgz ==> /tmp/stderr <== curl: (22) The requested URL returned error: 403 Forbidden Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 801649cf4..436d43840 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -88,6 +88,8 @@ request_pkg() create_cgz_name() { + [ -n "${pkgbase}" ] && pkgname="${pkgbase}" + cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" pkg_args="-A --check --skippgpcheck" -- 2.23.0

1 0

[PATCH lkp-tests] api-avx2neon/PKGBUILD: add multi arch build
by Wang Yong 31 Dec '20

31 Dec '20

build packages through $(arch) Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- pkg/api-avx2neon/PKGBUILD | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/api-avx2neon/PKGBUILD b/pkg/api-avx2neon/PKGBUILD index 3ccebca86..fac047add 100644 --- a/pkg/api-avx2neon/PKGBUILD +++ b/pkg/api-avx2neon/PKGBUILD @@ -9,7 +9,7 @@ md5sums=('SKIP') build() { cd "$srcdir/AvxToNeon/tests" - make + make ARCH=$(arch) } package() { -- 2.23.0

1 0