- Compass-ci - mailweb.openeuler.org

[PATCH v7 compass-ci 2/2] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 19 Nov '20

19 Nov '20

[why] Avoid to execute some unnecessary operations and improve performance. [how] Keep the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 62d174c..86cfc93 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

2 2

[PATCH lkp-tests] borrow-1h.yaml: fix unable to receive email
by Zhang Yale 19 Nov '20

19 Nov '20

[why] $my_email is used in sshd, so borrow_1h.yaml should use my_email field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-1h.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/borrow-1h.yaml b/jobs/borrow-1h.yaml index c9d6ff8e..6c2284f3 100644 --- a/jobs/borrow-1h.yaml +++ b/jobs/borrow-1h.yaml @@ -9,6 +9,6 @@ sshd: nil end %> - email: + my_email: # sleep at the bottom sleep: 1h -- 2.23.0

2 2

[PATCH v5 compass-ci] sparrow: fix 3-code/dev-env no such file error
by Liu Yinsi 19 Nov '20

19 Nov '20

[why] when execute 3-code/dev-env in openeuler and centos machine, error message as follows: ./dev-env: line 45: ./os/openEuler: No such file or directory the purpose of ". $(dirname ${BASH_SOURCE[0]})/os/${ID}" is to fix crystal-shards build error when adapt in debian. but it's unnecessary for other linux os. Please see: [PATCH compass-ci 3/4] sparrow: add domain name resolution for Docker images [how] hide this redundant and confusion error message. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8928816..08084b0 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -44,4 +44,4 @@ EOF source /etc/os-release -. $(dirname ${BASH_SOURCE[0]})/os/${ID} +. $(dirname ${BASH_SOURCE[0]})/os/${ID} 2> /dev/null -- 2.23.0

3 2

[PATCH v3 lkp-tests] jobs: add new job yaml for lmbench3 performance test
by Bai Jing 19 Nov '20

19 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/lmbench3-performance.yaml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 jobs/lmbench3-performance.yaml diff --git a/jobs/lmbench3-performance.yaml b/jobs/lmbench3-performance.yaml new file mode 100644 index 00000000..76e5348b --- /dev/null +++ b/jobs/lmbench3-performance.yaml @@ -0,0 +1,17 @@ +suite: lmbench3 +category: benchmark + +test_memory_size: 4096 + +# Test mdoe all, hardware, os were commented temporally, since it will take very long time +# more than 2 hours on test machine lkp-bdw-ep4. +# lmbench3: +# - mode: all +# - mode: hardware +# - mode: os + +nr_threads: +- 1 + +lmbench3: + mode: all -- 2.23.0

1 0

[PATCH compass-ci] container/rabbitmq: separate docker pull and run
by Liu Yinsi 19 Nov '20

19 Nov '20

"docker pull" in build script, "docker run" in start script looks more reasonable. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/rabbitmq/build | 5 +++++ container/rabbitmq/start | 1 - 2 files changed, 5 insertions(+), 1 deletion(-) create mode 100755 container/rabbitmq/build diff --git a/container/rabbitmq/build b/container/rabbitmq/build new file mode 100755 index 0000000..c0bc179 --- /dev/null +++ b/container/rabbitmq/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker pull rabbitmq:3-management diff --git a/container/rabbitmq/start b/container/rabbitmq/start index 7e25417..6b875f3 100755 --- a/container/rabbitmq/start +++ b/container/rabbitmq/start @@ -5,7 +5,6 @@ . $CCI_SRC/container/defconfig.sh docker_rm rabbitmq -docker pull rabbitmq:3-management cmd=( docker run -- 2.23.0

2 1

[PATCH lkp-tests] jobs: add new job yaml for unixbench performance tuning
by Bai Jing 19 Nov '20

19 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/unixbench-tuning.yaml | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 jobs/unixbench-tuning.yaml diff --git a/jobs/unixbench-tuning.yaml b/jobs/unixbench-tuning.yaml new file mode 100644 index 00000000..d18e0dfc --- /dev/null +++ b/jobs/unixbench-tuning.yaml @@ -0,0 +1,10 @@ +suite: unixbench-tuning +category: benchmark + +runtime: 300s + +nr_task: + - 1 + - 96 + +unixbench-tuning: -- 2.23.0

1 0

[PATCH lkp-tests] stats/sysbench-mariadb: fix rubocop offenses
by Zhang Yale 19 Nov '20

19 Nov '20

[errors] Inspecting 1 file C offenses: sysbench-mariadb:4:1: C: Layout/IndentationWidth: Use 2 (not 4) spaces for indentation. case line.chomp! ^^^^ 1 file inspected, 1 offense detected Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/sysbench-mariadb | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/stats/sysbench-mariadb b/stats/sysbench-mariadb index 0c7ba1d9..0bed3bc0 100755 --- a/stats/sysbench-mariadb +++ b/stats/sysbench-mariadb @@ -1,10 +1,10 @@ #!/usr/bin/env ruby while (line = STDIN.gets) - case line.chomp! - when /transactions:\s+(\d+)/ - puts "transactions: #{$1}" - when /95th percentile:\s+(\S+)/ - puts "95th percentile: #{$1}" - end + case line.chomp! + when /transactions:\s+(\d+)/ + puts "transactions: #{$1}" + when /95th percentile:\s+(\S+)/ + puts "95th percentile: #{$1}" + end end -- 2.23.0

3 5

[PATCH lkp-tests] add the case that on_fail is nil
by Hu Xuejiao 19 Nov '20

19 Nov '20

[why] When sh_on_fail is nil, on_fail will add to the job.yaml Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/sbin/submit b/sbin/submit index a325e912..9bba1018 100755 --- a/sbin/submit +++ b/sbin/submit @@ -121,11 +121,14 @@ ARGV.each do |jobfile| 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files, - 'on_fail' => sh_on_fail + 'define_files' => sh_define_files } } + if sh_on_fail + sh_hash['job2sh']['on_fail'] = sh_on_fail + end + # merge job info job_hash = job.to_hash job_hash_list << job_hash -- 2.23.0

2 1

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: disable default dynamic mapping
by Zhang Yuhang 19 Nov '20

19 Nov '20

[why] By default, fields can be added dynamically to a document, or to inner objects within a document, just by indexing a document containing the new field. However, there are many fields that we do not need, and "Limit of total fields [1000] in index [jobs] has been exceeded" exception will be reported when the number of fields exceeds 1000. [how] Disable default dynamic mapping. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 9d420f7..076bdd3 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -38,6 +38,7 @@ else curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ "mappings": { "_doc": { + "dynamic": false, "dynamic_templates": [ { "pp": { @@ -61,6 +62,10 @@ else } ], "properties": { + "pp": { + "dynamic": true, + "properties": {} + }, "suite": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/sysbench-mariadb: fix rubocop offenses
by Zhang Yale 19 Nov '20

19 Nov '20

[errors] Inspecting 1 file C offenses: sysbench-mariadb:4:1: C: Layout/IndentationWidth: Use 2 (not 4) spaces for indentation. case line.chomp! ^^^^ 1 file inspected, 1 offense detected Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/sysbench-mariadb | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/stats/sysbench-mariadb b/stats/sysbench-mariadb index 0c7ba1d9..54a31be0 100755 --- a/stats/sysbench-mariadb +++ b/stats/sysbench-mariadb @@ -1,10 +1,10 @@ #!/usr/bin/env ruby while (line = STDIN.gets) - case line.chomp! - when /transactions:\s+(\d+)/ - puts "transactions: #{$1}" - when /95th percentile:\s+(\S+)/ - puts "95th percentile: #{$1}" - end + case line + when /transactions:\s+(\d+)/ + puts "transactions: #{$1}" + when /95th percentile:\s+(\S+)/ + puts "95th percentile: #{$1}" + end end -- 2.23.0

1 0

[PATCH v8 compass-ci] make the container/initrd-cifs to share /srv/initrd
by Xiao Shenwei 19 Nov '20

19 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result in order to reduce the coupling, need split them [how] one folder mount by one container [usage] two ways for the client to mount initrd: 1. initrd mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage: for lkp compatibility mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 22 ++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 89 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..c2ec808 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +RUN adduser -u 1090 -D lkp +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash +RUN rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..4b93647 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..77558a1 --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

2 2

[PATCH compass-ci] sched: fix cast from Nil to Array failed
by Cao Xueliang 19 Nov '20

19 Nov '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/jobfile_operate.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 16a4d06..98bf011 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -85,6 +85,7 @@ module Jobfile::Operate job_sh_array = [] of JSON::Any tmp_job_sh_content.as_h.each do |_key, val| + next if val == nil job_sh_array += val.as_a end else -- 2.23.0

1 0

[PATCH lkp-tests] tests/cci-makepkg: simplify centos7 test
by Lin Jiaxin 19 Nov '20

19 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 905ad689..8b72f395 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -90,20 +90,21 @@ update_shared_pkg() centos_update_gcc() { - if grep -sqF "CentOS Linux release 7" /etc/centos-release; then - [ "$os_mount" = "initramfs" ] && { - yum install -y centos-release-scl-rh - yum install -y "devtoolset-8-gcc*" - } - source /opt/rh/devtoolset-8/enable - fi + [ "$os" = "centos" ] || return + grep -sqF "CentOS Linux release 7" /etc/centos-release || return + + [ "$os_mount" = "initramfs" ] && { + yum install -y centos-release-scl-rh + yum install -y "devtoolset-8-gcc*" + } + source /opt/rh/devtoolset-8/enable } [ -n "$PKGBUILD_TAG" ] && download_pkgfile distro_install_depends lkp-dev -[ "$os" = "centos" ] && centos_update_gcc +centos_update_gcc distro_install_depends ${benchmark}-dev -- 2.23.0

3 2

[PATCH lkp-tests 6/6] iso2rootfs: allow the use of job.yaml and test script from the internet
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] Not all the yaml and script of test job will be committed to the lkp-tests repo. [How] Support three optional fields for every test: - testX_git_url: the url of git repo which contains yaml and script; - testX_git_yaml: the relative path of the yaml file in the git repo; - testX_git_script: the relative path of the test script in the git repo; Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 5 +++++ tests/iso2rootfs | 25 +++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 3c69a9d98430..c9a31cd0ad00 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -22,3 +22,8 @@ iso2rootfs: test1_yaml: test1_os_mount: test1_testbox: + ## follow three fields is required when your job.yaml and script for + ## this test are from the internet. + test1_git_url: + test1_git_yaml: + test1_git_script: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index a38685118b7f..1ba2e8a31ef8 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -447,10 +447,35 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } +prepare_submit() +{ + local submit_dir="/tmp/iso2rootfs/submit_tmp" + [ -d "${submit_dir}" ] || mkdir -p "${submit_dir}" + local submit_git_dir="${submit_dir}/${test_git_url##*/}" + submit_git_dir="${submit_git_dir%.git}" + + [ -d "${submit_git_dir}" ] || { + cd "${submit_dir}" + git clone "${test_git_url}" + } + + local test_git_yaml="${submit_git_dir}/$(eval echo "\$${test_num}_git_yaml")" + local test_git_script="${submit_git_dir}/$(eval echo "\$${test_num}_git_script")" + + [ -f "${test_git_yaml}" ] || die "cannot find file: ${test_num}_git_yaml." + cp "${test_git_yaml}" "${LKP_SRC}/jobs/" + + [ -f "${test_git_script}" ] || return 0 + cp "${test_git_script}" "${LKP_SRC}/tests/" +} + submit_one_yaml() { log_info "starting submit ${test_yaml} ..." + local test_git_url="$(eval echo "\$${test_num}_git_url")" + [ -z "${test_git_url}" ] || prepare_submit + cd "${LKP_SRC}/jobs" [ -f "${test_yaml}" ] || die "cannot find test yaml in LKP_SRC/jobs. -- 2.23.0

3 6

[PATCH lkp-tests 5/6] iso2rootfs: allow submit multiple test job.yaml for result rootfs
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] Result rootfs of dailybuild requires multiple test jobs. [How] 1. jobs/iso2rootfs.yaml. Before: test_yaml, test_os_mount, test_testbox After: test1_yaml,test1_os_mount, test1_testbox test2_yaml,test2_os_mount, test2_testbox ... 2. tests/iso2rootfs. - Use for-loop to traverse if $testX_yaml exists. - [ -n "${test1_yaml}" ] || return 0. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 8 ++++--- tests/iso2rootfs | 54 ++++++++++++++++++++++++++++++-------------- 2 files changed, 42 insertions(+), 20 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index fc3fc1d2e3ba..3c69a9d98430 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -17,6 +17,8 @@ iso2rootfs: config_rootfs: # submit test yaml related fields - test_yaml: - test_os_mount: - test_testbox: + ## you can add as many test jobs as you like. + ## but follow three fields is requied for every test job. + test1_yaml: + test1_os_mount: + test1_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index e9ed955441b3..a38685118b7f 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -56,7 +56,7 @@ check_yaml_vars() for yaml_t in "${yaml_vars[@]}" do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." + [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot find value of var: ${yaml_t}." done } @@ -447,17 +447,35 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } -submit_test_job_yaml() +submit_one_yaml() { - [ -n ${test_yaml} ] || return 0 - log_info "starting submit ${test_yaml} ..." - [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + cd "${LKP_SRC}/jobs" + [ -f "${test_yaml}" ] || + die "cannot find test yaml in LKP_SRC/jobs. + test yaml: ${test_yaml}." + + local test_os_mount="$(eval echo "\$${test_num}_os_mount")" + local test_testbox="$(eval echo "\$${test_num}_testbox")" + + [ -n "${test_os_mount}" ] || die "cannot find value of var: ${test_num}_os_mount." + [ -n "${test_testbox}" ] || die "cannot find value of var: ${test_num}_testbox." - [ -n "${test_os_mount}" ] || die "cannot fine value of var: test_os_mount." - [ -n "${test_testbox}" ] || die "cannot fine value of var: test_testbox." + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" + log_info "submit ${test_yaml} finished" +} + +submit_yamls() +{ + [ -n "$test1_yaml" ] || return 0 + + log_info "starting submit test yamls ..." + + [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh ./sbin/install-dependencies.sh @@ -465,17 +483,19 @@ submit_test_job_yaml() generate_global_yaml } - cd "${LKP_SRC}/jobs" && { - [ -f "${test_yaml}" ] || - die "cannot find test yaml in LKP_SRC/jobs. - test yaml: ${test_yaml}." + local test_yaml_index=1 + local test_num="test${test_yaml_index}" + local test_yaml="$(eval echo "\$${test_num}_yaml")" - "${LKP_SRC}/sbin/submit" \ - $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ - "${test_yaml}" - } + while [ -n "${test_yaml}" ] + do + submit_one_yaml + test_yaml_index=$((${test_yaml_index} +1)) + test_num="test${test_yaml_index}" + test_yaml="$(eval echo "\$${test_num}_yaml")" + done - log_info "submit ${test_yaml} finished" + log_info "submit test yamls finished" } ############ post works ############ @@ -511,7 +531,7 @@ main() config_rootfs test_rootfs - submit_test_job_yaml + submit_yamls post_works } -- 2.23.0

3 4

[PATCH lkp-tests 4/6] iso2rootfs: delete the useless fields
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] The 'os', 'os_arch', 'os_version' fields to be use by test job are the same as the 'iso_os', 'iso_arch', 'iso_version' of the destination iso this time. So delete the 'test_os', 'test_os_arch', 'test_os_version'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 3 --- tests/iso2rootfs | 33 +++++++-------------------------- 2 files changed, 7 insertions(+), 29 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index c81d548b49ea..fc3fc1d2e3ba 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -18,8 +18,5 @@ iso2rootfs: # submit test yaml related fields test_yaml: - test_os: - test_os_arch: - test_os_version: test_os_mount: test_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 34c8e84d9d74..e9ed955441b3 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -443,17 +443,8 @@ generate_global_yaml() generate_submit_append_str() { - local wait_add_vars=( - "os" - "os_arch" - "os_version" - "os_mount" - "testbox" - ) - for vt in "${wait_add_vars[@]}" - do - SUBMIT_APPEND_STR="${SUBMIT_APPEND_STR} ${vt}=$(eval echo "\$test_${vt}")" - done + [ "$#" -eq 2 ] || die "cannot get enough submit params." + echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } submit_test_job_yaml() @@ -464,18 +455,8 @@ submit_test_job_yaml() [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." - local test_yaml_vars=( - "test_os" - "test_os_arch" - "test_os_version" - "test_os_mount" - "test_testbox" - ) - - for yaml_t in "${test_yaml_vars[@]}" - do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." - done + [ -n "${test_os_mount}" ] || die "cannot fine value of var: test_os_mount." + [ -n "${test_testbox}" ] || die "cannot fine value of var: test_testbox." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh @@ -489,9 +470,9 @@ submit_test_job_yaml() die "cannot find test yaml in LKP_SRC/jobs. test yaml: ${test_yaml}." - generate_submit_append_str - - "${LKP_SRC}/sbin/submit" ${SUBMIT_APPEND_STR} "${test_yaml}" + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" } log_info "submit ${test_yaml} finished" -- 2.23.0

2 2

[PATCH lkp-tests 3/6] iso2rootfs: add a switch to configure rootfs
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] In some cases, we do not need to configure the result rootfs, so add a switch to configure rootfs. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 2 ++ tests/iso2rootfs | 1 + 2 files changed, 3 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 4e8729997a00..c81d548b49ea 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -13,6 +13,8 @@ iso2rootfs: # dailybuild related fields iso_url_file: + ## 0 - config; 1- don't config + config_rootfs: # submit test yaml related fields test_yaml: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index d42c38909f68..34c8e84d9d74 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -355,6 +355,7 @@ disable_selinux() config_rootfs() { + [ "${config_rootfs}" == "0" ] || return 0 config_dns_resolver disable_selinux } -- 2.23.0

3 4

[PATCH v7 compass-ci 1/2] lib: keep suite result in top for compare output
by Lu Kaiyi 19 Nov '20

19 Nov '20

[why] Currently, the compare output result default order by alphabet. If there are too many lines in result, it will be difficult for user to find the lines that starts with $suite in metric column. So, need reorder compare output result with keeping $suite item on the top of the table. [how] Divide the lines into two parts before print according to whether the line start with $suite or not, then combine output result into a whole to highlight the important info. [example] lukaiyi@z9 ~% compare "id=z9.148096" "id=z9.148143" in z9 suites_list:["sysbench-mysql"] [before] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 ... 1266042.00 -18.3% 1034206.00 softirqs.RCU 547849.00 -27.4% 397593.00 softirqs.SCHED 159990.00 -80.5% 31181.00 softirqs.TASKLET 1617967.00 -23.0% 1245331.00 softirqs.TIMER 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 2149.23 -100.0% 0.00 time.elapsed_time 2149.23 -100.0% 0.00 time.elapsed_time.max 214458.00 -100.0% 0.00 time.involuntary_context_switches 2.00 -100.0% 0.00 time.major_page_faults 192832.00 -100.0% 0.00 time.maximum_resident_set_size ... [after] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 2894304.00 +12.8% 3265730.00 interrupts.4:GICv3.26.Level.arch_timer 11546998.00 -100.0% 0.00 interrupts.746:ITS-MSI.99092481.Edge.hns3-0000:bd:00.1-TxRx-0 7476064.00 -100.0% 0.00 interrupts.747:ITS-MSI.99092482.Edge.hns3-0000:bd:00.1-TxRx-2 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 16 ++++++++++------ lib/compare_matrixes.rb | 41 ++++++++++++++++++++++++++++------------- lib/matrix2.rb | 20 +++++++++++++++----- 3 files changed, 53 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..f34e239 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suite_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suite_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,16 @@ end def create_matrices_list(conditions) matrices_list = [] + suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suite_list.concat(suites) end - matrices_list + + return matrices_list, suite_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..5039b01 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -128,7 +128,7 @@ def set_compare_values(index, values, field, success, options) end def get_values_by_field(matrixes_list, field, matrixes_size, success, options) - # get values by field, values struce example: values[0][:average] + # get values by field, values struct example: values[0][:average] # values = {} matrixes_list.length.times do |index| @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suite_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -807,26 +808,40 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suite_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suite_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suite_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suite_list, matrixes_titles, success, theme) return '' if values.empty? + suite_set = Set.new(suite_list) result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + ranked_str = get_ranked_str(values, suite_set, success, theme) + result_str += ranked_str + result_str +end + +def get_ranked_str(values, suite_set, success, theme) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suite_set.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end - result_str + suite_str + common_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..62d174c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,23 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end - groups.delete(group_key) if value.size < 2 + if value.size < 2 + groups.delete(group_key) + next + end + suites_list << suite_list end - groups + + return groups, suites_list end # input: @@ -139,7 +149,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], = create_matrix(jobs) end end -- 2.23.0

3 2

[PATCH v2 lkp-tests 1/2] stats/iozone: fix iozone cannot match the line of part test
by Lu Kaiyi 19 Nov '20

19 Nov '20

[why] iozone can match the line like: 64 4 1599680 3057153 4274062 4897948 3588436 2892445 2772930 3057153 3958892 3057153 3022727 3363612 4564786 but cannot match the line as below(there are extra spaces after '2892445'): 64 4 1599680 3057153 4274062 4897948 3588436 2892445 [how] Add line.rstrip! before match Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/iozone | 1 + 1 file changed, 1 insertion(+) diff --git a/stats/iozone b/stats/iozone index 9f7ee72f..ff389c48 100755 --- a/stats/iozone +++ b/stats/iozone @@ -8,6 +8,7 @@ per_record = {} all_sum = 0 nr_records = 0 while (line = STDIN.gets) + line.rstrip! next unless line =~ /^\s*\d+.*\d+$/ data = line.split -- 2.23.0

3 3

[PATCH v2 compass-ci] lib/compare_data_format: optimize the format for echart
by Lu Weitao 19 Nov '20

19 Nov '20

[Why] With template compare mode, the result(data_set) isn't friendly for echart like: { "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "bs|test_size", "tables": { "1G|64k": { "fio.read_iops": { "average": { "openeuler 20.03": 14010.34254665909 }, "standard_deviation": { "openeuler 20.03": null }, "change": { } }, "fio.write_iops": { "average": { "openeuler 20.03": 13870.135498022726 }, "standard_deviation": { "openeuler 20.03": null }, "change": { } } }, "1G|4k": {...} } } [How] Make the result more friendly for echart like: { "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "test_size|bs", "tables": { "fio.read_iops": { "average": { "dimensions": [ "compare_dimension", "openeuler 20.03" ], "source": [ ["1G|4k", "1G|16k", "1G|32k", "1G|64k", "1G|128k", "1G|256k", "1G|512k", "1G|1024k", "80G|4k"], ["openeuler 20.03", 144076.2903315, 37865.30472368628, 21145.10375497826, 14010.34254665909, 6701.240849466667, 3205.077255, 1367.476930860465, 673.3270888666667, 11601.099817 ] ] }, "standard_deviation": {...}, "change": {...} }, "fio.write_iops": {...} ... } } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_data_format.rb | 77 ++++++++++++++++++++++++++++++++++---- lib/compare_matrixes.rb | 3 +- 2 files changed, 71 insertions(+), 9 deletions(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index 3d82550..b4f26ee 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -5,14 +5,75 @@ # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # +class FormatEchartData + def initialize(metrics_compare_results, template_params) + @metrics_compare_results = metrics_compare_results + @template_params = template_params + @data_set = {} + end -def format_for_echart(metrics_compare_results, template_params) - echart_result = {} - echart_result['title'] = template_params['title'] - echart_result['unit'] = template_params['unit'] - x_params = template_params['x_params'] - echart_result['x_name'] = x_params.join('|') if x_params - echart_result['tables'] = metrics_compare_results + def format_for_echart + echart_result = {} + echart_result['title'] = @template_params['title'] + echart_result['unit'] = @template_params['unit'] + @x_name = @template_params['x_params'] + echart_result['x_name'] = @x_name.join('|') if @x_name + echart_result['tables'] = convert_to_echart_dataset - echart_result + echart_result + end + + def convert_to_echart_dataset + @x_params = sort_x_params((a)metrics_compare_results.keys) + @metrics_compare_results.each_value do |metrics_values| + metrics_values.each do |metric, metric_value| + assign_echart_data_set(metric, metric_value) + end + end + + @data_set + end + + def assign_echart_data_set(metric, metric_value) + @data_set[metric] = {} + metric_value.each do |value_type, values| # value_type can be: average, standard_deviation, change + @data_set[metric][value_type] = { + 'dimensions' => ['compare_dimension'] + } + + dimension_list = values.keys.sort + @data_set[metric][value_type]['dimensions'] += dimension_list + @data_set[metric][value_type]['source'] = assign_echart_source(metric, value_type, dimension_list) + end + end + + def assign_echart_source(metric, value_type, dimensions) + source = [] + source << @x_params + dimensions.each do |dimension| + dimension_values = [dimension] + @x_params.each do |x_param| + if @metrics_compare_results[x_param][metric] + dimension_values << @metrics_compare_results[x_param][metric][value_type][dimension] + end + end + source << dimension_values + end + + source + end +end + +# input: x_params_list +# eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] +# output: +# ["1G|4K", "1G|128k", "1G|1024k", "2G|4k"] +def sort_x_params(x_params_list) + x_params_hash = {} + x_params_list.each do |param| + params = param.gsub(/[a-zA-Z]+$/, '0').split('|').map(&:to_i) + x_params_hash[params] = param + end + + x_params_hash.sort.map { |h| h[1] } end diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..762c087 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -498,7 +498,8 @@ def get_dimentions_combination(dimension_list) end def show_compare_result(metrics_compare_results, template_params) - echart_results = format_for_echart(metrics_compare_results, template_params) + formatter = FormatEchartData.new(metrics_compare_results, template_params) + echart_results = formatter.format_for_echart print JSON.pretty_generate(echart_results) end -- 2.23.0

1 0

[PATCH lkp-tests] create a multi-level directory
by Hu Xuejiao 19 Nov '20

19 Nov '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..6f603abe 100755 --- a/sbin/submit +++ b/sbin/submit @@ -9,6 +9,7 @@ require "#{LKP_SRC}/lib/distro_info" require "#{LKP_SRC}/lib/scheduler_client" require 'optparse' require 'yaml' +require 'fileutils' opt_set_key_value = {} opt_cmdline_defaults = {} @@ -35,7 +36,7 @@ options = OptionParser.new do |opts| exit 1 end opt_output_dir = dir - Dir.mkdir(dir) unless File.directory? dir + FileUtils.mkdir_p(dir) unless File.directory? dir end opts.on('-a', '--auto-define-files', 'auto add define_files') do -- 2.23.0

5 4

[PATCH lkp-tests 1/6] iso2rootfs: change how to get the kernel and initrd.lkp of rootfs
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] We use the container in compass-ci repo to generate rootfs from a qcow2, it will generate the follow two soft links: - {rootfs_dir}/boot/vmlinuz # for vmlinuz file - {rootfs_dir}/initrd.lkp # for initrd file So we can use the below two soft links to get the absolute path of vmlinuz and initrd. [Addition] URL of qcow2rootfs: - https://gitee.com/wu_fengguang/compass-ci/tree/master/container/qcow2rootfs Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 05274164f8c1..9c71b8427d90 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -375,8 +375,12 @@ test_rootfs() log_info "starting test rootfs ..." local test_script="${CGZ_PATH}/kvm.sh" - local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz*|grep -v rescue) - local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/boot/initramfs.lkp*) + local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz) + local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/initrd.lkp) + + [ -f "$kernel_path" ] || die "cannot find kernel of rootfs: $kernel_path" + [ -f "$initrd_lkp_path" ] || die "cannot find initrd_lkp of rootfs: $initrd_lkp_path" + local root_path case ${rootfs_protocol} in "nfs") -- 2.23.0

1 0

[PATCH compass-ci] git_mirror: fix that too few repos in upstream-repos cause memory leak
by Li Yuanchao 18 Nov '20

18 Nov '20

[why] when there is only a few repo files in upstream-repos, that means the same repo will be called very frequently. As the main loop is too fast, it seems the new flow would be called before the last flow of the same repo end completely. That cause a memory problem. put a sleep in main loop would give the flow enough time to end completely before the same flow be called again Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 72ca328..5b6b889 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,6 +203,7 @@ class MirrorMain push_git_queue handle_feedback Signal.trap(:SIGCHLD, 'SIG_IGN') + sleep(0.1) end end end -- 2.23.0

3 3

[PATCH v2 lkp-tests] tests/build-pkg: add download PKGBUILD-$tag function
by Liu Shaofei 18 Nov '20

18 Nov '20

[why] Support more version's PKGBUILD to build different version software. [how] Download the PKGBUILD-$tag file through remote-git server from "/srv/git/pkg/" usage: 1) append upstream_tag field from build-pkg.yaml 2) create git repo that named by software name of $upstream_repo in "/srv/git/pkg/", and add PKGBUILD-$tag file to the directory of "/srv/git/pkg/$upstream_repo". if not PKGBUILD-$tag, the software is built with PKGBUILD. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 4aaa4fef..c5d55eda 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -7,7 +7,7 @@ # - upstream_repo # - upstream_url # - upstream_commit -# - upstream_tag +# - PKGBUILD_TAG ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -61,6 +61,15 @@ request_pkg() get_pkgfile "${pkgrepo}" "$pkgfile" done + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" + } + [ -f PKGBUILD ] || die "PKGBUILD not exist" [ -s PKGBUILD ] || die "PKGBUILD is empty" } @@ -72,9 +81,9 @@ create_softlink() local soft_path="$PKG_MNT/${pack_to}/${pkgname}" local bm_name=$(basename $(realpath ${cgz_name})) - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + if [ -n "$PKGBUILD_TAG" ]; then + echo "create soft link: ${PKGBUILD_TAG}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${PKGBUILD_TAG}.cgz" fi echo "create soft link: latest.cgz -> ${bm_name}" @@ -96,6 +105,9 @@ build_source_pkg() upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\")" sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD + [ -n "$PKGBUILD_TAG" ] && { + sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" + } } cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: umask 002 before download git trees
by Liu Yinsi 18 Nov '20

18 Nov '20

because ENV CCI_SRC LKP_SRC, REPO_SRC all under /c, some containers service start need umask 002, set it before download git trees. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/git | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 5588696..494107e 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -2,6 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# fix files permissions cause container service start failed +umask 002 + # download git trees cd /c || exit -- 2.23.0

2 2