- Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests] lib: fix request state
by Ren Wen 20 Nov '20

20 Nov '20

When run *start_daemon*, lkp reports a "finished" state to scheduler if daemon started. When run *run_test*, lkp also reports a "finished" state if tests finished. Now i want to stop machines after all tests finished, so alter state "finished" to "started" when running *start_daemon*. This makes it easier to handle request state at scheduler. Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.sh b/lib/job.sh index 81523a98..b84e6b18 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -278,7 +278,7 @@ start_daemon() run_program daemon "$@" fi - sync_cluster_state 'finished' + sync_cluster_state 'started' # If failed to start the daemon above, the job will abort. # LKP server on notice of the failed job will abort the other waiting nodes. -- 2.23.0

1 0

[PATCH lkp-tests] lib: fix request state
by Ren Wen 20 Nov '20

20 Nov '20

When run *start_daemon*, lkp reports a "finished" state to scheduler if daemon started. When run *run_test*, lkp also reports a "finished" state if tests finished. Now i want to stop machines after all tests finishing, so alter state "finished" to "started" when run *start_daemon*. This makes it easier to handle request state at scheduler. Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.sh b/lib/job.sh index 81523a98..b84e6b18 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -278,7 +278,7 @@ start_daemon() run_program daemon "$@" fi - sync_cluster_state 'finished' + sync_cluster_state 'started' # If failed to start the daemon above, the job will abort. # LKP server on notice of the failed job will abort the other waiting nodes. -- 2.23.0

2 2

[PATCH compass-ci] container/logging-es: fix failed to build images in x86 machine
by Liu Yinsi 20 Nov '20

20 Nov '20

[why] when build logging-es images in x86 machine, error message: libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error" The command '/bin/sh -c sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture In addition, 'declare', arrays, arrays references are undefined in POSIX sh by shellcheck, so change '#!/bin/sh' to '#!/bin/bash'. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 3 ++- container/logging-es/build | 13 +++++++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 707299f..6c01c43 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -1,7 +1,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM gagara/elasticsearch-oss-arm64:7.6.2 +ARG BASE_IMAGE +FROM $BASE_IMAGE # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 diff --git a/container/logging-es/build b/container/logging-es/build index 0ddecbf..451041b 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,5 +1,14 @@ -#!/bin/sh +#!/bin/bash # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t logging-es:7.6.2 . +declare -A BASE_IMAGE_DICT + +BASE_IMAGE_DICT=( + ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" + ["x86_64"]="elasticsearch:7.6.2" +) + +BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} + +docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . -- 2.23.0

1 0

[PATCH compass-ci] container/minio: fix build permission denied
by Liu Yinsi 20 Nov '20

20 Nov '20

lys@crystal ~/compass-ci/container/minio% ./build zsh: permission denied: ./build Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/minio/build | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 container/minio/build diff --git a/container/minio/build b/container/minio/build old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH lkp-tests] stats: add new parsing parameters for performance test
by Bai Jing 20 Nov '20

20 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/lmbench3 | 62 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 2 deletions(-) diff --git a/stats/lmbench3 b/stats/lmbench3 index 211c3abc..750e8bfc 100755 --- a/stats/lmbench3 +++ b/stats/lmbench3 @@ -21,6 +21,27 @@ def largest_bandwidth bandwidth end +def designated_file_size + designated_hash = {} + result_str = '' + + $stdin.each_line do |line| + result_str += line + break if line =~ /(^\s|^\t)/ + end + + result_str.split("\n").each do |e| + designated_hash['L1_$'] = e.split[1] if e =~ /0.01758/ + designated_hash['Rand_mem'] = e.split[1] if e =~ /512.0000/ + designated_hash['Mmap_Latency'] = e.split[1] if e =~ /536.870912/ + designated_hash['L2_$'] = e.split[1] if e =~ /0.05078/ + end + + designated_hash +end + +null_io_array = [] + while (line = STDIN.gets) line = line.resolve_invalid_bytes case line @@ -32,14 +53,15 @@ while (line = STDIN.gets) # Simple fstat: 0.3517 microseconds # Simple open/close: 1.3696 microseconds when /^Simple (\S+): (\d+.\d+) microseconds$/ + null_io_array << $2.to_f if $1.to_s == 'read' || $1.to_s == 'write' puts "syscall.#{$1}.latency.us: #{$2}" # Extract select test result. # Select on 100 fd's: 1.2293 microseconds # Select on 100 tcp fd's: 5.0377 microseconds - when /^Select on 100 fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 fd's: (\d+.\d+) microseconds$/ puts "Select.100fd.latency.us: #{$1}" - when /^Select on 100 tcp fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 tcp fd's: (\d+.\d+) microseconds$/ puts "Select.100tcp.latency.us: #{$1}" # Extract proc test result. @@ -198,5 +220,41 @@ while (line = STDIN.gets) # ... when /^"Mmap read open2close bandwidth$/ puts "MMAP.read_open2close.bandwidth.MB/sec: #{largest_bandwidth}" + + # "Random Memory load latency" + # 416.00000 92.824 + # 448.00000 92.772 + # 512.00000 92.640 + # ... + when /^Random load latency$/ + result_rand_mem = designated_file_size + puts "Rand_mem: #{result_rand_mem['Rand_mem']}" + + # "Memory load latency" + # 0.00049 1.539 + # 0.00098 1.539 + # 512.00000 74.421 + # ... + when /^Memory load latency$/ + result_rand_mem = designated_file_size + puts "L1_$: #{result_rand_mem['L1_$']}" + puts "L2_$: #{result_rand_mem['L2_$']}" + + # "mappings" + # 67.108864 82 + # 134.217728 151 + # 268.435456 309 + # 536.870912 615 + # ... + when /^"mappings$/ + result_map = designated_file_size + puts "Mmap_Latency: #{result_map['Mmap_Latency']}" end end + +num = 0 +null_io_array.each do |v| + num += v +end +null_num = format('%.4f', num / 2).to_f +puts "null_io: #{null_num}" -- 2.23.0

3 4

[PATCH v3 lkp-tests 6/6] iso2rootfs: allow the use of job.yaml and test script from the internet
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] Not all the yaml and script of test job will be committed to the lkp-tests repo. [How] Support three optional fields for every test in job.yaml: - testX_git_url: the url of git repo which contains yaml and script; - testX_git_yaml: the relative path of the yaml file in the git repo; - testX_git_script: the relative path of the test script in the git repo; Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 5 +++++ tests/iso2rootfs | 26 ++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 1631aca6d23f..1a643b538ac1 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -26,3 +26,8 @@ iso2rootfs: test1_yaml: test1_os_mount: test1_testbox: + ## the following three fields is required when your job.yaml and + ## script for this test are from the internet. + test1_git_url: + test1_git_yaml: + test1_git_script: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 4135cdea7059..0353c045a4bc 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -457,10 +457,36 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } +prepare_submit() +{ + local submit_dir="/tmp/iso2rootfs/submit_tmp" + mkdir -p "${submit_dir}" + + local submit_git_dir="${submit_dir}/${test_git_url##*/}" + submit_git_dir="${submit_git_dir%.git}" + + [ -d "${submit_git_dir}" ] || { + cd "${submit_dir}" + git clone "${test_git_url}" + } + + local test_git_yaml="${submit_git_dir}/$(eval echo "\$${test_num}_git_yaml")" + local test_git_script="${submit_git_dir}/$(eval echo "\$${test_num}_git_script")" + + [ -f "${test_git_yaml}" ] || die "cannot find file: ${test_num}_git_yaml." + cp "${test_git_yaml}" "${LKP_SRC}/jobs/" + + [ -f "${test_git_script}" ] || die "cannot find file: ${test_num}_git_script." + cp "${test_git_script}" "${LKP_SRC}/tests/" +} + submit_one_yaml() { log_info "starting submit ${test_yaml} ..." + local test_git_url="$(eval echo "\$${test_num}_git_url")" + [ -z "${test_git_url}" ] || prepare_submit + cd "${LKP_SRC}/jobs" [ -f "${test_yaml}" ] || die "cannot find yaml in LKP_SRC/jobs: ${test_yaml}." -- 2.23.0

1 0

[PATCH v3 lkp-tests 5/6] iso2rootfs: allow submit multiple test job.yaml for result rootfs
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] Result rootfs of dailybuild requires multiple test jobs. [How] 1. jobs/iso2rootfs.yaml. Before: test_yaml, test_os_mount, test_testbox After: test1_yaml, test1_os_mount, test1_testbox test2_yaml, test2_os_mount, test2_testbox ... 2. tests/iso2rootfs. - Use while-loop to traverse if $testX_yaml exists. - [ -n "${test1_yaml}" ] || return 0. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 8 ++++--- tests/iso2rootfs | 50 ++++++++++++++++++++++++++++++-------------- 2 files changed, 39 insertions(+), 19 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 629b53febb2f..1631aca6d23f 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -21,6 +21,8 @@ iso2rootfs: dailybuild_iso_url_file: # submit test yaml related fields - test_yaml: - test_os_mount: - test_testbox: + ## 1. you can add as many jobs as you like. + ## 2. the following three fields is requied for every test job. + test1_yaml: + test1_os_mount: + test1_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 3015c6f62947..4135cdea7059 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -457,17 +457,33 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } -submit_test_job_yaml() +submit_one_yaml() { - [ -n ${test_yaml} ] || return 0 - log_info "starting submit ${test_yaml} ..." - [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + cd "${LKP_SRC}/jobs" + [ -f "${test_yaml}" ] || die "cannot find yaml in LKP_SRC/jobs: ${test_yaml}." + + local test_os_mount="$(eval echo "\$${test_num}_os_mount")" + local test_testbox="$(eval echo "\$${test_num}_testbox")" + + [ -n "${test_os_mount}" ] || die "cannot find value of var: ${test_num}_os_mount." + [ -n "${test_testbox}" ] || die "cannot find value of var: ${test_num}_testbox." - [ -n "${test_os_mount}" ] || die "cannot find value of var: test_os_mount." - [ -n "${test_testbox}" ] || die "cannot find value of var: test_testbox." + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" + log_info "submit ${test_yaml} finished" +} + +submit_yamls() +{ + [ -n "$test1_yaml" ] || return 0 + + log_info "starting submit test yamls ..." + + [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh ./sbin/install-dependencies.sh @@ -475,17 +491,19 @@ submit_test_job_yaml() generate_global_yaml } - cd "${LKP_SRC}/jobs" && { - [ -f "${test_yaml}" ] || - die "cannot find test yaml in LKP_SRC/jobs. - test yaml: ${test_yaml}." + local test_yaml_index=1 + local test_num="test${test_yaml_index}" + local test_yaml="$(eval echo "\$${test_num}_yaml")" - "${LKP_SRC}/sbin/submit" \ - $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ - "${test_yaml}" - } + while [ -n "${test_yaml}" ] + do + submit_one_yaml + test_yaml_index=$((${test_yaml_index} +1)) + test_num="test${test_yaml_index}" + test_yaml="$(eval echo "\$${test_num}_yaml")" + done - log_info "submit ${test_yaml} finished" + log_info "submit test yamls finished" } ############ post works ############ @@ -525,7 +543,7 @@ main() config_rootfs test_rootfs - submit_test_job_yaml + submit_yamls post_works } -- 2.23.0

1 0

[PATCH v3 lkp-tests 4/6] iso2rootfs: delete the useless fields
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] The 'os', 'os_arch', 'os_version' fields to be use by test job are the same as the 'iso_os', 'iso_arch', 'iso_version' of the destination iso this time. So delete the 'test_os', 'test_os_arch', 'test_os_version'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 3 --- tests/iso2rootfs | 35 ++++++++--------------------------- 2 files changed, 8 insertions(+), 30 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 1e1dfc862136..629b53febb2f 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -22,8 +22,5 @@ iso2rootfs: # submit test yaml related fields test_yaml: - test_os: - test_os_arch: - test_os_version: test_os_mount: test_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index bb2da3208646..3015c6f62947 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -56,7 +56,7 @@ check_yaml_vars() for yaml_t in "${yaml_vars[@]}" do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." + [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot find value of var: ${yaml_t}." done } @@ -453,17 +453,8 @@ generate_global_yaml() generate_submit_append_str() { - local wait_add_vars=( - "os" - "os_arch" - "os_version" - "os_mount" - "testbox" - ) - for vt in "${wait_add_vars[@]}" - do - SUBMIT_APPEND_STR="${SUBMIT_APPEND_STR} ${vt}=$(eval echo "\$test_${vt}")" - done + [ "$#" -eq 2 ] || die "generate_submit_append_str: please give 2 parameters." + echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } submit_test_job_yaml() @@ -474,18 +465,8 @@ submit_test_job_yaml() [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." - local test_yaml_vars=( - "test_os" - "test_os_arch" - "test_os_version" - "test_os_mount" - "test_testbox" - ) - - for yaml_t in "${test_yaml_vars[@]}" - do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." - done + [ -n "${test_os_mount}" ] || die "cannot find value of var: test_os_mount." + [ -n "${test_testbox}" ] || die "cannot find value of var: test_testbox." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh @@ -499,9 +480,9 @@ submit_test_job_yaml() die "cannot find test yaml in LKP_SRC/jobs. test yaml: ${test_yaml}." - generate_submit_append_str - - "${LKP_SRC}/sbin/submit" ${SUBMIT_APPEND_STR} "${test_yaml}" + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" } log_info "submit ${test_yaml} finished" -- 2.23.0

1 0

[PATCH v3 lkp-tests 3/6] iso2rootfs: add a switch to configure rootfs
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] In some cases, we do not need to configure the result rootfs, so add a switch to configure rootfs. [How] Add 'config_rootfs' field in job.yaml. And support the following two fields now: - dns: will config /etc/resolv.conf. - no_selinux: will disable selinux in /etc/selinux/config. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 6 ++++++ tests/iso2rootfs | 2 ++ 2 files changed, 8 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 8796874ce79d..1e1dfc862136 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -11,6 +11,12 @@ iso2rootfs: initrd_http_host: initrd_http_port: + # config rootfs related fields + ## supported fields: + ## - dns: will config /etc/resolv.conf. + ## - no_selinux: will disable selinux in /etc/selinux/config. + config_rootfs: + # dailybuild related fields dailybuild_iso_url_file: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 893c010e6ad5..bb2da3208646 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -351,6 +351,7 @@ run_qcow2rootfs() ############ config rootfs ############ config_dns_resolver() { + echo "${config_rootfs}" | grep -qw 'dns' || return 0 cat <<-EOF > "${ROOTFS_DES_DIR}/etc/resolv.conf" nameserver 114.114.114.114 nameserver 8.8.8.8 @@ -359,6 +360,7 @@ config_dns_resolver() disable_selinux() { + echo "${config_rootfs}" | grep -qw 'no_selinux' || return 0 sed -i 's/^SELINUX=.*/SELINUX=disabled/g' "${ROOTFS_DES_DIR}/etc/selinux/config" } -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/6] iso2rootfs: allow non-dailybuild jobs to be submitted
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] 1. Decouple the code related to dailybuild. The 'iso_url_file' field is only relate to the dailybuild function, and this field is unnecessary when user only want generate a rootfs from offical distribution. But now if this field is none, this job will failed. 2. Modify the field name. 'iso_url_file' -> 'dailybuild_iso_url_file'. [Addition] if the `iso_url_file` field exists, this job will generate a symbolic link which point to the result rootfs. - symbolic link name: {iso_version}-dailybuild. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 4 +++- tests/iso2rootfs | 49 +++++++++++++++++++++++++++++--------------- 2 files changed, 35 insertions(+), 18 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index d4f0ffc0e80c..8796874ce79d 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -2,7 +2,6 @@ suite: iso2rootfs category: benchmark iso2rootfs: iso2rootfs_cgz: - iso_url_file: iso_os: iso_arch: iso_version: @@ -12,6 +11,9 @@ iso2rootfs: initrd_http_host: initrd_http_port: + # dailybuild related fields + dailybuild_iso_url_file: + # submit test yaml related fields test_yaml: test_os: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 9c71b8427d90..893c010e6ad5 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -44,7 +44,6 @@ check_yaml_vars() "os_arch" "os_version" "iso2rootfs_cgz" - "iso_url_file" "iso_os" "iso_arch" "iso_version" @@ -105,10 +104,10 @@ check_iso_name() get_daily_iso_checksum() { - ISO_URL="$(curl "${iso_url_file}")" + ISO_URL="$(curl "${dailybuild_iso_url_file}")" [ "${iso_os}" == "openeuler" ] && { - local pub_ip=$(echo "${iso_url_file}" | grep -oEw "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}") - ISO_URL="$(curl "${iso_url_file}" |sed -r "s/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/${pub_ip}/g")" + local pub_ip=$(echo "${dailybuild_iso_url_file}" | grep -oEw "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}") + ISO_URL="$(curl "${dailybuild_iso_url_file}" | sed -r "s/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/${pub_ip}/g")" } ISO_NAME=$(basename "$ISO_URL") @@ -149,15 +148,15 @@ mount_rootfs() get_cache_iso_checksum() { - mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-latest.sha256sum" - [ ! -f "${CHECKSUM_FILE_CACHE}" ] || - SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") + [ -f "${CHECKSUM_FILE_CACHE}" ] || return 0 + SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } check_sha256sum_update() { + [ -n "${dailybuild_iso_url_file}" ] || return 0 + get_cache_iso_checksum get_daily_iso_checksum @@ -183,6 +182,7 @@ check_sha256sum_update() check_os() { log_info "starting check os ..." + echo "${ALLOW_OS[@]}" | grep -wq "${os}" || die "current os is not in allow os. current os: ${os}. @@ -243,7 +243,10 @@ config_iso2rootfs() pre_works() { check_yaml_vars + + mount_rootfs check_sha256sum_update + check_os config_iso2rootfs @@ -274,14 +277,17 @@ config_iso_conf() config_iso_url() { - local iso_url_file="${I2Q_SRC}/conf/iso/net-iso/${iso_os}/${iso_arch}/${iso_version}" - sed -i "s|^iso_url=.*|iso_url=\"${ISO_URL}\"|g" "${iso_url_file}" - sed -i "s|^iso_checksum_url=.*|iso_checksum_url=\"${ISO_CHECKSUM_URL}\"|g" "${iso_url_file}" + [ -n "${dailybuild_iso_url_file}" ] || return 0 + + local i2q_iso_url_file="${I2Q_SRC}/conf/iso/net-iso/${iso_os}/${iso_arch}/${iso_version}" + sed -i "s|^iso_url=.*|iso_url=\"${ISO_URL}\"|g" "${i2q_iso_url_file}" + sed -i "s|^iso_checksum_url=.*|iso_checksum_url=\"${ISO_CHECKSUM_URL}\"|g" "${i2q_iso_url_file}" } config_iso2qcow2() { log_info "starting config iso2qcow2 env ..." + download_iso2qcow2 config_pip_proxy @@ -292,6 +298,7 @@ config_iso2qcow2() run_iso2qcow2() { log_info "starting run iso2qcow2 ..." + ${I2Q_SRC}/auto-install-iso.sh } @@ -313,6 +320,7 @@ config_rootfs_dir() config_qcow2rootfs() { log_info "starting config qcow2rootfs env ..." + download_compass_ci install_pkgs "docker" @@ -332,7 +340,8 @@ config_qcow2rootfs() run_qcow2rootfs() { log_info "starting run qcow2rootfs ..." - local qcow2_path="${I2Q_SRC}/output/${ISO_NAME%.iso}.qcow2" + + local qcow2_path="$(ls ${I2Q_SRC}/output/*.qcow2)" cd "${CCI_SRC}/container/qcow2rootfs" ./run "${qcow2_path}" "${ROOTFS_DES_DIR}" @@ -373,11 +382,11 @@ get_qemu_efi_fd() test_rootfs() { log_info "starting test rootfs ..." + local test_script="${CGZ_PATH}/kvm.sh" local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz) local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/initrd.lkp) - [ -f "$kernel_path" ] || die "cannot find kernel of rootfs: $kernel_path" [ -f "$initrd_lkp_path" ] || die "cannot find initrd_lkp of rootfs: $initrd_lkp_path" @@ -497,16 +506,22 @@ submit_test_job_yaml() } ############ post works ############ -post_works() +update_dailybuild_soft_link() { - log_info "starting post works ..." + [ -n "${dailybuild_iso_url_file}" ] || return 0 mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && - rm -f "${soft_link}" && - ln -s "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" + ln -sfT "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" +} + +post_works() +{ + log_info "starting post works ..." + + update_dailybuild_soft_link cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/6] iso2rootfs: change how to get the kernel and initrd.lkp of rootfs
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] We use the container in compass-ci repo to generate rootfs from a qcow2, it will generate the follow two symbolic links: - {rootfs_dir}/boot/vmlinuz # for vmlinuz file - {rootfs_dir}/initrd.lkp # for initrd file So we can use the below two symbolic links to get the absolute path of vmlinuz and initrd. [Addition] URL of qcow2rootfs: - https://gitee.com/wu_fengguang/compass-ci/tree/master/container/qcow2rootfs Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 05274164f8c1..9c71b8427d90 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -375,8 +375,12 @@ test_rootfs() log_info "starting test rootfs ..." local test_script="${CGZ_PATH}/kvm.sh" - local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz*|grep -v rescue) - local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/boot/initramfs.lkp*) + local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz) + local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/initrd.lkp) + + [ -f "$kernel_path" ] || die "cannot find kernel of rootfs: $kernel_path" + [ -f "$initrd_lkp_path" ] || die "cannot find initrd_lkp of rootfs: $initrd_lkp_path" + local root_path case ${rootfs_protocol} in "nfs") -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit: encapsulate function for submit job yaml
by Wei Jihui 20 Nov '20

20 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit_spec.rb | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 983d050e..f365f35b 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -20,20 +20,24 @@ def traverse_file(output_dir) end end +def submit_job(submit_item) + submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) + Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| + output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) + end +end + describe 'submit job spec' do - it 'link jobs spec' do - Dir.glob("#{LKP_SRC}/spec/submit/link_jobs/*.yaml").each do |yaml_file| - job_name = File.basename(yaml_file, '.yaml') - output_dir = "#{LKP_SRC}/spec/submit/link_jobs/#{job_name}" - submit_cmd = [ - "#{LKP_SRC}/sbin/submit", - '-o', output_dir, - '-s', 'lab: spec_lab', - '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file - ] - system(*submit_cmd) - traverse_file(output_dir) - end + it 'link jobs spec' do + submit_job('link_jobs') end end -- 2.23.0

2 1

[PATCH v8 compass-ci 2/2] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 20 Nov '20

20 Nov '20

[why] Avoid to execute some unnecessary operations and improve performance. [how] Keep the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ff4ea31..2065039 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

3 4

[PATCH compass-ci] stats_worker.cr: add blank line after next cmd
by Xu Xijian 20 Nov '20

20 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/extract-stats/stats_worker.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 2af07d4..bcf6e52 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -103,6 +103,7 @@ class StatsWorker next end next if is_exists + new_error_ids << error_id @rc.store_error_info error_id, job_id end @@ -114,6 +115,7 @@ class StatsWorker ERROR_ID_FILES.each do |filename| filepath = File.join(result_root, filename) next unless File.exists?(filepath) + content = File.open(filepath) do |file| JSON.parse(file) end -- 2.23.0

1 0

[PATCH v4 compass-ci] lib/log.rb: add new class for json log
by Wu Zhende 20 Nov '20

20 Nov '20

[Why] The built-in log system prints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example1] use: log.info("test") output: {"level":"INFO","level_num":1,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"test"} [Example2] use: log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"", "job_id":"1","job_state":"submit","caller":["/usr/share/ruby/logger.rb:545:in `error'","./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/log.rb | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..0ffc0f9 --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' +require 'json' + +# log class to print logs in JSON format +class Log < Logger + LEVEL_INFO = { + 'DEBUG' => 0, + 'INFO' => 1, + 'WARN' => 2, + 'ERROR' => 3, + 'FATAL' => 4, + 'UNKNOWN' => 5 + }.freeze + + FORMATTER = proc { |severity, datetime, progname, msg| + puts + level_num = LEVEL_INFO[severity] + begin + message = JSON.parse(msg) + rescue JSON::ParserError + message = { 'message' => msg } + end + h = { + 'level' => severity.to_s, + 'level_num' => level_num, + 'datetime' => datetime, + 'progname' => progname, + 'message' => '' + } + h.merge!(message) + h.merge!({ 'caller' => caller }) if level_num >= 2 + h.to_json + } + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

3 2

[PATCH v2 lkp-tests] common.sh: add function for count input cpu number
by Wei Jihui 20 Nov '20

20 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/common.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/common.sh b/lib/common.sh index 2a93aafa..95d7953c 100755 --- a/lib/common.sh +++ b/lib/common.sh @@ -85,6 +85,18 @@ cpu_list_ref() echo $cpu_list | cut -d ' ' -f $((n+1)) } + +# input: 1-3 output: 3 +# input: 1,3 output: 2 +cpu_list_num() +{ + cpu_list="" + for cpu in $(expand_cpu_list $1); do + cpu_list+="$cpu " + done + cpu_list_count "$cpu_list" +} + # if str starts with prefix, output remaining part, otherwise output empty string remove_prefix() { -- 2.23.0

1 0

[PATCH lkp-tests] sshd: fix error when execute in docker
by Zhang Yale 20 Nov '20

20 Nov '20

error: bash: 1605766611473034209%""+"": syntax error: operand expected (error token is """+""") Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daemon/sshd b/daemon/sshd index 1a1d59b4..a3fdf954 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -77,7 +77,7 @@ set_port() for i in $(seq 1 10) do - port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) + port=$(($(date +%s%N)%${sshr_port_len}+${sshr_port_base})) ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null [ $? -eq 0 ] && return -- 2.23.0

2 2

[PATCH lkp-tests] stats/netperf: fix rubocop offenses
by Zhang Yale 20 Nov '20

20 Nov '20

[Why] Inspecting 1 file C Offenses: netperf:51:29: C: Style/NumericPredicate: Use iterations.positive? instead of iterations > 0. throughtput /= iteration if iterations > 0 ^^^^^^^^^^^^^^ 1 file inspectef, 1 offense detected Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/netperf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/netperf b/stats/netperf index 7965b1ed..80b07ae5 100755 --- a/stats/netperf +++ b/stats/netperf @@ -48,7 +48,7 @@ avg_throughput = throughput / num # We only cares for the average total throughput/workload # within each iteration. -throughput /= iterations if iterations > 0 +throughput /= iterations if iterations.positive? workload = throughput * time workload = workload * 10**6 / 8.0 / pkt_size if unit == 'Mbps' -- 2.23.0

2 2

[PATCH v2 lkp-tests] create merge_yaml and submit merge_yaml/*.yaml
by Hu Xuejiao 20 Nov '20

20 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories [how] Create merge_yaml and submit merge_yaml/*.yaml Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit_spec.rb | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 983d050e..5c72447a 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -34,6 +34,22 @@ describe 'submit job spec' do ] system(*submit_cmd) traverse_file(output_dir) + end + end + + it 'merge yaml spec' do + Dir.glob("#{LKP_SRC}/spec/submit/merge_yaml/*.yaml").each do |yaml_file| + job_name = File.basename(yaml_file, '.yaml') + output_dir = "#{LKP_SRC}/spec/submit/merge_yaml/#{job_name}/output" + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) + end end - end end -- 2.23.0

2 1

[PATCH v8 compass-ci] container/initrd-cifs: add the container to share /srv/initrd
by Xiao Shenwei 20 Nov '20

20 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result in order to reduce the coupling, need split them [how] one folder mount by one container [usage] two ways for the client to mount initrd: 1. initrd mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage: for lkp compatibility mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 22 ++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 89 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..c2ec808 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +RUN adduser -u 1090 -D lkp +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash +RUN rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..4b93647 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..77558a1 --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

2 2

[PATCH v2 lkp-tests] add the case of yaml_merge_included_files.yaml
by Hu Xuejiao 20 Nov '20

20 Nov '20

[description] The content of template.yaml will be import in yaml_merge_included_files Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 3 +++ spec/submit/merge_yaml/yaml_merge_included_files.yaml | 1 + 2 files changed, 4 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..0415606a --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,3 @@ +suite: test +category: functional +test: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml -- 2.23.0

1 0

[PATCH v8 compass-ci 1/2] lib: keep suite result in top for compare output
by Lu Kaiyi 20 Nov '20

20 Nov '20

[why] Currently, the compare output result default ordered by alphabet. If there are too many lines in result, it will be difficult for user to find the lines that start with $suite in metric column. So, need reorder compare output result with keeping $suite item on the top of the table. [how] Divide the lines into two parts before printing output according to whether the lines start with $suite, then combine output result into a whole to highlight the important info. [example] lukaiyi@z9 ~% compare "id=z9.148096" "id=z9.148143" in z9 suites_list:["sysbench-mysql"] [before] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 ... 1266042.00 -18.3% 1034206.00 softirqs.RCU 547849.00 -27.4% 397593.00 softirqs.SCHED 159990.00 -80.5% 31181.00 softirqs.TASKLET 1617967.00 -23.0% 1245331.00 softirqs.TIMER 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 2149.23 -100.0% 0.00 time.elapsed_time 2149.23 -100.0% 0.00 time.elapsed_time.max 214458.00 -100.0% 0.00 time.involuntary_context_switches 2.00 -100.0% 0.00 time.major_page_faults 192832.00 -100.0% 0.00 time.maximum_resident_set_size ... [after] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 2894304.00 +12.8% 3265730.00 interrupts.4:GICv3.26.Level.arch_timer 11546998.00 -100.0% 0.00 interrupts.746:ITS-MSI.99092481.Edge.hns3-0000:bd:00.1-TxRx-0 7476064.00 -100.0% 0.00 interrupts.747:ITS-MSI.99092482.Edge.hns3-0000:bd:00.1-TxRx-2 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 16 ++++++++++------ lib/compare_matrixes.rb | 41 ++++++++++++++++++++++++++++------------- lib/matrix2.rb | 20 +++++++++++++++----- 3 files changed, 53 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..f34e239 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suite_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suite_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,16 @@ end def create_matrices_list(conditions) matrices_list = [] + suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suite_list.concat(suites) end - matrices_list + + return matrices_list, suite_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..5039b01 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -128,7 +128,7 @@ def set_compare_values(index, values, field, success, options) end def get_values_by_field(matrixes_list, field, matrixes_size, success, options) - # get values by field, values struce example: values[0][:average] + # get values by field, values struct example: values[0][:average] # values = {} matrixes_list.length.times do |index| @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suite_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -807,26 +808,40 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suite_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suite_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suite_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suite_list, matrixes_titles, success, theme) return '' if values.empty? + suite_set = Set.new(suite_list) result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + ranked_str = get_ranked_str(values, suite_set, success, theme) + result_str += ranked_str + result_str +end + +def get_ranked_str(values, suite_set, success, theme) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suite_set.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end - result_str + suite_str + common_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..ff4ea31 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,23 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end - groups.delete(group_key) if value.size < 2 + if value.size < 2 + groups.delete(group_key) + next + end + suites_list << suite_list end - groups + + return groups, suites_list end # input: @@ -139,7 +149,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], _ = create_matrix(jobs) end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 4/6] iso2rootfs: delete the useless fields
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] The 'os', 'os_arch', 'os_version' fields to be use by test job are the same as the 'iso_os', 'iso_arch', 'iso_version' of the destination iso this time. So delete the 'test_os', 'test_os_arch', 'test_os_version'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 3 --- tests/iso2rootfs | 33 +++++++-------------------------- 2 files changed, 7 insertions(+), 29 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 55bf71517174..222ac309d33e 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -22,8 +22,5 @@ iso2rootfs: # submit test yaml related fields test_yaml: - test_os: - test_os_arch: - test_os_version: test_os_mount: test_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index c2e823c93ff9..4f6e3e578ce4 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -447,17 +447,8 @@ generate_global_yaml() generate_submit_append_str() { - local wait_add_vars=( - "os" - "os_arch" - "os_version" - "os_mount" - "testbox" - ) - for vt in "${wait_add_vars[@]}" - do - SUBMIT_APPEND_STR="${SUBMIT_APPEND_STR} ${vt}=$(eval echo "\$test_${vt}")" - done + [ "$#" -eq 2 ] || die "generate_submit_append_str: please give 2 parameters." + echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } submit_test_job_yaml() @@ -468,18 +459,8 @@ submit_test_job_yaml() [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." - local test_yaml_vars=( - "test_os" - "test_os_arch" - "test_os_version" - "test_os_mount" - "test_testbox" - ) - - for yaml_t in "${test_yaml_vars[@]}" - do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." - done + [ -n "${test_os_mount}" ] || die "cannot fine value of var: test_os_mount." + [ -n "${test_testbox}" ] || die "cannot fine value of var: test_testbox." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh @@ -493,9 +474,9 @@ submit_test_job_yaml() die "cannot find test yaml in LKP_SRC/jobs. test yaml: ${test_yaml}." - generate_submit_append_str - - "${LKP_SRC}/sbin/submit" ${SUBMIT_APPEND_STR} "${test_yaml}" + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" } log_info "submit ${test_yaml} finished" -- 2.23.0

2 2

[PATCH v2 lkp-tests 2/6] iso2rootfs: allow non-dailybuild jobs to be submitted
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] 1. Decouple the code related to dailybuild. The 'iso_url_file' field is only relate to the dailybuild function, and this field is unnecessary when user only want generate a rootfs from offical distribution. But now if this field is none, this job will failed. 2. Modify the field name. 'iso_url_file' -> 'dailybuild_iso_url_file'. [Addition] if the `iso_url_file` field exists, this job will generate a soft link which point to the result rootfs. - soft link name: {iso_version}-dailybuild. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 4 +++- tests/iso2rootfs | 41 +++++++++++++++++++++++++---------------- 2 files changed, 28 insertions(+), 17 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index d4f0ffc0e80c..8796874ce79d 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -2,7 +2,6 @@ suite: iso2rootfs category: benchmark iso2rootfs: iso2rootfs_cgz: - iso_url_file: iso_os: iso_arch: iso_version: @@ -12,6 +11,9 @@ iso2rootfs: initrd_http_host: initrd_http_port: + # dailybuild related fields + dailybuild_iso_url_file: + # submit test yaml related fields test_yaml: test_os: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 9c71b8427d90..5f85cc9db01e 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -44,7 +44,6 @@ check_yaml_vars() "os_arch" "os_version" "iso2rootfs_cgz" - "iso_url_file" "iso_os" "iso_arch" "iso_version" @@ -105,10 +104,10 @@ check_iso_name() get_daily_iso_checksum() { - ISO_URL="$(curl "${iso_url_file}")" + ISO_URL="$(curl "${dailybuild_iso_url_file}")" [ "${iso_os}" == "openeuler" ] && { - local pub_ip=$(echo "${iso_url_file}" | grep -oEw "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}") - ISO_URL="$(curl "${iso_url_file}" |sed -r "s/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/${pub_ip}/g")" + local pub_ip=$(echo "${dailybuild_iso_url_file}" | grep -oEw "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}") + ISO_URL="$(curl "${dailybuild_iso_url_file}" |sed -r "s/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/${pub_ip}/g")" } ISO_NAME=$(basename "$ISO_URL") @@ -149,15 +148,15 @@ mount_rootfs() get_cache_iso_checksum() { - mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-latest.sha256sum" - [ ! -f "${CHECKSUM_FILE_CACHE}" ] || - SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") + [ -f "${CHECKSUM_FILE_CACHE}" ] || return 0 + SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } check_sha256sum_update() { + [ -n "${dailybuild_iso_url_file}" ] || return 0 + get_cache_iso_checksum get_daily_iso_checksum @@ -243,7 +242,10 @@ config_iso2rootfs() pre_works() { check_yaml_vars + + mount_rootfs check_sha256sum_update + check_os config_iso2rootfs @@ -274,9 +276,10 @@ config_iso_conf() config_iso_url() { - local iso_url_file="${I2Q_SRC}/conf/iso/net-iso/${iso_os}/${iso_arch}/${iso_version}" - sed -i "s|^iso_url=.*|iso_url=\"${ISO_URL}\"|g" "${iso_url_file}" - sed -i "s|^iso_checksum_url=.*|iso_checksum_url=\"${ISO_CHECKSUM_URL}\"|g" "${iso_url_file}" + [ -n "${dailybuild_iso_url_file}" ] || return 0 + local i2q_iso_url_file="${I2Q_SRC}/conf/iso/net-iso/${iso_os}/${iso_arch}/${iso_version}" + sed -i "s|^iso_url=.*|iso_url=\"${ISO_URL}\"|g" "${i2q_iso_url_file}" + sed -i "s|^iso_checksum_url=.*|iso_checksum_url=\"${ISO_CHECKSUM_URL}\"|g" "${i2q_iso_url_file}" } config_iso2qcow2() @@ -332,7 +335,7 @@ config_qcow2rootfs() run_qcow2rootfs() { log_info "starting run qcow2rootfs ..." - local qcow2_path="${I2Q_SRC}/output/${ISO_NAME%.iso}.qcow2" + local qcow2_path="$(ls ${I2Q_SRC}/output/*.qcow2)" cd "${CCI_SRC}/container/qcow2rootfs" ./run "${qcow2_path}" "${ROOTFS_DES_DIR}" @@ -497,16 +500,22 @@ submit_test_job_yaml() } ############ post works ############ -post_works() +update_dailybuild_soft_link() { - log_info "starting post works ..." + [ -n "${dailybuild_iso_url_file}" ] || return 0 mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && - rm -f "${soft_link}" && - ln -s "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" + ln -sfT "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" +} + +post_works() +{ + log_info "starting post works ..." + + update_dailybuild_soft_link cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/6] iso2rootfs: change how to get the kernel and initrd.lkp of rootfs
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] We use the container in compass-ci repo to generate rootfs from a qcow2, it will generate the follow two soft links: - {rootfs_dir}/boot/vmlinuz # for vmlinuz file - {rootfs_dir}/initrd.lkp # for initrd file So we can use the below two soft links to get the absolute path of vmlinuz and initrd. [Addition] URL of qcow2rootfs: - https://gitee.com/wu_fengguang/compass-ci/tree/master/container/qcow2rootfs Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 05274164f8c1..9c71b8427d90 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -375,8 +375,12 @@ test_rootfs() log_info "starting test rootfs ..." local test_script="${CGZ_PATH}/kvm.sh" - local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz*|grep -v rescue) - local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/boot/initramfs.lkp*) + local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz) + local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/initrd.lkp) + + [ -f "$kernel_path" ] || die "cannot find kernel of rootfs: $kernel_path" + [ -f "$initrd_lkp_path" ] || die "cannot find initrd_lkp of rootfs: $initrd_lkp_path" + local root_path case ${rootfs_protocol} in "nfs") -- 2.23.0

2 2

[PATCH lkp-tests] common.sh: a new function for count input cpu number
by Wei Jihui 20 Nov '20

20 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/common.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/common.sh b/lib/common.sh index 2a93aafa..b1ca1b13 100755 --- a/lib/common.sh +++ b/lib/common.sh @@ -85,6 +85,18 @@ cpu_list_ref() echo $cpu_list | cut -d ' ' -f $((n+1)) } + +# input: 1-3 output: 3 +# input: 1,3 output: 2 +cpu_list_num() +{ + cpu_list="" + for c in $(expand_cpu_list $1); do + cpu_list+="$c " + done + cpu_list_count "$cpu_list" +} + # if str starts with prefix, output remaining part, otherwise output empty string remove_prefix() { -- 2.23.0

2 1

[PATCH v3 compass-ci] lib/log.rb: add new class for json log
by Wu Zhende 19 Nov '20

19 Nov '20

[Why] The built-in log system prints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example] require './log' log = Log.new log.info("test") output: {"level":"INFO","level_num":1,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"test"} log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"", "job_id":"1","job_state":"submit","caller":["/usr/share/ruby/logger.rb:4:in `format_message'", "/usr/share/ruby/logger.rb:472:in`add'","/usr/share/ruby/logger.rb:545:in `error'","./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/log.rb | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..0ffc0f9 --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' +require 'json' + +# log class to print logs in JSON format +class Log < Logger + LEVEL_INFO = { + 'DEBUG' => 0, + 'INFO' => 1, + 'WARN' => 2, + 'ERROR' => 3, + 'FATAL' => 4, + 'UNKNOWN' => 5 + }.freeze + + FORMATTER = proc { |severity, datetime, progname, msg| + puts + level_num = LEVEL_INFO[severity] + begin + message = JSON.parse(msg) + rescue JSON::ParserError + message = { 'message' => msg } + end + h = { + 'level' => severity.to_s, + 'level_num' => level_num, + 'datetime' => datetime, + 'progname' => progname, + 'message' => '' + } + h.merge!(message) + h.merge!({ 'caller' => caller }) if level_num >= 2 + h.to_json + } + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

2 2

[PATCH lkp-tests] spec/submit: add two job yaml links for spec
by Wei Jihui 19 Nov '20

19 Nov '20

[test cmd] respec $LKP_SRC/spec/submit_spec.rb [output] lkp-tests/spec/submit/link_jobs/redis.yaml => lkp-tests/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml lkp-tests/spec/submit/link_jobs/ftq.yaml => lkp-tests/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml lkp-tests/spec/submit/link_jobs/ftq.yaml => lkp-tests/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml lkp-tests/spec/submit/link_jobs/ftq.yaml => lkp-tests/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml Finished in 3.9 seconds (files took 0.20035 seconds to load) 1 example, 0 failures Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/link_jobs/ftq.yaml | 1 + spec/submit/link_jobs/redis.yaml | 1 + 2 files changed, 2 insertions(+) create mode 120000 spec/submit/link_jobs/ftq.yaml create mode 120000 spec/submit/link_jobs/redis.yaml diff --git a/spec/submit/link_jobs/ftq.yaml b/spec/submit/link_jobs/ftq.yaml new file mode 120000 index 00000000..b472f871 --- /dev/null +++ b/spec/submit/link_jobs/ftq.yaml @@ -0,0 +1 @@ +../../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/link_jobs/redis.yaml b/spec/submit/link_jobs/redis.yaml new file mode 120000 index 00000000..438bdab7 --- /dev/null +++ b/spec/submit/link_jobs/redis.yaml @@ -0,0 +1 @@ +../../../jobs/redis.yaml \ No newline at end of file -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] job.cr: give a default kernel_version and
by Xu Xijian 19 Nov '20

19 Nov '20

The versions of modules and headers should be same with vmlinuz, so give kernel_version a default value and delete soft link modules.cgz and headers.cgz in $nfsroot/boot, they are redundant. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..fc27c5c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -46,6 +46,7 @@ class Job os_version: "20.03", lkp_initrd_user: "latest", docker_image: "centos:7", + kernel_version: "4.19.90-2003" } def initialize(job_content : JSON::Any, id) @@ -312,7 +313,7 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? + suffix = "-#{kernel_version}" self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") if "#{os_mount}" == "initramfs" self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") -- 2.23.0

2 2

[PATCH compass-ci] add api /compare_template
by Lu Weitao 19 Nov '20

19 Nov '20

[Why] user need to compare the specific metrics [how] add new api: /compare_template that can be access like: curl -X POST 'localhost:32767/compare_template' -H 'Content-Type: Application/json' -d '{ "compare_metrics": ["fio.write_iops", "fio.read_iops"], "filter": {"suite": ["fio-basic"], "os_arch": ["aarch64", "x86"]}, "compare_dimensions": [ {"os": "debian", "os_version": "sid"}, {"os": "openeuler", "os_version": 20.03} ], "x_params": ["bs", "test_size"], "title": "Hackbench Performance Testing", "unit": "KB/s"}' Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 32 +++++++++++++++++++++++++++++++ src/lib/web_backend.rb | 18 +++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 7db82ac..d68e9f2 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -71,3 +71,35 @@ end get '/get_repos' do get_repos(params) end + +# POST /compare_template +# - header: "Content-Type: Application/json" +# - body: json +# eg: +# { +# "compare_metrics": ["fio.write_iops", "fio.read_iops"], +# "filter":{ +# "suite": ["fio-basic"], +# "os_arch": ["aarch64", "x86"] +# }, +# "compare_dimensions": [ +# { +# "os": "debian", +# "os_version": "sid" +# }, +# { +# "os": "openeuler", +# "os_version: 20.03 +# } +# ], +# "x_params": ["bs", "test_size"], +# "title": "Hackbench Performance Testing", +# "unit": "KB/s" +# } +# +# Response: +# - compare result +post '/compare_template' do + request.body.rewind # in case someone already read it + compare_template(request.body.read) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 1f7224e..23716cb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -413,3 +414,20 @@ def get_repos(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def compare_template(data) + begin + body = template_body(JSON.parse(data)) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'compare error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def template_body(request_body) + groups_matrices = create_groups_matrices(request_body) + compare_results = compare_metrics_values(groups_matrices) + formatter = FormatEchartData.new(compare_results, request_body) + formatter.format_for_echart.to_json +end -- 2.23.0

1 0

[PATCH v3 compass-ci] lib/compare_data_format: optimize the format for echart
by Lu Weitao 19 Nov '20

19 Nov '20

[Why] With template compare mode, the result(data_set) isn't friendly for echart like: { "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "bs|test_size", "tables": { "1G|64k": { "fio.read_iops": { "average": { "openeuler 20.03": 14010.34254665909 }, "standard_deviation": { "openeuler 20.03": null }, "change": { } }, "fio.write_iops": { "average": { "openeuler 20.03": 13870.135498022726 }, "standard_deviation": { "openeuler 20.03": null }, "change": { } } }, "1G|4k": {...} } } [How] Make the result more friendly for echart like: { "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "test_size|bs", "tables": { "fio.read_iops": { "average": { "dimensions": [ "compare_dimension", "openeuler 20.03" ], "source": [ ["1G|4k", "1G|16k", "1G|32k", "1G|64k", "1G|128k", "1G|256k", "1G|512k", "1G|1024k", "80G|4k"], ["openeuler 20.03", 144076.2903315, 37865.30472368628, 21145.10375497826, 14010.34254665909, 6701.240849466667, 3205.077255, 1367.476930860465, 673.3270888666667, 11601.099817 ] ] }, "standard_deviation": {...}, "change": {...} }, "fio.write_iops": {...} ... } } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_data_format.rb | 77 ++++++++++++++++++++++++++++++++++---- lib/compare_matrixes.rb | 3 +- 2 files changed, 71 insertions(+), 9 deletions(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index 3d82550..484b186 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -5,14 +5,75 @@ # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # +class FormatEchartData + def initialize(metrics_compare_results, template_params) + @metrics_compare_results = metrics_compare_results + @template_params = template_params + @data_set = {} + end -def format_for_echart(metrics_compare_results, template_params) - echart_result = {} - echart_result['title'] = template_params['title'] - echart_result['unit'] = template_params['unit'] - x_params = template_params['x_params'] - echart_result['x_name'] = x_params.join('|') if x_params - echart_result['tables'] = metrics_compare_results + def format_for_echart + echart_result = {} + echart_result['title'] = @template_params['title'] + echart_result['unit'] = @template_params['unit'] + @x_name = @template_params['x_params'] + echart_result['x_name'] = @x_name.join('|') if @x_name + echart_result['tables'] = convert_to_echart_dataset - echart_result + echart_result + end + + def convert_to_echart_dataset + @x_params = sort_x_params((a)metrics_compare_results.keys) + @metrics_compare_results.each_value do |metrics_values| + metrics_values.each do |metric, metric_value| + assign_echart_data_set(metric, metric_value) + end + end + + @data_set + end + + def assign_echart_data_set(metric, metric_value) + @data_set[metric] = {} + metric_value.each do |value_type, values| # value_type can be: average, standard_deviation, change + @data_set[metric][value_type] = { + 'dimensions' => ['compare_dimension'] + } + + dimension_list = values.keys.sort + @data_set[metric][value_type]['dimensions'] += dimension_list + @data_set[metric][value_type]['source'] = assign_echart_source(metric, value_type, dimension_list) + end + end + + def assign_echart_source(metric, value_type, dimensions) + source = [] + source << @x_params + dimensions.each do |dimension| + dimension_values = [dimension] + @x_params.each do |x_param| + if @metrics_compare_results[x_param][metric] + dimension_values << @metrics_compare_results[x_param][metric][value_type][dimension] + end + end + source << dimension_values + end + + source + end +end + +# input: x_params_list +# eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] +# output: +# ["1G|4K", "1G|128k", "1G|1024k", "2G|4k"] +def sort_x_params(x_params_list) + x_params_hash = {} + x_params_list.each do |param| + params = param.gsub(/[a-zA-Z]+$/, '').split('|').map(&:to_i) + x_params_hash[params] = param + end + + x_params_hash.sort.map { |h| h[1] } end diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..762c087 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -498,7 +498,8 @@ def get_dimentions_combination(dimension_list) end def show_compare_result(metrics_compare_results, template_params) - echart_results = format_for_echart(metrics_compare_results, template_params) + formatter = FormatEchartData.new(metrics_compare_results, template_params) + echart_results = formatter.format_for_echart print JSON.pretty_generate(echart_results) end -- 2.23.0

1 0

[PATCH v2 lkp-tests 6/6] iso2rootfs: allow the use of job.yaml and test script from the internet
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] Not all the yaml and script of test job will be committed to the lkp-tests repo. [How] Support three optional fields for every test in job.yaml: - testX_git_url: the url of git repo which contains yaml and script; - testX_git_yaml: the relative path of the yaml file in the git repo; - testX_git_script: the relative path of the test script in the git repo; Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 5 +++++ tests/iso2rootfs | 26 ++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index f88ff02ad592..29bf84013f8e 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -26,3 +26,8 @@ iso2rootfs: test1_yaml: test1_os_mount: test1_testbox: + ## follow three fields is required when your job.yaml and script for + ## this test are from the internet. + test1_git_url: + test1_git_yaml: + test1_git_script: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 48c1fc772890..e30cde9af1e6 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -451,10 +451,36 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } +prepare_submit() +{ + local submit_dir="/tmp/iso2rootfs/submit_tmp" + mkdir -p "${submit_dir}" + + local submit_git_dir="${submit_dir}/${test_git_url##*/}" + submit_git_dir="${submit_git_dir%.git}" + + [ -d "${submit_git_dir}" ] || { + cd "${submit_dir}" + git clone "${test_git_url}" + } + + local test_git_yaml="${submit_git_dir}/$(eval echo "\$${test_num}_git_yaml")" + local test_git_script="${submit_git_dir}/$(eval echo "\$${test_num}_git_script")" + + [ -f "${test_git_yaml}" ] || die "cannot find file: ${test_num}_git_yaml." + cp "${test_git_yaml}" "${LKP_SRC}/jobs/" + + [ -f "${test_git_script}" ] || die "cannot find file: ${test_num}_git_script." + cp "${test_git_script}" "${LKP_SRC}/tests/" +} + submit_one_yaml() { log_info "starting submit ${test_yaml} ..." + local test_git_url="$(eval echo "\$${test_num}_git_url")" + [ -z "${test_git_url}" ] || prepare_submit + cd "${LKP_SRC}/jobs" [ -f "${test_yaml}" ] || die "cannot find yaml in LKP_SRC/jobs: ${test_yaml}." -- 2.23.0

1 0

[PATCH v2 lkp-tests 5/6] iso2rootfs: allow submit multiple test job.yaml for result rootfs
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] Result rootfs of dailybuild requires multiple test jobs. [How] 1. jobs/iso2rootfs.yaml. Before: test_yaml, test_os_mount, test_testbox After: test1_yaml, test1_os_mount, test1_testbox test2_yaml, test2_os_mount, test2_testbox ... 2. tests/iso2rootfs. - Use while-loop to traverse if $testX_yaml exists. - [ -n "${test1_yaml}" ] || return 0. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 8 ++++--- tests/iso2rootfs | 52 +++++++++++++++++++++++++++++--------------- 2 files changed, 40 insertions(+), 20 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 222ac309d33e..f88ff02ad592 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -21,6 +21,8 @@ iso2rootfs: dailybuild_iso_url_file: # submit test yaml related fields - test_yaml: - test_os_mount: - test_testbox: + ## 1. you can add as many jobs as you like. + ## 2. the following three fields is requied for every test job. + test1_yaml: + test1_os_mount: + test1_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 4f6e3e578ce4..48c1fc772890 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -56,7 +56,7 @@ check_yaml_vars() for yaml_t in "${yaml_vars[@]}" do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." + [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot find value of var: ${yaml_t}." done } @@ -451,17 +451,33 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } -submit_test_job_yaml() +submit_one_yaml() { - [ -n ${test_yaml} ] || return 0 - log_info "starting submit ${test_yaml} ..." - [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + cd "${LKP_SRC}/jobs" + [ -f "${test_yaml}" ] || die "cannot find yaml in LKP_SRC/jobs: ${test_yaml}." + + local test_os_mount="$(eval echo "\$${test_num}_os_mount")" + local test_testbox="$(eval echo "\$${test_num}_testbox")" + + [ -n "${test_os_mount}" ] || die "cannot find value of var: ${test_num}_os_mount." + [ -n "${test_testbox}" ] || die "cannot find value of var: ${test_num}_testbox." - [ -n "${test_os_mount}" ] || die "cannot fine value of var: test_os_mount." - [ -n "${test_testbox}" ] || die "cannot fine value of var: test_testbox." + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" + log_info "submit ${test_yaml} finished" +} + +submit_yamls() +{ + [ -n "$test1_yaml" ] || return 0 + + log_info "starting submit test yamls ..." + + [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh ./sbin/install-dependencies.sh @@ -469,17 +485,19 @@ submit_test_job_yaml() generate_global_yaml } - cd "${LKP_SRC}/jobs" && { - [ -f "${test_yaml}" ] || - die "cannot find test yaml in LKP_SRC/jobs. - test yaml: ${test_yaml}." + local test_yaml_index=1 + local test_num="test${test_yaml_index}" + local test_yaml="$(eval echo "\$${test_num}_yaml")" - "${LKP_SRC}/sbin/submit" \ - $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ - "${test_yaml}" - } + while [ -n "${test_yaml}" ] + do + submit_one_yaml + test_yaml_index=$((${test_yaml_index} +1)) + test_num="test${test_yaml_index}" + test_yaml="$(eval echo "\$${test_num}_yaml")" + done - log_info "submit ${test_yaml} finished" + log_info "submit test yamls finished" } ############ post works ############ @@ -519,7 +537,7 @@ main() config_rootfs test_rootfs - submit_test_job_yaml + submit_yamls post_works } -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/6] iso2rootfs: add a switch to configure rootfs
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] In some cases, we do not need to configure the result rootfs, so add a switch to configure rootfs. [How] Add 'config_rootfs' field in job.yaml. and support the following two fields now: - dns: will config /etc/resolv.conf. - no_selinux: will disable selinux in /etc/selinux/config. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 6 ++++++ tests/iso2rootfs | 2 ++ 2 files changed, 8 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 8796874ce79d..55bf71517174 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -11,6 +11,12 @@ iso2rootfs: initrd_http_host: initrd_http_port: + # config rootfs related fields + ## supportted fields: + ## - dns: will config /etc/resolv.conf. + ## - no_selinux: will disable selinux in /etc/selinux/config. + config_rootfs: + # dailybuild related fields dailybuild_iso_url_file: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 5f85cc9db01e..c2e823c93ff9 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -345,6 +345,7 @@ run_qcow2rootfs() ############ config rootfs ############ config_dns_resolver() { + echo "${config_rootfs}" | grep -qw 'dns' || return 0 cat <<-EOF > "${ROOTFS_DES_DIR}/etc/resolv.conf" nameserver 114.114.114.114 nameserver 8.8.8.8 @@ -353,6 +354,7 @@ config_dns_resolver() disable_selinux() { + echo "${config_rootfs}" | grep -qw 'no_selinux' || return 0 sed -i 's/^SELINUX=.*/SELINUX=disabled/g' "${ROOTFS_DES_DIR}/etc/selinux/config" } -- 2.23.0

1 0

[PATCH v6 compass-ci] sparrow: fix 3-code/dev-env no such file error
by Liu Yinsi 19 Nov '20

19 Nov '20

[why] when execute 3-code/dev-env in openeuler and centos machine, error message as follows: ./dev-env: line 45: ./os/openEuler: No such file or directory the purpose of ". $(dirname ${BASH_SOURCE[0]})/os/${ID}" is to fix crystal-shards build error when adapt in debian. but it's unnecessary for other linux os. Please see: [PATCH compass-ci 3/4] sparrow: add domain name resolution for Docker images [how] add judge to avoid redundant error message. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8928816..218b6c1 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -44,4 +44,6 @@ EOF source /etc/os-release -. $(dirname ${BASH_SOURCE[0]})/os/${ID} +path=$(dirname ${BASH_SOURCE[0]})/os/${ID} +[ -x "$path" ] || exit 0 +. "$path" -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/es-query: fix NoMethodError
by Lu Weitao 19 Nov '20

19 Nov '20

[Why] Traceback (most recent call last): 4: from /c/compass-ci/sbin/compare:70:in `<main>' 3: from /c/compass-ci/lib/compare.rb:97:in `compare_by_template' 2: from /c/compass-ci/lib/compare.rb:128:in `create_groups_matrices' 1: from /c/compass-ci/lib/es_query.rb:21:in `multi_field_query' /c/compass-ci/lib/es_query.rb:54:in `build_mutli_field_subquery_body': undefined method `each' for nil:NilClass (NoMethodError) "compare_temlpate.yaml" may pass empty filter as input parameter [How] exit when the filter is empty Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 037213d..c027aa9 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -18,6 +18,10 @@ class ESQuery # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) def multi_field_query(items, size: 10_000) + unless items + warn 'empty filter!' + exit + end query_fields = build_mutli_field_subquery_body items query = { query: { -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] job.cr: give a default kernel_version and
by Xu Xijian 19 Nov '20

19 Nov '20

The versions of modules and headers should be same with vmlinuz, so give kernel_version a default value and delete soft link modules.cgz and headers.cgz in $nfsroot/boot, they are redundant. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..fc27c5c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -46,6 +46,7 @@ class Job os_version: "20.03", lkp_initrd_user: "latest", docker_image: "centos:7", + kernel_version: "4.19.90-2003" } def initialize(job_content : JSON::Any, id) @@ -312,7 +313,7 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? + suffix = "-#{kernel_version}" self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") if "#{os_mount}" == "initramfs" self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") -- 2.23.0

2 2

[PATCH lkp-tests 3/3] add the case of yaml_merge_included_files.yaml
by Hu Xuejiao 19 Nov '20

19 Nov '20

[description] it is situation of yaml_merge_included_files Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 3 +++ spec/submit/merge_yaml/yaml_merge_included_files.yaml | 1 + 2 files changed, 4 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..0415606a --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,3 @@ +suite: test +category: functional +test: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml -- 2.23.0

3 2

[PATCH lkp-tests 1/3] create merge_yaml and submit merge_yaml/*.yaml
by Hu Xuejiao 19 Nov '20

19 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories [how] Create merge_yaml and submit merge_yaml/*.yaml Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit_spec.rb | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 983d050e..b2ad7168 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -34,6 +34,21 @@ describe 'submit job spec' do ] system(*submit_cmd) traverse_file(output_dir) + end + end + it 'merge yaml spec' do + Dir.glob("#{LKP_SRC}/spec/submit/merge_yaml/*.yaml").each do |yaml_file| + job_name = File.basename(yaml_file, '.yaml') + output_dir = "#{LKP_SRC}/spec/submit/merge_yaml/#{job_name}/output" + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) + end end - end end -- 2.23.0

2 1

[PATCH lkp-tests] jobs: fix the spell error in the original lmbench3.ayml
by Bai Jing 19 Nov '20

19 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/lmbench3.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/lmbench3.yaml b/jobs/lmbench3.yaml index de1c9aa8..f84c6910 100644 --- a/jobs/lmbench3.yaml +++ b/jobs/lmbench3.yaml @@ -4,7 +4,7 @@ category: benchmark test_memory_size: 50% -# Test mdoe all,hardware,os were commented temporally,since it will take very long time +# Test mode all, hardware, os were commented temporally, since it will take very long time # more than 2 hours on test machine lkp-bdw-ep4. # lmbench3: # - mode: all -- 2.23.0

2 2

[PATCH v2 lkp-tests] jobs: add new job yaml for lmbench3 performance test
by Bai Jing 19 Nov '20

19 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/lmbench3-performance.yaml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 jobs/lmbench3-performance.yaml diff --git a/jobs/lmbench3-performance.yaml b/jobs/lmbench3-performance.yaml new file mode 100644 index 00000000..c733d7d2 --- /dev/null +++ b/jobs/lmbench3-performance.yaml @@ -0,0 +1,17 @@ +suite: lmbench3 +category: benchmark + +test_memory_size: 4096 + +# Test mdoe all,hardware,os were commented temporally,since it will take very long time +# more than 2 hours on test machine lkp-bdw-ep4. +# lmbench3: +# - mode: all +# - mode: hardware +# - mode: os + +nr_threads: +- 1 + +lmbench3: + mode: all -- 2.23.0

4 6

[PATCH compass-ci] web_backend.rb: add space line after return cmd
by Xu Xijian 19 Nov '20

19 Nov '20

When there is a return cmd in a function, and it's not the last line of this function, there should have a blank line after return cmd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/web_backend.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index b8df6f4..1f7224e 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -220,7 +220,9 @@ end def wrong_size?(size, from) return true if from.negative? || size.negative? + return true if from > 1000000 || size > 1000000 + return true if size + from > 1000000 end @@ -344,6 +346,7 @@ def get_repo_url(urls) urls.each do |url| return url if url[0, 4] == 'http' end + urls[0] end -- 2.23.0

1 0

[PATCH v3 compass-ci] sparrow: fix 3-code/dev-env no such file error
by Liu Yinsi 19 Nov '20

19 Nov '20

when execute 3-code/dev-env, error message as follows: ./dev-env: line 45: ./os/openEuler: No such file or directory Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 77027d9..98da280 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -42,4 +42,5 @@ EOF source /etc/os-release -. $(dirname ${BASH_SOURCE[0]})/os/${ID} +path=$(dirname ${BASH_SOURCE[0]})/os/${ID} +[ -x "$path" ] && . "$path" -- 2.23.0

3 5

[PATCH compass-ci] lib/log.rb: add new class for json log
by Wu Zhende 19 Nov '20

19 Nov '20

[Why] The built-in log system peints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example] require './log' log = Log.new use: log.info("tset") output: {"level":"INFO","level_num":1,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"test"} use: log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"", "job_id":"1","job_state":"submit","caller":["/usr/share/ruby/logger.rb:4:in `format_message'", "/usr/share/ruby/logger.rb:472:in`add'","/usr/share/ruby/logger.rb:545:in `error'","./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/log.rb | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..0ffc0f9 --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' +require 'json' + +# log class to print logs in JSON format +class Log < Logger + LEVEL_INFO = { + 'DEBUG' => 0, + 'INFO' => 1, + 'WARN' => 2, + 'ERROR' => 3, + 'FATAL' => 4, + 'UNKNOWN' => 5 + }.freeze + + FORMATTER = proc { |severity, datetime, progname, msg| + puts + level_num = LEVEL_INFO[severity] + begin + message = JSON.parse(msg) + rescue JSON::ParserError + message = { 'message' => msg } + end + h = { + 'level' => severity.to_s, + 'level_num' => level_num, + 'datetime' => datetime, + 'progname' => progname, + 'message' => '' + } + h.merge!(message) + h.merge!({ 'caller' => caller }) if level_num >= 2 + h.to_json + } + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/log.rb: add new class for json log
by Wu Zhende 19 Nov '20

19 Nov '20

[Why] The built-in log system peints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example] require './log' log = Log.new log.info("tset") output: {"level":"INFO","level_num":1,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"test"} log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"", "job_id":"1","job_state":"submit","caller":["/usr/share/ruby/logger.rb:4:in `format_message'", "/usr/share/ruby/logger.rb:472:in`add'","/usr/share/ruby/logger.rb:545:in `error'","./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/log.rb | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..0ffc0f9 --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' +require 'json' + +# log class to print logs in JSON format +class Log < Logger + LEVEL_INFO = { + 'DEBUG' => 0, + 'INFO' => 1, + 'WARN' => 2, + 'ERROR' => 3, + 'FATAL' => 4, + 'UNKNOWN' => 5 + }.freeze + + FORMATTER = proc { |severity, datetime, progname, msg| + puts + level_num = LEVEL_INFO[severity] + begin + message = JSON.parse(msg) + rescue JSON::ParserError + message = { 'message' => msg } + end + h = { + 'level' => severity.to_s, + 'level_num' => level_num, + 'datetime' => datetime, + 'progname' => progname, + 'message' => '' + } + h.merge!(message) + h.merge!({ 'caller' => caller }) if level_num >= 2 + h.to_json + } + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

2 1

[PATCH compass-ci] qcow2rootfs: change yum source to fix build timeout error
by Yu Chuan 19 Nov '20

19 Nov '20

[Error Msg] Step 4/7 : RUN yum makecache && yum install -y --skip-broken libvirt libguestfs-tools xz expect openssl && yum clean all ---> Running in c443a95dbd5f CentOS-8 - AppStream2.2 kB/s | 2.5 MB 19:23 Errors during downloading metadata for repository 'AppStream': - Curl error (28): Timeout was reached for http://mirror.centos.org/centos/8/AppStream/aarch64/os/repodata/ce3501f1421… [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds] - Curl error (28): Timeout was reached for http://mirror.centos.org/centos/8/AppStream/aarch64/os/repodata/cc0e75f5237… [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds] - Curl error (18): Transferred a partial file for http://mirror.centos.org/centos/8/AppStream/aarch64/os/repodata/ce3501f1421… [transfer closed with 1750223 bytes remaining to read] Error: Failed to download metadata for repo 'AppStream': Yum repo downloading error: Downloading error(s): repodata/cc0e75f5237c3f5ddfea6ab0de1056df5bb8bbf1273b5ecb907ed7c1c72897f3-primary.xml.gz - Cannot download, all mirrors were already tried without success; repodata/ce3501f14219e71c21c1e40c6f91ba41bd37b59345c5a7dd6d680ef4b279b95a-filelists.xml.gz - Cannot download, all mirrors were already tried without success Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/qcow2rootfs/Dockerfile | 4 +- .../etc/yum.repos.d/CentOS-AltArch-7.repo | 43 +++++++++++++++++++ 2 files changed, 46 insertions(+), 1 deletion(-) create mode 100644 container/qcow2rootfs/root/etc/yum.repos.d/CentOS-AltArch-7.repo diff --git a/container/qcow2rootfs/Dockerfile b/container/qcow2rootfs/Dockerfile index a0f8b042520b..38418c7eb226 100644 --- a/container/qcow2rootfs/Dockerfile +++ b/container/qcow2rootfs/Dockerfile @@ -1,12 +1,14 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM centos:8 +FROM centos:7 MAINTAINER Yu Chuan <13186087857(a)163.com>, Shi Zhichao <angus_robot(a)163.com> RUN rpm --import /etc/pki/rpm-gpg/RPM* +RUN rm -rf /etc/yum.repos.d/* +COPY root / RUN yum makecache && \ yum install -y --skip-broken libvirt libguestfs-tools xz expect openssl && \ yum clean all diff --git a/container/qcow2rootfs/root/etc/yum.repos.d/CentOS-AltArch-7.repo b/container/qcow2rootfs/root/etc/yum.repos.d/CentOS-AltArch-7.repo new file mode 100644 index 000000000000..b99c0a60895d --- /dev/null +++ b/container/qcow2rootfs/root/etc/yum.repos.d/CentOS-AltArch-7.repo @@ -0,0 +1,43 @@ +# CentOS-Base.repo +# +# The mirror system uses the connecting IP address of the client and the +# update status of each mirror to pick mirrors that are updated to and +# geographically close to the client. You should use this for CentOS updates +# unless you are manually picking other mirrors. +# +# If the mirrorlist= does not work for you, as a fall back you can try the +# remarked out baseurl= line instead. +# +# + +[base] +name=CentOS-$releasever - Base - mirrors.huaweicloud.com +baseurl=https://mirrors.huaweicloud.com/centos-altarch/7/os/$basearch/ +#mirrorlist=https://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os +gpgcheck=0 +gpgkey=https://mirrors.huaweicloud.com/centos-altarch/7/os/$basearch/RPM-GPG-KEY-CentOS-7-$basearch + +#released updates +[updates] +name=CentOS-$releasever - Updates - mirrors.huaweicloud.com +baseurl=https://mirrors.huaweicloud.com/centos-altarch/7/updates/$basearch/ +#mirrorlist=https://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updates +gpgcheck=0 +gpgkey=https://mirrors.huaweicloud.com/centos-altarch/7/os/$basearch/RPM-GPG-KEY-CentOS-7-$basearch + +#additional packages that may be useful +[extras] +name=CentOS-$releasever - Extras - mirrors.huaweicloud.com +baseurl=https://mirrors.huaweicloud.com/centos-altarch/7/extras/$basearch/ +#mirrorlist=https://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extras +gpgcheck=0 +gpgkey=https://mirrors.huaweicloud.com/centos-altarch/7/os/$basearch/RPM-GPG-KEY-CentOS-7-$basearch + +#additional packages that extend functionality of existing packages +[centosplus] +name=CentOS-$releasever - Plus - mirrors.huaweicloud.com +baseurl=https://mirrors.huaweicloud.com/centos-altarch/7/centosplus/$basearch/ +#mirrorlist=https://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplus +gpgcheck=0 +enabled=0 +gpgkey=https://mirrors.huaweicloud.com/centos-altarch/7/os/$basearch/RPM-GPG-KEY-CentOS-7-$basearch -- 2.23.0

1 0

[PATCH lkp-tests] jobs: add new job yaml for lmbench3 performance test
by Bai Jing 19 Nov '20

19 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/lmbench3-performance.yaml | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 jobs/lmbench3-performance.yaml diff --git a/jobs/lmbench3-performance.yaml b/jobs/lmbench3-performance.yaml new file mode 100644 index 00000000..63af5866 --- /dev/null +++ b/jobs/lmbench3-performance.yaml @@ -0,0 +1,26 @@ +suite: lmbench3 +testcase: lmbench3 +category: benchmark +os: openeuler +os_version: 20.03 +os_arch: aarch64 + +testbox: vm-2p8g-1 +queue: vm-2p8g~baijing + +test_memory_size: 4096 +on_fail: + sleep: + 300 +# Test mdoe all,hardware,os were commented temporally,since it will take very long time +# more than 2 hours on test machine lkp-bdw-ep4. +# lmbench3: +# - mode: all +# - mode: hardware +# - mode: os + +nr_threads: +- 1 + +lmbench3: + mode: all -- 2.23.0

3 4

[PATCH compass-ci] block_helper.cr: simplify if block to one line
by Xu Xijian 19 Nov '20

19 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/block_helper.cr | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/lib/block_helper.cr b/src/lib/block_helper.cr index 37ad367..596ef7e 100644 --- a/src/lib/block_helper.cr +++ b/src/lib/block_helper.cr @@ -28,9 +28,7 @@ class BlockHelper @block_helper[uuid] = fiber end - if yield == true - spawn fiber.run - end + spawn fiber.run if yield == true until fiber.dead? Fiber.yield -- 2.23.0

1 0

[PATCH lkp-tests 2/3] create a multi-level directory
by Hu Xuejiao 19 Nov '20

19 Nov '20

[why] Because after using "rspec spec/submit_spec.rb", i need the output of different case.yaml to be placed in output (spec/submit/merge_yaml/*.yaml/output) Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 9bba1018..050e3d44 100755 --- a/sbin/submit +++ b/sbin/submit @@ -9,6 +9,7 @@ require "#{LKP_SRC}/lib/distro_info" require "#{LKP_SRC}/lib/scheduler_client" require 'optparse' require 'yaml' +require 'fileutils' opt_set_key_value = {} opt_cmdline_defaults = {} @@ -35,7 +36,7 @@ options = OptionParser.new do |opts| exit 1 end opt_output_dir = dir - Dir.mkdir(dir) unless File.directory? dir + FileUtils.mkdir_p(dir) unless File.directory? dir end opts.on('-a', '--auto-define-files', 'auto add define_files') do -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/cci-makepkg: simplify centos7 test
by Lin Jiaxin 19 Nov '20

19 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 905ad689..e49f6699 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -90,20 +90,20 @@ update_shared_pkg() centos_update_gcc() { - if grep -sqF "CentOS Linux release 7" /etc/centos-release; then - [ "$os_mount" = "initramfs" ] && { - yum install -y centos-release-scl-rh - yum install -y "devtoolset-8-gcc*" - } - source /opt/rh/devtoolset-8/enable - fi + grep -sqF "CentOS Linux release 7" /etc/centos-release || return + + [ "$os_mount" = "initramfs" ] && { + yum install -y centos-release-scl-rh + yum install -y "devtoolset-8-gcc*" + } + source /opt/rh/devtoolset-8/enable } [ -n "$PKGBUILD_TAG" ] && download_pkgfile distro_install_depends lkp-dev -[ "$os" = "centos" ] && centos_update_gcc +centos_update_gcc distro_install_depends ${benchmark}-dev -- 2.23.0

3 3

[PATCH compass-ci] container/result-cifs: add the container to share /srv/result
by Xiao Shenwei 19 Nov '20

19 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result in order to reduce the coupling, need split them [how] one folder mount by one container [usage] mount -t cifs o guest,port=447 //ip/result /tmp/result Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/result-cifs/Dockerfile | 22 ++++++++++++++++++++++ container/result-cifs/build | 5 +++++ container/result-cifs/smb.conf | 29 +++++++++++++++++++++++++++++ container/result-cifs/start | 24 ++++++++++++++++++++++++ 4 files changed, 80 insertions(+) create mode 100644 container/result-cifs/Dockerfile create mode 100755 container/result-cifs/build create mode 100644 container/result-cifs/smb.conf create mode 100755 container/result-cifs/start diff --git a/container/result-cifs/Dockerfile b/container/result-cifs/Dockerfile new file mode 100644 index 0000000..a0d47c2 --- /dev/null +++ b/container/result-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +RUN adduser -u 1090 -D lkp +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash +RUN rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 447/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/result-cifs/build b/container/result-cifs/build new file mode 100755 index 0000000..d97b6dd --- /dev/null +++ b/container/result-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t result-cifs . diff --git a/container/result-cifs/smb.conf b/container/result-cifs/smb.conf new file mode 100644 index 0000000..4ccd943 --- /dev/null +++ b/container/result-cifs/smb.conf @@ -0,0 +1,29 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[result] + path = /srv/result/ + comment = result + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/result-cifs/start b/container/result-cifs/start new file mode 100755 index 0000000..2d6acba --- /dev/null +++ b/container/result-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm result-cifs + +cmd=( + docker run + -d + -p 447:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result + --name result-cifs + --restart=always + result-cifs +) + +"${cmd[@]}" -- 2.23.0

2 2

[PATCH v3 compass-ci 3/3] kernel_version.md: delete unused modules.cgz and headers.cgz
by Xu Xijian 19 Nov '20

19 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md index 3cf5e26..3e94ca8 100644 --- a/doc/job/kernel_version.md +++ b/doc/job/kernel_version.md @@ -30,11 +30,9 @@ Related files: - In initramfs boot process, every kernel version is related with a vmlinuz, module and headers. - Files like below under $boot_dir, an example $boot_dir can be "/srv/os/openeuler/aarch64/20.03/boot". ├── headers-4.19.90-2003.cgz -├── headers.cgz -> headers-4.19.90-2003.cgz ├── modules-4.19.90-2003.cgz -├── modules.cgz -> modules-4.19.90-2003.cgz ├── vmlinuz-4.19.90-2003 -└── vmlinuz -> vmlinuz-4.19.90-2003 +└── vmlinuz -> vmlinuz-4.19.90-2003 Usage example: - submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] dracut-initrd: delete unused kernel module iscsi from dracut
by Xu Xijian 19 Nov '20

19 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/run b/container/dracut-initrd/run index b0c9b30..6301ee2 100755 --- a/container/dracut-initrd/run +++ b/container/dracut-initrd/run @@ -43,7 +43,7 @@ cmd=( -v $root/lib/modules:/lib/modules debian:dracut bash -c - "dracut --force --kver $kver -k $kernel_modules $initrd_output && + "dracut -o iscsi --omit-drivers iscsi --force --kver $kver -k $kernel_modules $initrd_output && chmod 644 $initrd_output" # example: -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] dracut-initrd: delete unused kernel module iscsi from dracut
by Xu Xijian 19 Nov '20

19 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/run b/container/dracut-initrd/run index b0c9b30..6301ee2 100755 --- a/container/dracut-initrd/run +++ b/container/dracut-initrd/run @@ -43,7 +43,7 @@ cmd=( -v $root/lib/modules:/lib/modules debian:dracut bash -c - "dracut --force --kver $kver -k $kernel_modules $initrd_output && + "dracut -o iscsi --omit-drivers iscsi --force --kver $kver -k $kernel_modules $initrd_output && chmod 644 $initrd_output" # example: -- 2.23.0

1 0

[PATCH lkp-tests 2/6] iso2rootfs: allow non-dailybuild jobs to be submitted
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] We can only generate the rootfs of dailybuild, so decouple the code related to dailybuild. [How] Use 'iso_url_file' as a switch. - If this field is specified, so generate the rootfs of dailybuild; - If not, generate the rootfs of the official iso of the target version. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 4 +++- tests/iso2rootfs | 24 +++++++++++++----------- 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index d4f0ffc0e80c..4e8729997a00 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -2,7 +2,6 @@ suite: iso2rootfs category: benchmark iso2rootfs: iso2rootfs_cgz: - iso_url_file: iso_os: iso_arch: iso_version: @@ -12,6 +11,9 @@ iso2rootfs: initrd_http_host: initrd_http_port: + # dailybuild related fields + iso_url_file: + # submit test yaml related fields test_yaml: test_os: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 9c71b8427d90..d42c38909f68 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -44,7 +44,6 @@ check_yaml_vars() "os_arch" "os_version" "iso2rootfs_cgz" - "iso_url_file" "iso_os" "iso_arch" "iso_version" @@ -149,8 +148,6 @@ mount_rootfs() get_cache_iso_checksum() { - mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-latest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") @@ -243,7 +240,10 @@ config_iso2rootfs() pre_works() { check_yaml_vars - check_sha256sum_update + + mount_rootfs + [ -n "${iso_url_file}" ] && check_sha256sum_update + check_os config_iso2rootfs @@ -286,7 +286,7 @@ config_iso2qcow2() config_pip_proxy config_iso_conf - config_iso_url + [ -n "${iso_url_file}" ] && config_iso_url } run_iso2qcow2() @@ -332,7 +332,7 @@ config_qcow2rootfs() run_qcow2rootfs() { log_info "starting run qcow2rootfs ..." - local qcow2_path="${I2Q_SRC}/output/${ISO_NAME%.iso}.qcow2" + local qcow2_path="$(ls ${I2Q_SRC}/output/*.qcow2)" cd "${CCI_SRC}/container/qcow2rootfs" ./run "${qcow2_path}" "${ROOTFS_DES_DIR}" @@ -501,12 +501,14 @@ post_works() { log_info "starting post works ..." - mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" + [ -n "${iso_url_file}" ] && { + mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" - local soft_link="${iso_version}-dailybuild" - cd "${ROOTFS_LOCAL_PATH}" && - rm -f "${soft_link}" && - ln -s "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" + local soft_link="${iso_version}-dailybuild" + cd "${ROOTFS_LOCAL_PATH}" && + rm -f "${soft_link}" && + ln -s "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" + } cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

4 8

[PATCH v4 lkp-tests] jobs: add new job yaml for lmbench3 performance test
by Bai Jing 19 Nov '20

19 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/lmbench3-performance.yaml | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 jobs/lmbench3-performance.yaml diff --git a/jobs/lmbench3-performance.yaml b/jobs/lmbench3-performance.yaml new file mode 100644 index 00000000..60ef80fe --- /dev/null +++ b/jobs/lmbench3-performance.yaml @@ -0,0 +1,16 @@ +suite: lmbench3 +category: benchmark + +test_memory_size: 4096 + +# Test mode all, hardware, os were commented temporally, since it will take very long time +# lmbench3: +# - mode: all +# - mode: hardware +# - mode: os + +nr_threads: + - 1 + +lmbench3: + mode: all -- 2.23.0

1 0

[PATCH v2 lkp-tests] add on_fail to job2sh when sh_on_fail is not nil
by Hu Xuejiao 19 Nov '20

19 Nov '20

[description] When sh_on_fail is nil, on_fail does not need to add to the job.yaml Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/sbin/submit b/sbin/submit index a325e912..9bba1018 100755 --- a/sbin/submit +++ b/sbin/submit @@ -121,11 +121,14 @@ ARGV.each do |jobfile| 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files, - 'on_fail' => sh_on_fail + 'define_files' => sh_define_files } } + if sh_on_fail + sh_hash['job2sh']['on_fail'] = sh_on_fail + end + # merge job info job_hash = job.to_hash job_hash_list << job_hash -- 2.23.0

1 0

[PATCH v7 compass-ci 2/2] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 19 Nov '20

19 Nov '20

[why] Avoid to execute some unnecessary operations and improve performance. [how] Keep the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 62d174c..86cfc93 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

2 2

[PATCH lkp-tests] borrow-1h.yaml: fix unable to receive email
by Zhang Yale 19 Nov '20

19 Nov '20

[why] $my_email is used in sshd, so borrow_1h.yaml should use my_email field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-1h.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/borrow-1h.yaml b/jobs/borrow-1h.yaml index c9d6ff8e..6c2284f3 100644 --- a/jobs/borrow-1h.yaml +++ b/jobs/borrow-1h.yaml @@ -9,6 +9,6 @@ sshd: nil end %> - email: + my_email: # sleep at the bottom sleep: 1h -- 2.23.0

2 2

[PATCH v5 compass-ci] sparrow: fix 3-code/dev-env no such file error
by Liu Yinsi 19 Nov '20

19 Nov '20

[why] when execute 3-code/dev-env in openeuler and centos machine, error message as follows: ./dev-env: line 45: ./os/openEuler: No such file or directory the purpose of ". $(dirname ${BASH_SOURCE[0]})/os/${ID}" is to fix crystal-shards build error when adapt in debian. but it's unnecessary for other linux os. Please see: [PATCH compass-ci 3/4] sparrow: add domain name resolution for Docker images [how] hide this redundant and confusion error message. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8928816..08084b0 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -44,4 +44,4 @@ EOF source /etc/os-release -. $(dirname ${BASH_SOURCE[0]})/os/${ID} +. $(dirname ${BASH_SOURCE[0]})/os/${ID} 2> /dev/null -- 2.23.0

3 2

[PATCH v3 lkp-tests] jobs: add new job yaml for lmbench3 performance test
by Bai Jing 19 Nov '20

19 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/lmbench3-performance.yaml | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 jobs/lmbench3-performance.yaml diff --git a/jobs/lmbench3-performance.yaml b/jobs/lmbench3-performance.yaml new file mode 100644 index 00000000..76e5348b --- /dev/null +++ b/jobs/lmbench3-performance.yaml @@ -0,0 +1,17 @@ +suite: lmbench3 +category: benchmark + +test_memory_size: 4096 + +# Test mdoe all, hardware, os were commented temporally, since it will take very long time +# more than 2 hours on test machine lkp-bdw-ep4. +# lmbench3: +# - mode: all +# - mode: hardware +# - mode: os + +nr_threads: +- 1 + +lmbench3: + mode: all -- 2.23.0

1 0

[PATCH compass-ci] container/rabbitmq: separate docker pull and run
by Liu Yinsi 19 Nov '20

19 Nov '20

"docker pull" in build script, "docker run" in start script looks more reasonable. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/rabbitmq/build | 5 +++++ container/rabbitmq/start | 1 - 2 files changed, 5 insertions(+), 1 deletion(-) create mode 100755 container/rabbitmq/build diff --git a/container/rabbitmq/build b/container/rabbitmq/build new file mode 100755 index 0000000..c0bc179 --- /dev/null +++ b/container/rabbitmq/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker pull rabbitmq:3-management diff --git a/container/rabbitmq/start b/container/rabbitmq/start index 7e25417..6b875f3 100755 --- a/container/rabbitmq/start +++ b/container/rabbitmq/start @@ -5,7 +5,6 @@ . $CCI_SRC/container/defconfig.sh docker_rm rabbitmq -docker pull rabbitmq:3-management cmd=( docker run -- 2.23.0

2 1

[PATCH lkp-tests] jobs: add new job yaml for unixbench performance tuning
by Bai Jing 19 Nov '20

19 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/unixbench-tuning.yaml | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 jobs/unixbench-tuning.yaml diff --git a/jobs/unixbench-tuning.yaml b/jobs/unixbench-tuning.yaml new file mode 100644 index 00000000..d18e0dfc --- /dev/null +++ b/jobs/unixbench-tuning.yaml @@ -0,0 +1,10 @@ +suite: unixbench-tuning +category: benchmark + +runtime: 300s + +nr_task: + - 1 + - 96 + +unixbench-tuning: -- 2.23.0

1 0

[PATCH lkp-tests] stats/sysbench-mariadb: fix rubocop offenses
by Zhang Yale 19 Nov '20

19 Nov '20

[errors] Inspecting 1 file C offenses: sysbench-mariadb:4:1: C: Layout/IndentationWidth: Use 2 (not 4) spaces for indentation. case line.chomp! ^^^^ 1 file inspected, 1 offense detected Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/sysbench-mariadb | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/stats/sysbench-mariadb b/stats/sysbench-mariadb index 0c7ba1d9..0bed3bc0 100755 --- a/stats/sysbench-mariadb +++ b/stats/sysbench-mariadb @@ -1,10 +1,10 @@ #!/usr/bin/env ruby while (line = STDIN.gets) - case line.chomp! - when /transactions:\s+(\d+)/ - puts "transactions: #{$1}" - when /95th percentile:\s+(\S+)/ - puts "95th percentile: #{$1}" - end + case line.chomp! + when /transactions:\s+(\d+)/ + puts "transactions: #{$1}" + when /95th percentile:\s+(\S+)/ + puts "95th percentile: #{$1}" + end end -- 2.23.0

3 5

[PATCH lkp-tests] add the case that on_fail is nil
by Hu Xuejiao 19 Nov '20

19 Nov '20

[why] When sh_on_fail is nil, on_fail will add to the job.yaml Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/sbin/submit b/sbin/submit index a325e912..9bba1018 100755 --- a/sbin/submit +++ b/sbin/submit @@ -121,11 +121,14 @@ ARGV.each do |jobfile| 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files, - 'on_fail' => sh_on_fail + 'define_files' => sh_define_files } } + if sh_on_fail + sh_hash['job2sh']['on_fail'] = sh_on_fail + end + # merge job info job_hash = job.to_hash job_hash_list << job_hash -- 2.23.0

2 1

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: disable default dynamic mapping
by Zhang Yuhang 19 Nov '20

19 Nov '20

[why] By default, fields can be added dynamically to a document, or to inner objects within a document, just by indexing a document containing the new field. However, there are many fields that we do not need, and "Limit of total fields [1000] in index [jobs] has been exceeded" exception will be reported when the number of fields exceeds 1000. [how] Disable default dynamic mapping. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 9d420f7..076bdd3 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -38,6 +38,7 @@ else curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ "mappings": { "_doc": { + "dynamic": false, "dynamic_templates": [ { "pp": { @@ -61,6 +62,10 @@ else } ], "properties": { + "pp": { + "dynamic": true, + "properties": {} + }, "suite": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/sysbench-mariadb: fix rubocop offenses
by Zhang Yale 19 Nov '20

19 Nov '20

[errors] Inspecting 1 file C offenses: sysbench-mariadb:4:1: C: Layout/IndentationWidth: Use 2 (not 4) spaces for indentation. case line.chomp! ^^^^ 1 file inspected, 1 offense detected Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/sysbench-mariadb | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/stats/sysbench-mariadb b/stats/sysbench-mariadb index 0c7ba1d9..54a31be0 100755 --- a/stats/sysbench-mariadb +++ b/stats/sysbench-mariadb @@ -1,10 +1,10 @@ #!/usr/bin/env ruby while (line = STDIN.gets) - case line.chomp! - when /transactions:\s+(\d+)/ - puts "transactions: #{$1}" - when /95th percentile:\s+(\S+)/ - puts "95th percentile: #{$1}" - end + case line + when /transactions:\s+(\d+)/ + puts "transactions: #{$1}" + when /95th percentile:\s+(\S+)/ + puts "95th percentile: #{$1}" + end end -- 2.23.0

1 0

[PATCH v8 compass-ci] make the container/initrd-cifs to share /srv/initrd
by Xiao Shenwei 19 Nov '20

19 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result in order to reduce the coupling, need split them [how] one folder mount by one container [usage] two ways for the client to mount initrd: 1. initrd mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage: for lkp compatibility mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 22 ++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 89 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..c2ec808 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +RUN adduser -u 1090 -D lkp +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash +RUN rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..4b93647 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..77558a1 --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

2 2

[PATCH compass-ci] sched: fix cast from Nil to Array failed
by Cao Xueliang 19 Nov '20

19 Nov '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/jobfile_operate.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 16a4d06..98bf011 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -85,6 +85,7 @@ module Jobfile::Operate job_sh_array = [] of JSON::Any tmp_job_sh_content.as_h.each do |_key, val| + next if val == nil job_sh_array += val.as_a end else -- 2.23.0

1 0

[PATCH lkp-tests] tests/cci-makepkg: simplify centos7 test
by Lin Jiaxin 19 Nov '20

19 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 905ad689..8b72f395 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -90,20 +90,21 @@ update_shared_pkg() centos_update_gcc() { - if grep -sqF "CentOS Linux release 7" /etc/centos-release; then - [ "$os_mount" = "initramfs" ] && { - yum install -y centos-release-scl-rh - yum install -y "devtoolset-8-gcc*" - } - source /opt/rh/devtoolset-8/enable - fi + [ "$os" = "centos" ] || return + grep -sqF "CentOS Linux release 7" /etc/centos-release || return + + [ "$os_mount" = "initramfs" ] && { + yum install -y centos-release-scl-rh + yum install -y "devtoolset-8-gcc*" + } + source /opt/rh/devtoolset-8/enable } [ -n "$PKGBUILD_TAG" ] && download_pkgfile distro_install_depends lkp-dev -[ "$os" = "centos" ] && centos_update_gcc +centos_update_gcc distro_install_depends ${benchmark}-dev -- 2.23.0

3 2

[PATCH lkp-tests 6/6] iso2rootfs: allow the use of job.yaml and test script from the internet
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] Not all the yaml and script of test job will be committed to the lkp-tests repo. [How] Support three optional fields for every test: - testX_git_url: the url of git repo which contains yaml and script; - testX_git_yaml: the relative path of the yaml file in the git repo; - testX_git_script: the relative path of the test script in the git repo; Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 5 +++++ tests/iso2rootfs | 25 +++++++++++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 3c69a9d98430..c9a31cd0ad00 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -22,3 +22,8 @@ iso2rootfs: test1_yaml: test1_os_mount: test1_testbox: + ## follow three fields is required when your job.yaml and script for + ## this test are from the internet. + test1_git_url: + test1_git_yaml: + test1_git_script: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index a38685118b7f..1ba2e8a31ef8 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -447,10 +447,35 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } +prepare_submit() +{ + local submit_dir="/tmp/iso2rootfs/submit_tmp" + [ -d "${submit_dir}" ] || mkdir -p "${submit_dir}" + local submit_git_dir="${submit_dir}/${test_git_url##*/}" + submit_git_dir="${submit_git_dir%.git}" + + [ -d "${submit_git_dir}" ] || { + cd "${submit_dir}" + git clone "${test_git_url}" + } + + local test_git_yaml="${submit_git_dir}/$(eval echo "\$${test_num}_git_yaml")" + local test_git_script="${submit_git_dir}/$(eval echo "\$${test_num}_git_script")" + + [ -f "${test_git_yaml}" ] || die "cannot find file: ${test_num}_git_yaml." + cp "${test_git_yaml}" "${LKP_SRC}/jobs/" + + [ -f "${test_git_script}" ] || return 0 + cp "${test_git_script}" "${LKP_SRC}/tests/" +} + submit_one_yaml() { log_info "starting submit ${test_yaml} ..." + local test_git_url="$(eval echo "\$${test_num}_git_url")" + [ -z "${test_git_url}" ] || prepare_submit + cd "${LKP_SRC}/jobs" [ -f "${test_yaml}" ] || die "cannot find test yaml in LKP_SRC/jobs. -- 2.23.0

3 6

[PATCH lkp-tests 5/6] iso2rootfs: allow submit multiple test job.yaml for result rootfs
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] Result rootfs of dailybuild requires multiple test jobs. [How] 1. jobs/iso2rootfs.yaml. Before: test_yaml, test_os_mount, test_testbox After: test1_yaml,test1_os_mount, test1_testbox test2_yaml,test2_os_mount, test2_testbox ... 2. tests/iso2rootfs. - Use for-loop to traverse if $testX_yaml exists. - [ -n "${test1_yaml}" ] || return 0. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 8 ++++--- tests/iso2rootfs | 54 ++++++++++++++++++++++++++++++-------------- 2 files changed, 42 insertions(+), 20 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index fc3fc1d2e3ba..3c69a9d98430 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -17,6 +17,8 @@ iso2rootfs: config_rootfs: # submit test yaml related fields - test_yaml: - test_os_mount: - test_testbox: + ## you can add as many test jobs as you like. + ## but follow three fields is requied for every test job. + test1_yaml: + test1_os_mount: + test1_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index e9ed955441b3..a38685118b7f 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -56,7 +56,7 @@ check_yaml_vars() for yaml_t in "${yaml_vars[@]}" do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." + [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot find value of var: ${yaml_t}." done } @@ -447,17 +447,35 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } -submit_test_job_yaml() +submit_one_yaml() { - [ -n ${test_yaml} ] || return 0 - log_info "starting submit ${test_yaml} ..." - [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + cd "${LKP_SRC}/jobs" + [ -f "${test_yaml}" ] || + die "cannot find test yaml in LKP_SRC/jobs. + test yaml: ${test_yaml}." + + local test_os_mount="$(eval echo "\$${test_num}_os_mount")" + local test_testbox="$(eval echo "\$${test_num}_testbox")" + + [ -n "${test_os_mount}" ] || die "cannot find value of var: ${test_num}_os_mount." + [ -n "${test_testbox}" ] || die "cannot find value of var: ${test_num}_testbox." - [ -n "${test_os_mount}" ] || die "cannot fine value of var: test_os_mount." - [ -n "${test_testbox}" ] || die "cannot fine value of var: test_testbox." + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" + log_info "submit ${test_yaml} finished" +} + +submit_yamls() +{ + [ -n "$test1_yaml" ] || return 0 + + log_info "starting submit test yamls ..." + + [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh ./sbin/install-dependencies.sh @@ -465,17 +483,19 @@ submit_test_job_yaml() generate_global_yaml } - cd "${LKP_SRC}/jobs" && { - [ -f "${test_yaml}" ] || - die "cannot find test yaml in LKP_SRC/jobs. - test yaml: ${test_yaml}." + local test_yaml_index=1 + local test_num="test${test_yaml_index}" + local test_yaml="$(eval echo "\$${test_num}_yaml")" - "${LKP_SRC}/sbin/submit" \ - $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ - "${test_yaml}" - } + while [ -n "${test_yaml}" ] + do + submit_one_yaml + test_yaml_index=$((${test_yaml_index} +1)) + test_num="test${test_yaml_index}" + test_yaml="$(eval echo "\$${test_num}_yaml")" + done - log_info "submit ${test_yaml} finished" + log_info "submit test yamls finished" } ############ post works ############ @@ -511,7 +531,7 @@ main() config_rootfs test_rootfs - submit_test_job_yaml + submit_yamls post_works } -- 2.23.0

3 4

[PATCH lkp-tests 4/6] iso2rootfs: delete the useless fields
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] The 'os', 'os_arch', 'os_version' fields to be use by test job are the same as the 'iso_os', 'iso_arch', 'iso_version' of the destination iso this time. So delete the 'test_os', 'test_os_arch', 'test_os_version'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 3 --- tests/iso2rootfs | 33 +++++++-------------------------- 2 files changed, 7 insertions(+), 29 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index c81d548b49ea..fc3fc1d2e3ba 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -18,8 +18,5 @@ iso2rootfs: # submit test yaml related fields test_yaml: - test_os: - test_os_arch: - test_os_version: test_os_mount: test_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 34c8e84d9d74..e9ed955441b3 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -443,17 +443,8 @@ generate_global_yaml() generate_submit_append_str() { - local wait_add_vars=( - "os" - "os_arch" - "os_version" - "os_mount" - "testbox" - ) - for vt in "${wait_add_vars[@]}" - do - SUBMIT_APPEND_STR="${SUBMIT_APPEND_STR} ${vt}=$(eval echo "\$test_${vt}")" - done + [ "$#" -eq 2 ] || die "cannot get enough submit params." + echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } submit_test_job_yaml() @@ -464,18 +455,8 @@ submit_test_job_yaml() [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." - local test_yaml_vars=( - "test_os" - "test_os_arch" - "test_os_version" - "test_os_mount" - "test_testbox" - ) - - for yaml_t in "${test_yaml_vars[@]}" - do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." - done + [ -n "${test_os_mount}" ] || die "cannot fine value of var: test_os_mount." + [ -n "${test_testbox}" ] || die "cannot fine value of var: test_testbox." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh @@ -489,9 +470,9 @@ submit_test_job_yaml() die "cannot find test yaml in LKP_SRC/jobs. test yaml: ${test_yaml}." - generate_submit_append_str - - "${LKP_SRC}/sbin/submit" ${SUBMIT_APPEND_STR} "${test_yaml}" + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" } log_info "submit ${test_yaml} finished" -- 2.23.0

2 2

[PATCH lkp-tests 3/6] iso2rootfs: add a switch to configure rootfs
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] In some cases, we do not need to configure the result rootfs, so add a switch to configure rootfs. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 2 ++ tests/iso2rootfs | 1 + 2 files changed, 3 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 4e8729997a00..c81d548b49ea 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -13,6 +13,8 @@ iso2rootfs: # dailybuild related fields iso_url_file: + ## 0 - config; 1- don't config + config_rootfs: # submit test yaml related fields test_yaml: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index d42c38909f68..34c8e84d9d74 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -355,6 +355,7 @@ disable_selinux() config_rootfs() { + [ "${config_rootfs}" == "0" ] || return 0 config_dns_resolver disable_selinux } -- 2.23.0

3 4

[PATCH v7 compass-ci 1/2] lib: keep suite result in top for compare output
by Lu Kaiyi 19 Nov '20

19 Nov '20

[why] Currently, the compare output result default order by alphabet. If there are too many lines in result, it will be difficult for user to find the lines that starts with $suite in metric column. So, need reorder compare output result with keeping $suite item on the top of the table. [how] Divide the lines into two parts before print according to whether the line start with $suite or not, then combine output result into a whole to highlight the important info. [example] lukaiyi@z9 ~% compare "id=z9.148096" "id=z9.148143" in z9 suites_list:["sysbench-mysql"] [before] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 ... 1266042.00 -18.3% 1034206.00 softirqs.RCU 547849.00 -27.4% 397593.00 softirqs.SCHED 159990.00 -80.5% 31181.00 softirqs.TASKLET 1617967.00 -23.0% 1245331.00 softirqs.TIMER 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 2149.23 -100.0% 0.00 time.elapsed_time 2149.23 -100.0% 0.00 time.elapsed_time.max 214458.00 -100.0% 0.00 time.involuntary_context_switches 2.00 -100.0% 0.00 time.major_page_faults 192832.00 -100.0% 0.00 time.maximum_resident_set_size ... [after] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 2894304.00 +12.8% 3265730.00 interrupts.4:GICv3.26.Level.arch_timer 11546998.00 -100.0% 0.00 interrupts.746:ITS-MSI.99092481.Edge.hns3-0000:bd:00.1-TxRx-0 7476064.00 -100.0% 0.00 interrupts.747:ITS-MSI.99092482.Edge.hns3-0000:bd:00.1-TxRx-2 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 16 ++++++++++------ lib/compare_matrixes.rb | 41 ++++++++++++++++++++++++++++------------- lib/matrix2.rb | 20 +++++++++++++++----- 3 files changed, 53 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..f34e239 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suite_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suite_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,16 @@ end def create_matrices_list(conditions) matrices_list = [] + suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suite_list.concat(suites) end - matrices_list + + return matrices_list, suite_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..5039b01 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -128,7 +128,7 @@ def set_compare_values(index, values, field, success, options) end def get_values_by_field(matrixes_list, field, matrixes_size, success, options) - # get values by field, values struce example: values[0][:average] + # get values by field, values struct example: values[0][:average] # values = {} matrixes_list.length.times do |index| @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suite_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -807,26 +808,40 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suite_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suite_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suite_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suite_list, matrixes_titles, success, theme) return '' if values.empty? + suite_set = Set.new(suite_list) result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + ranked_str = get_ranked_str(values, suite_set, success, theme) + result_str += ranked_str + result_str +end + +def get_ranked_str(values, suite_set, success, theme) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suite_set.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end - result_str + suite_str + common_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..62d174c 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,23 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end - groups.delete(group_key) if value.size < 2 + if value.size < 2 + groups.delete(group_key) + next + end + suites_list << suite_list end - groups + + return groups, suites_list end # input: @@ -139,7 +149,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], = create_matrix(jobs) end end -- 2.23.0

3 2

[PATCH v2 lkp-tests 1/2] stats/iozone: fix iozone cannot match the line of part test
by Lu Kaiyi 19 Nov '20

19 Nov '20

[why] iozone can match the line like: 64 4 1599680 3057153 4274062 4897948 3588436 2892445 2772930 3057153 3958892 3057153 3022727 3363612 4564786 but cannot match the line as below(there are extra spaces after '2892445'): 64 4 1599680 3057153 4274062 4897948 3588436 2892445 [how] Add line.rstrip! before match Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/iozone | 1 + 1 file changed, 1 insertion(+) diff --git a/stats/iozone b/stats/iozone index 9f7ee72f..ff389c48 100755 --- a/stats/iozone +++ b/stats/iozone @@ -8,6 +8,7 @@ per_record = {} all_sum = 0 nr_records = 0 while (line = STDIN.gets) + line.rstrip! next unless line =~ /^\s*\d+.*\d+$/ data = line.split -- 2.23.0

3 3

[PATCH v2 compass-ci] lib/compare_data_format: optimize the format for echart
by Lu Weitao 19 Nov '20

19 Nov '20

[Why] With template compare mode, the result(data_set) isn't friendly for echart like: { "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "bs|test_size", "tables": { "1G|64k": { "fio.read_iops": { "average": { "openeuler 20.03": 14010.34254665909 }, "standard_deviation": { "openeuler 20.03": null }, "change": { } }, "fio.write_iops": { "average": { "openeuler 20.03": 13870.135498022726 }, "standard_deviation": { "openeuler 20.03": null }, "change": { } } }, "1G|4k": {...} } } [How] Make the result more friendly for echart like: { "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "test_size|bs", "tables": { "fio.read_iops": { "average": { "dimensions": [ "compare_dimension", "openeuler 20.03" ], "source": [ ["1G|4k", "1G|16k", "1G|32k", "1G|64k", "1G|128k", "1G|256k", "1G|512k", "1G|1024k", "80G|4k"], ["openeuler 20.03", 144076.2903315, 37865.30472368628, 21145.10375497826, 14010.34254665909, 6701.240849466667, 3205.077255, 1367.476930860465, 673.3270888666667, 11601.099817 ] ] }, "standard_deviation": {...}, "change": {...} }, "fio.write_iops": {...} ... } } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_data_format.rb | 77 ++++++++++++++++++++++++++++++++++---- lib/compare_matrixes.rb | 3 +- 2 files changed, 71 insertions(+), 9 deletions(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index 3d82550..b4f26ee 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -5,14 +5,75 @@ # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # +class FormatEchartData + def initialize(metrics_compare_results, template_params) + @metrics_compare_results = metrics_compare_results + @template_params = template_params + @data_set = {} + end -def format_for_echart(metrics_compare_results, template_params) - echart_result = {} - echart_result['title'] = template_params['title'] - echart_result['unit'] = template_params['unit'] - x_params = template_params['x_params'] - echart_result['x_name'] = x_params.join('|') if x_params - echart_result['tables'] = metrics_compare_results + def format_for_echart + echart_result = {} + echart_result['title'] = @template_params['title'] + echart_result['unit'] = @template_params['unit'] + @x_name = @template_params['x_params'] + echart_result['x_name'] = @x_name.join('|') if @x_name + echart_result['tables'] = convert_to_echart_dataset - echart_result + echart_result + end + + def convert_to_echart_dataset + @x_params = sort_x_params((a)metrics_compare_results.keys) + @metrics_compare_results.each_value do |metrics_values| + metrics_values.each do |metric, metric_value| + assign_echart_data_set(metric, metric_value) + end + end + + @data_set + end + + def assign_echart_data_set(metric, metric_value) + @data_set[metric] = {} + metric_value.each do |value_type, values| # value_type can be: average, standard_deviation, change + @data_set[metric][value_type] = { + 'dimensions' => ['compare_dimension'] + } + + dimension_list = values.keys.sort + @data_set[metric][value_type]['dimensions'] += dimension_list + @data_set[metric][value_type]['source'] = assign_echart_source(metric, value_type, dimension_list) + end + end + + def assign_echart_source(metric, value_type, dimensions) + source = [] + source << @x_params + dimensions.each do |dimension| + dimension_values = [dimension] + @x_params.each do |x_param| + if @metrics_compare_results[x_param][metric] + dimension_values << @metrics_compare_results[x_param][metric][value_type][dimension] + end + end + source << dimension_values + end + + source + end +end + +# input: x_params_list +# eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] +# output: +# ["1G|4K", "1G|128k", "1G|1024k", "2G|4k"] +def sort_x_params(x_params_list) + x_params_hash = {} + x_params_list.each do |param| + params = param.gsub(/[a-zA-Z]+$/, '0').split('|').map(&:to_i) + x_params_hash[params] = param + end + + x_params_hash.sort.map { |h| h[1] } end diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..762c087 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -498,7 +498,8 @@ def get_dimentions_combination(dimension_list) end def show_compare_result(metrics_compare_results, template_params) - echart_results = format_for_echart(metrics_compare_results, template_params) + formatter = FormatEchartData.new(metrics_compare_results, template_params) + echart_results = formatter.format_for_echart print JSON.pretty_generate(echart_results) end -- 2.23.0

1 0

[PATCH lkp-tests] create a multi-level directory
by Hu Xuejiao 19 Nov '20

19 Nov '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..6f603abe 100755 --- a/sbin/submit +++ b/sbin/submit @@ -9,6 +9,7 @@ require "#{LKP_SRC}/lib/distro_info" require "#{LKP_SRC}/lib/scheduler_client" require 'optparse' require 'yaml' +require 'fileutils' opt_set_key_value = {} opt_cmdline_defaults = {} @@ -35,7 +36,7 @@ options = OptionParser.new do |opts| exit 1 end opt_output_dir = dir - Dir.mkdir(dir) unless File.directory? dir + FileUtils.mkdir_p(dir) unless File.directory? dir end opts.on('-a', '--auto-define-files', 'auto add define_files') do -- 2.23.0

5 4

[PATCH lkp-tests 1/6] iso2rootfs: change how to get the kernel and initrd.lkp of rootfs
by Yu Chuan 19 Nov '20

19 Nov '20

[Why] We use the container in compass-ci repo to generate rootfs from a qcow2, it will generate the follow two soft links: - {rootfs_dir}/boot/vmlinuz # for vmlinuz file - {rootfs_dir}/initrd.lkp # for initrd file So we can use the below two soft links to get the absolute path of vmlinuz and initrd. [Addition] URL of qcow2rootfs: - https://gitee.com/wu_fengguang/compass-ci/tree/master/container/qcow2rootfs Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 05274164f8c1..9c71b8427d90 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -375,8 +375,12 @@ test_rootfs() log_info "starting test rootfs ..." local test_script="${CGZ_PATH}/kvm.sh" - local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz*|grep -v rescue) - local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/boot/initramfs.lkp*) + local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz) + local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/initrd.lkp) + + [ -f "$kernel_path" ] || die "cannot find kernel of rootfs: $kernel_path" + [ -f "$initrd_lkp_path" ] || die "cannot find initrd_lkp of rootfs: $initrd_lkp_path" + local root_path case ${rootfs_protocol} in "nfs") -- 2.23.0

1 0

[PATCH compass-ci] git_mirror: fix that too few repos in upstream-repos cause memory leak
by Li Yuanchao 18 Nov '20

18 Nov '20

[why] when there is only a few repo files in upstream-repos, that means the same repo will be called very frequently. As the main loop is too fast, it seems the new flow would be called before the last flow of the same repo end completely. That cause a memory problem. put a sleep in main loop would give the flow enough time to end completely before the same flow be called again Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 72ca328..5b6b889 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,6 +203,7 @@ class MirrorMain push_git_queue handle_feedback Signal.trap(:SIGCHLD, 'SIG_IGN') + sleep(0.1) end end end -- 2.23.0

3 3

[PATCH v2 lkp-tests] tests/build-pkg: add download PKGBUILD-$tag function
by Liu Shaofei 18 Nov '20

18 Nov '20

[why] Support more version's PKGBUILD to build different version software. [how] Download the PKGBUILD-$tag file through remote-git server from "/srv/git/pkg/" usage: 1) append upstream_tag field from build-pkg.yaml 2) create git repo that named by software name of $upstream_repo in "/srv/git/pkg/", and add PKGBUILD-$tag file to the directory of "/srv/git/pkg/$upstream_repo". if not PKGBUILD-$tag, the software is built with PKGBUILD. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 4aaa4fef..c5d55eda 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -7,7 +7,7 @@ # - upstream_repo # - upstream_url # - upstream_commit -# - upstream_tag +# - PKGBUILD_TAG ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -61,6 +61,15 @@ request_pkg() get_pkgfile "${pkgrepo}" "$pkgfile" done + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" + } + [ -f PKGBUILD ] || die "PKGBUILD not exist" [ -s PKGBUILD ] || die "PKGBUILD is empty" } @@ -72,9 +81,9 @@ create_softlink() local soft_path="$PKG_MNT/${pack_to}/${pkgname}" local bm_name=$(basename $(realpath ${cgz_name})) - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + if [ -n "$PKGBUILD_TAG" ]; then + echo "create soft link: ${PKGBUILD_TAG}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${PKGBUILD_TAG}.cgz" fi echo "create soft link: latest.cgz -> ${bm_name}" @@ -96,6 +105,9 @@ build_source_pkg() upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\")" sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD + [ -n "$PKGBUILD_TAG" ] && { + sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" + } } cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: umask 002 before download git trees
by Liu Yinsi 18 Nov '20

18 Nov '20

because ENV CCI_SRC LKP_SRC, REPO_SRC all under /c, some containers service start need umask 002, set it before download git trees. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/git | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 5588696..494107e 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -2,6 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# fix files permissions cause container service start failed +umask 002 + # download git trees cd /c || exit -- 2.23.0

2 2

[PATCH compass-ci] mail-robot: Dockerfile
by Luan Shengde 18 Nov '20

18 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v9 compass-ci 08/11] mail-robot: Dockerfile
by Luan Shengde 18 Nov '20

18 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..b3bceef --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport + +RUN umask 0000 -- 2.23.0

2 2

[PATCH v4 lkp-tests] tests/sysbench-threads-git: update command parameters
by Hu Xuejiao 18 Nov '20

18 Nov '20

[why] The current result: value of "total_time" is the same for different threads: nr_threads: 2 "total_time" is 0.3927 nr_threads: 4 "total_time" is 0.3927 ... nr_threads: 8192 "total_time" is 0.3927 [how] After updating --threads to --num-threads: nr_threads: 2 "total_time" is 0.3927 nr_threads: 4 "total_time" is 0.5643 ... nr_threads: 8192 "total_time" is 1.678 Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git index df1bb5cf..be83b441 100755 --- a/tests/sysbench-threads-git +++ b/tests/sysbench-threads-git @@ -10,7 +10,7 @@ : "${thread_locks:=2}" args=( - --threads=$nr_threads + --num-threads=$nr_threads --thread-yields=$thread_yields --thread-locks=$thread_locks ) -- 2.23.0

1 0

[PATCH v9 lkp-tests] add on_fail function when submit
by Hu Xuejiao 18 Nov '20

18 Nov '20

[why] When the task is failed(state), it need in the waiting state Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 6f603abe..4e80deef 100755 --- a/sbin/submit +++ b/sbin/submit @@ -116,12 +116,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_state(state='on_fail') sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

1 0

[PATCH v3 compass-ci 1/4] assign-account: Dockerfile add install ssh
by Luan Shengde 18 Nov '20

18 Nov '20

add install ssh for assign-account [why] need ssh-keygen to generate rsa public/private key for user Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/Dockerfile b/container/assign-account/Dockerfile index f84b5e6..12faafa 100644 --- a/container/assign-account/Dockerfile +++ b/container/assign-account/Dockerfile @@ -2,13 +2,13 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER luanshd0525(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ruby-sinatra ruby-json curl && \ + apt-get install -y ssh ruby-sinatra ruby-json curl && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone COPY assign-account.rb /usr/local/bin/ -- 2.23.0

2 2

[PATCH v6 compass-ci 10/12] mail-robot: Dockerfile
by Luan Shengde 18 Nov '20

18 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

2 2

[PATCH lkp-tests] stats/hackbench: fix rubocop offenses
by Zhang Yu 18 Nov '20

18 Nov '20

[errors] Inspecting 1 file offenses: hackbench:26:18: C: [Corrected] Style/RedundantSort: Use min instead of sort...first. best_time = time.sort.first ^^^^^^^^^^ Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/hackbench | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/hackbench b/stats/hackbench index 9cd43277..89b2be35 100755 --- a/stats/hackbench +++ b/stats/hackbench @@ -23,5 +23,5 @@ while (line = gets) end kb = tasks * messages * bytes >> 10 -best_time = time.sort.first +best_time = time.min puts "throughput: #{kb / best_time}" -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-jobs-mapping.sh: disable default dynamic mapping
by Zhang Yuhang 18 Nov '20

18 Nov '20

[why] By default, fields can be added dynamically to a document, or to inner objects within a document, just by indexing a document containing the new field. However, there are many fields that we do not need, and "Limit of total fields [1000] in index [jobs] has been exceeded" exception will be reported when the number of fields exceeds 1000. [how] Disable default dynamic mapping. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 9d420f7..076bdd3 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -38,6 +38,7 @@ else curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ "mappings": { "_doc": { + "dynamic": false, "dynamic_templates": [ { "pp": { @@ -61,6 +62,10 @@ else } ], "properties": { + "pp": { + "dynamic": true, + "properties": {} + }, "suite": { "type": "keyword" }, -- 2.23.0

2 2

[PATCH v9 compass-ci 11/11] mail-robot: documentation for service
by Luan Shengde 18 Nov '20

18 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 98 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..ec1a866 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,98 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## usage + +build email as the following section: allowed email format +send mail to: compass-ci(a)qq.com + +## allowed email format: + +subject: apply account +mail content + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. apply-account + init my_info + - my_email + - my_name + check_to_send_account + - parse_mail_content + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there has an attachment file to the email file + attachment: + first attachment + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + - no more available account + build_error_email with raised error message + + - send_mail + - build success email + - build failed email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

1 0

[PATCH v9 compass-ci 10/11] mail-robot: start file
by Luan Shengde 18 Nov '20

18 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..5827e18 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +defaults['MAILDIR'] = '/srv/cci/Maildir/.compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v9 compass-ci 09/11] mail-robot: build file
by Luan Shengde 18 Nov '20

18 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v9 compass-ci 07/11] mail-robot: build fail message
by Luan Shengde 18 Nov '20

18 Nov '20

build fail email message with raised error message - lack commit url - repo url for the commit url not in upstream - commit not exist or not available - lack pubkey - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 container/mail-robot/lib/assign-account-fail-eamil.rb diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..593f345 --- /dev/null +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following error: + + #{message} + + In order to successfully apply an account, please pay attention to the following points: + + 1. mail subject + The subject should exactly: apply account + + 2. commit url + When you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + Ensure you commit url exist and available to access. + + 3. ssh pubkey + You need to add a pubkey as an attachment to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v9 compass-ci 06/11] mail-robot: build success email
by Luan Shengde 18 Nov '20

18 Nov '20

build email message for successfully assigning account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/assign-account-email.rb | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 container/mail-robot/lib/assign-account-email.rb diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb new file mode 100755 index 0000000..b793232 --- /dev/null +++ b/container/mail-robot/lib/assign-account-email.rb @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_email(my_info) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{my_info['my_name']}, + + Thank you for joining us. + + You can use the following info to submit jobs: + + notice: + the followings are ONE-TIME setup + + 1) setup default config + run the following command to add the below setup to default config file + + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF + my_email: #{my_info['my_email']} + my_name: #{my_info['my_name']} + my_uuid: #{my_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + + reference to 'how to write job yaml' section to write the job yaml + you can also reference to files in lkp-tests/jobs as example. + + submit jobs, for example: + + submit -m iperf.yaml testbox=vm-2p8g queue=vm-2p8g~\\$USER + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v9 compass-ci 05/11] mail-robot: apply new account for user
by Luan Shengde 18 Nov '20

18 Nov '20

ApplyJumperAccount used to apply account for user input: my_info output: account_info apply_jumper_account apply account with my_info my_info: - my_email - my_name - my_uuid - my_ssh_pubkey account_info: - my_login_name - my_password - jumper_host - jumper_port - my_jumper_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/apply-jumper-account.rb | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100755 container/mail-robot/lib/apply-jumper-account.rb diff --git a/container/mail-robot/lib/apply-jumper-account.rb b/container/mail-robot/lib/apply-jumper-account.rb new file mode 100755 index 0000000..44f412b --- /dev/null +++ b/container/mail-robot/lib/apply-jumper-account.rb @@ -0,0 +1,42 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 + +# used to apply account +# be called after AssignAccount successfully parsed my_commit_url and my_ssh_pubkey +# apply_jumper_account +# apply jumper account with my_info and my_ssh_pubkey +# account_info_exist +# check account exists +class ApplyJumperAccount + def initialize(my_info) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + end + + def apply_jumper_account + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = 'No more available jumper account.' + error_message += 'You may try again later or consulting the manager for a solution.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v9 compass-ci 04/11] mail-robot: check commit url for gitee.com
by Luan Shengde 18 Nov '20

18 Nov '20

GiteeCommitUrlCheck used to check commit for gitee.com input: my_commit_url gitee_commit_check clone the repo and check commit exists and availalbe Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 container/mail-robot/lib/gitee-commit-url-check.rb diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb new file mode 100755 index 0000000..72c948f --- /dev/null +++ b/container/mail-robot/lib/gitee-commit-url-check.rb @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# used to check commit available for hub gitee.com +# it is called when: +# - ParseApplyAccountEmail execute commit_url_availability +# - hub is gitee.com +# gitee_commit_check +# clone the repo +# check commit available +class GiteeCommitUrlCheck + def initialize(my_info, url, base_url) + @my_info = my_info + @url = url + @base_url = base_url + end + + def gitee_commit_check + repo_dir = @url.split('/')[-3] + repo_url = [@base_url, 'git'].join('.') + commit_id = @url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] + + FileUtils.rm_rf repo_dir + + gitee_commit_exist(email_index) + end + + def gitee_commit_exist(email_index) + return if email_index + + error_message = "We can not confirm whether the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with your email.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v9 compass-ci 03/11] mail-robot: parse apply account email
by Luan Shengde 18 Nov '20

18 Nov '20

ParseApplyAccountEmail used to parse my_email, my_name, my_commit_url and my_ssh_pubkey input: mail_content output: my_info - my_email - my_name - my_commit_url - my_ssh_pubkey init my_info parse_commit_url extract_commit_url extract my_commit_url from mail_content base_url_in_upstream_repos check whether the repo's url in upstream-repos commit_url_availability check the commit available - gitee.com call GiteeCommitUrlCheck to check commit available - non-gitee.com execute curl and check feedback for the commit parse_pub_key check attachment for my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 138 ++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100755 container/mail-robot/lib/parse-apply-account-email.rb diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb new file mode 100755 index 0000000..8c55d8f --- /dev/null +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -0,0 +1,138 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'gitee-commit-url-check' + +# used to parse mail_content for my_commit_url and my_ssh_pubkey +# be called by AssignAccount when it needs to extract required data: +# - my_commit_url +# - my_ssh_pubkey +# +# input: mail_content +# output: my_commit_url, my_ssh_pubkey +# +# parse_commit_url +# parse my_commit_url from the mail_content and return it +# extract my_commit_url +# extract my_coomit_url from the mail_content +# check commit url exists +# base_url_in_upstream_repos +# check whether the base url in upstream-repos +# commit_url_availability +# check commit available +# gitee.com: +# call GiteeCommitUrlCheck to check the commit +# non-gitee.com: +# execute curl to check the commit +# parse_pub_key +# check my_ssh_pubkey exists and return it +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_ssh_pubkey' => [] + } + end + + def build_my_info + @my_info['my_commit_url'] = parse_commit_url + @my_info['my_ssh_pubkey'] << parse_pub_key + + return @my_info + end + + def extract_mail_content_line + mail_content_body = @mail_content.part[0].part[0].body.decoded || \ + @mail_content.part[0].body.decoded || \ + @mail_content.body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + + return mail_content_line + end + + def extract_commit_url + mail_content_line = extract_mail_content_line + # the commit url should be headed with a prefix: my oss commit + # the commit url should be in a standart format, example: + # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + error_message = "No matched commit url found.\n" + error_message += 'Ensure that you have add a right commit url, ' + error_message += "and with prefix 'my oss commit:'." + raise error_message + end + + mail_content_line.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def parse_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + base_url_in_upstream_repos('/c/upstream-repos', base_url) + commit_url_availability(url, base_url) + + return url + end + + def base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = "The repo url for your commit is not in our upstream-repo list.\n" + error_message += 'Use a new one, or consulting the manager for available repo list.' + + raise error_message + end + + def commit_url_availability(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit information + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + gitee_commit(url, base_url) + else + non_gitee_commit(url) + end + end + + def gitee_commit(url, base_url) + my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) + my_gitee_commit.gitee_commit_check + end + + def non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_info['my_email'] + + return unless email_index.nil? + + error_message = "We can not confirm the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end + + def parse_pub_key + error_message = "No pub_key found.\n" + error_message += 'Please add a pub_key as an attachment to your email.' + + raise error_message if @mail_content.attachments.empty? + raise error_message unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ + + pub_key = @mail_content.attachments[0].body.decoded + + return pub_key + end +end -- 2.23.0

1 0

[PATCH v9 compass-ci 02/11] mail-robot: check to apply account
by Luan Shengde 18 Nov '20

18 Nov '20

ApplyAccount check to apply account for user when listened new email with subject 'apply account' input: mail_content output: send mail to user check_to_send_account parse_mail_content update my_info apply_my_account generate uuid apply account with my_info my_info: - my_email - my_name - my_uuid - my_ssh_pubkey store account info to es send email - send success email - send fail email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 110 ++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100755 container/mail-robot/lib/apply-account.rb diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb new file mode 100755 index 0000000..bf40429 --- /dev/null +++ b/container/mail-robot/lib/apply-account.rb @@ -0,0 +1,110 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '../../../lib/es_client' +require_relative 'assign-account-email' +require_relative 'assign-account-fail-eamil' +require_relative 'apply-jumper-account' +require_relative 'parse-apply-account-email' + +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +# assign uuid/account for user +# when mail-robot listened new email, and the email's subject +# exactly equal 'apply account', mail-robot will call this class +# entry point: send_account +# input: email's content +# +# send_account +# parse_mail_content +# call ParseApplyAccountEmail to parse: +# - my_commit_url +# - my_ssh_pubkey +# apply_my_account +# call ApplyJumperAccount to apply new account +# required data: +# my_info: +# - my_email +# - my_name +# - my_uuid +# my_ssh_pubkey +# store_account_info +# call ESClient to store my_info +# my_info: +# - my_email +# - my_name +# - my_uuid +# - my_commit_url +# - my_login_name +# - my_ssh_pubkey +# send_mail +# when successfully applied an account +# call build_apply_account_email to send a successful email +# when rescued error message +# call build_apply_account_fail_email to send fail email +class ApplyAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = {} + end + + def check_to_send_account + # in case of failed parsing, parse_mail_content will return none info + # in order to successfully send email for failed parsing + # firstly get my_email before execute parse_mail_content is needed + @my_info['my_email'] = @mail_content.from[0] + parse_mail_content + acct_info = apply_my_account + + @my_info['my_login_name'] = acct_info['my_login_name'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_mail_content + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + parsed_email_info = parse_apply_account_email.build_my_info + + @my_info.update parsed_email_info + end + + def apply_my_account + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(error_message) + email_message = if error_message.empty? + build_apply_account_email(@my_info) + else + build_apply_account_fail_email(@my_info, error_message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

1 0

[PATCH v9 compass-ci 01/11] mail-robot: listen and handle new email
by Luan Shengde 18 Nov '20

18 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/mail-robot.rb | 41 ++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..a809eae --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'lib/apply-account' + +MAILDIR = ENV['MAILDIR'] + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = ApplyAccount.new(mail_content) + assign_uuid.check_to_send_account +end + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

1 0