- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/3] git_mirror: fix that README.md in upstream-repos cause a load error
by Li Yuanchao 20 Nov '20

20 Nov '20

[why] As README.md or other docs in upstream-repos, and they are not yaml file. But git-mirror load all files in upstream-repos as yaml file, so there comes out a load error. Now files whose dirname is 'xxx/upstream-repos' will not be loaded. [errmsg] in `load_repo_file': undefined method `[]=' for x:Integer (NoMethodError) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9648063..72ca328 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -135,6 +135,8 @@ class MirrorMain traverse_repodir("#{repodir}/#{entry}") end else + return if File.dirname(repodir) == REPO_DIR + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") fork_name = File.basename(repodir) load_repo_file(repodir, project, fork_name) -- 2.23.0

2 2

[PATCH lkp-tests] daemon/multi-qemu: add a default value for the queues field
by Zhang Yu 20 Nov '20

20 Nov '20

[why] For multi-qemu.yaml, if the queues field is not set, a default value is required. [how] If the multi-qemu.yaml not set queues field, there are two queues by default: "$tbox_group.$HOSTNAME, $tbox_group.$(arch)". Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- daemon/multi-qemu | 1 + 1 file changed, 1 insertion(+) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index 8d9dd38d..7cc625f4 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -6,6 +6,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +: ${queues:="$tbox_group.$HOSTNAME, $tbox_group.$(arch)"} multi_qemu() { export CCI_SRC=/c/compass-ci -- 2.23.0

3 3

[PATCH v2 lkp-tests] lib: fix request state
by Ren Wen 20 Nov '20

20 Nov '20

When run *start_daemon*, lkp reports a "finished" state to scheduler if daemon started. When run *run_test*, lkp also reports a "finished" state if tests finished. Now i want to stop machines after all tests finished, so alter state "finished" to "started" when running *start_daemon*. This makes it easier to handle request state at scheduler. Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.sh b/lib/job.sh index 81523a98..b84e6b18 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -278,7 +278,7 @@ start_daemon() run_program daemon "$@" fi - sync_cluster_state 'finished' + sync_cluster_state 'started' # If failed to start the daemon above, the job will abort. # LKP server on notice of the failed job will abort the other waiting nodes. -- 2.23.0

1 0

[PATCH lkp-tests] lib: fix request state
by Ren Wen 20 Nov '20

20 Nov '20

When run *start_daemon*, lkp reports a "finished" state to scheduler if daemon started. When run *run_test*, lkp also reports a "finished" state if tests finished. Now i want to stop machines after all tests finishing, so alter state "finished" to "started" when run *start_daemon*. This makes it easier to handle request state at scheduler. Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.sh b/lib/job.sh index 81523a98..b84e6b18 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -278,7 +278,7 @@ start_daemon() run_program daemon "$@" fi - sync_cluster_state 'finished' + sync_cluster_state 'started' # If failed to start the daemon above, the job will abort. # LKP server on notice of the failed job will abort the other waiting nodes. -- 2.23.0

2 2

[PATCH compass-ci] container/logging-es: fix failed to build images in x86 machine
by Liu Yinsi 20 Nov '20

20 Nov '20

[why] when build logging-es images in x86 machine, error message: libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error" The command '/bin/sh -c sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture In addition, 'declare', arrays, arrays references are undefined in POSIX sh by shellcheck, so change '#!/bin/sh' to '#!/bin/bash'. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 3 ++- container/logging-es/build | 13 +++++++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 707299f..6c01c43 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -1,7 +1,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM gagara/elasticsearch-oss-arm64:7.6.2 +ARG BASE_IMAGE +FROM $BASE_IMAGE # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 diff --git a/container/logging-es/build b/container/logging-es/build index 0ddecbf..451041b 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,5 +1,14 @@ -#!/bin/sh +#!/bin/bash # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t logging-es:7.6.2 . +declare -A BASE_IMAGE_DICT + +BASE_IMAGE_DICT=( + ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" + ["x86_64"]="elasticsearch:7.6.2" +) + +BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} + +docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . -- 2.23.0

1 0

[PATCH compass-ci] container/minio: fix build permission denied
by Liu Yinsi 20 Nov '20

20 Nov '20

lys@crystal ~/compass-ci/container/minio% ./build zsh: permission denied: ./build Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/minio/build | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 container/minio/build diff --git a/container/minio/build b/container/minio/build old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH lkp-tests] stats: add new parsing parameters for performance test
by Bai Jing 20 Nov '20

20 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/lmbench3 | 62 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 2 deletions(-) diff --git a/stats/lmbench3 b/stats/lmbench3 index 211c3abc..750e8bfc 100755 --- a/stats/lmbench3 +++ b/stats/lmbench3 @@ -21,6 +21,27 @@ def largest_bandwidth bandwidth end +def designated_file_size + designated_hash = {} + result_str = '' + + $stdin.each_line do |line| + result_str += line + break if line =~ /(^\s|^\t)/ + end + + result_str.split("\n").each do |e| + designated_hash['L1_$'] = e.split[1] if e =~ /0.01758/ + designated_hash['Rand_mem'] = e.split[1] if e =~ /512.0000/ + designated_hash['Mmap_Latency'] = e.split[1] if e =~ /536.870912/ + designated_hash['L2_$'] = e.split[1] if e =~ /0.05078/ + end + + designated_hash +end + +null_io_array = [] + while (line = STDIN.gets) line = line.resolve_invalid_bytes case line @@ -32,14 +53,15 @@ while (line = STDIN.gets) # Simple fstat: 0.3517 microseconds # Simple open/close: 1.3696 microseconds when /^Simple (\S+): (\d+.\d+) microseconds$/ + null_io_array << $2.to_f if $1.to_s == 'read' || $1.to_s == 'write' puts "syscall.#{$1}.latency.us: #{$2}" # Extract select test result. # Select on 100 fd's: 1.2293 microseconds # Select on 100 tcp fd's: 5.0377 microseconds - when /^Select on 100 fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 fd's: (\d+.\d+) microseconds$/ puts "Select.100fd.latency.us: #{$1}" - when /^Select on 100 tcp fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 tcp fd's: (\d+.\d+) microseconds$/ puts "Select.100tcp.latency.us: #{$1}" # Extract proc test result. @@ -198,5 +220,41 @@ while (line = STDIN.gets) # ... when /^"Mmap read open2close bandwidth$/ puts "MMAP.read_open2close.bandwidth.MB/sec: #{largest_bandwidth}" + + # "Random Memory load latency" + # 416.00000 92.824 + # 448.00000 92.772 + # 512.00000 92.640 + # ... + when /^Random load latency$/ + result_rand_mem = designated_file_size + puts "Rand_mem: #{result_rand_mem['Rand_mem']}" + + # "Memory load latency" + # 0.00049 1.539 + # 0.00098 1.539 + # 512.00000 74.421 + # ... + when /^Memory load latency$/ + result_rand_mem = designated_file_size + puts "L1_$: #{result_rand_mem['L1_$']}" + puts "L2_$: #{result_rand_mem['L2_$']}" + + # "mappings" + # 67.108864 82 + # 134.217728 151 + # 268.435456 309 + # 536.870912 615 + # ... + when /^"mappings$/ + result_map = designated_file_size + puts "Mmap_Latency: #{result_map['Mmap_Latency']}" end end + +num = 0 +null_io_array.each do |v| + num += v +end +null_num = format('%.4f', num / 2).to_f +puts "null_io: #{null_num}" -- 2.23.0

3 4

[PATCH v3 lkp-tests 6/6] iso2rootfs: allow the use of job.yaml and test script from the internet
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] Not all the yaml and script of test job will be committed to the lkp-tests repo. [How] Support three optional fields for every test in job.yaml: - testX_git_url: the url of git repo which contains yaml and script; - testX_git_yaml: the relative path of the yaml file in the git repo; - testX_git_script: the relative path of the test script in the git repo; Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 5 +++++ tests/iso2rootfs | 26 ++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 1631aca6d23f..1a643b538ac1 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -26,3 +26,8 @@ iso2rootfs: test1_yaml: test1_os_mount: test1_testbox: + ## the following three fields is required when your job.yaml and + ## script for this test are from the internet. + test1_git_url: + test1_git_yaml: + test1_git_script: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 4135cdea7059..0353c045a4bc 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -457,10 +457,36 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } +prepare_submit() +{ + local submit_dir="/tmp/iso2rootfs/submit_tmp" + mkdir -p "${submit_dir}" + + local submit_git_dir="${submit_dir}/${test_git_url##*/}" + submit_git_dir="${submit_git_dir%.git}" + + [ -d "${submit_git_dir}" ] || { + cd "${submit_dir}" + git clone "${test_git_url}" + } + + local test_git_yaml="${submit_git_dir}/$(eval echo "\$${test_num}_git_yaml")" + local test_git_script="${submit_git_dir}/$(eval echo "\$${test_num}_git_script")" + + [ -f "${test_git_yaml}" ] || die "cannot find file: ${test_num}_git_yaml." + cp "${test_git_yaml}" "${LKP_SRC}/jobs/" + + [ -f "${test_git_script}" ] || die "cannot find file: ${test_num}_git_script." + cp "${test_git_script}" "${LKP_SRC}/tests/" +} + submit_one_yaml() { log_info "starting submit ${test_yaml} ..." + local test_git_url="$(eval echo "\$${test_num}_git_url")" + [ -z "${test_git_url}" ] || prepare_submit + cd "${LKP_SRC}/jobs" [ -f "${test_yaml}" ] || die "cannot find yaml in LKP_SRC/jobs: ${test_yaml}." -- 2.23.0

1 0

[PATCH v3 lkp-tests 5/6] iso2rootfs: allow submit multiple test job.yaml for result rootfs
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] Result rootfs of dailybuild requires multiple test jobs. [How] 1. jobs/iso2rootfs.yaml. Before: test_yaml, test_os_mount, test_testbox After: test1_yaml, test1_os_mount, test1_testbox test2_yaml, test2_os_mount, test2_testbox ... 2. tests/iso2rootfs. - Use while-loop to traverse if $testX_yaml exists. - [ -n "${test1_yaml}" ] || return 0. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 8 ++++--- tests/iso2rootfs | 50 ++++++++++++++++++++++++++++++-------------- 2 files changed, 39 insertions(+), 19 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 629b53febb2f..1631aca6d23f 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -21,6 +21,8 @@ iso2rootfs: dailybuild_iso_url_file: # submit test yaml related fields - test_yaml: - test_os_mount: - test_testbox: + ## 1. you can add as many jobs as you like. + ## 2. the following three fields is requied for every test job. + test1_yaml: + test1_os_mount: + test1_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 3015c6f62947..4135cdea7059 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -457,17 +457,33 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } -submit_test_job_yaml() +submit_one_yaml() { - [ -n ${test_yaml} ] || return 0 - log_info "starting submit ${test_yaml} ..." - [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + cd "${LKP_SRC}/jobs" + [ -f "${test_yaml}" ] || die "cannot find yaml in LKP_SRC/jobs: ${test_yaml}." + + local test_os_mount="$(eval echo "\$${test_num}_os_mount")" + local test_testbox="$(eval echo "\$${test_num}_testbox")" + + [ -n "${test_os_mount}" ] || die "cannot find value of var: ${test_num}_os_mount." + [ -n "${test_testbox}" ] || die "cannot find value of var: ${test_num}_testbox." - [ -n "${test_os_mount}" ] || die "cannot find value of var: test_os_mount." - [ -n "${test_testbox}" ] || die "cannot find value of var: test_testbox." + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" + log_info "submit ${test_yaml} finished" +} + +submit_yamls() +{ + [ -n "$test1_yaml" ] || return 0 + + log_info "starting submit test yamls ..." + + [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh ./sbin/install-dependencies.sh @@ -475,17 +491,19 @@ submit_test_job_yaml() generate_global_yaml } - cd "${LKP_SRC}/jobs" && { - [ -f "${test_yaml}" ] || - die "cannot find test yaml in LKP_SRC/jobs. - test yaml: ${test_yaml}." + local test_yaml_index=1 + local test_num="test${test_yaml_index}" + local test_yaml="$(eval echo "\$${test_num}_yaml")" - "${LKP_SRC}/sbin/submit" \ - $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ - "${test_yaml}" - } + while [ -n "${test_yaml}" ] + do + submit_one_yaml + test_yaml_index=$((${test_yaml_index} +1)) + test_num="test${test_yaml_index}" + test_yaml="$(eval echo "\$${test_num}_yaml")" + done - log_info "submit ${test_yaml} finished" + log_info "submit test yamls finished" } ############ post works ############ @@ -525,7 +543,7 @@ main() config_rootfs test_rootfs - submit_test_job_yaml + submit_yamls post_works } -- 2.23.0

1 0

[PATCH v3 lkp-tests 4/6] iso2rootfs: delete the useless fields
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] The 'os', 'os_arch', 'os_version' fields to be use by test job are the same as the 'iso_os', 'iso_arch', 'iso_version' of the destination iso this time. So delete the 'test_os', 'test_os_arch', 'test_os_version'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 3 --- tests/iso2rootfs | 35 ++++++++--------------------------- 2 files changed, 8 insertions(+), 30 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 1e1dfc862136..629b53febb2f 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -22,8 +22,5 @@ iso2rootfs: # submit test yaml related fields test_yaml: - test_os: - test_os_arch: - test_os_version: test_os_mount: test_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index bb2da3208646..3015c6f62947 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -56,7 +56,7 @@ check_yaml_vars() for yaml_t in "${yaml_vars[@]}" do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." + [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot find value of var: ${yaml_t}." done } @@ -453,17 +453,8 @@ generate_global_yaml() generate_submit_append_str() { - local wait_add_vars=( - "os" - "os_arch" - "os_version" - "os_mount" - "testbox" - ) - for vt in "${wait_add_vars[@]}" - do - SUBMIT_APPEND_STR="${SUBMIT_APPEND_STR} ${vt}=$(eval echo "\$test_${vt}")" - done + [ "$#" -eq 2 ] || die "generate_submit_append_str: please give 2 parameters." + echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } submit_test_job_yaml() @@ -474,18 +465,8 @@ submit_test_job_yaml() [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." - local test_yaml_vars=( - "test_os" - "test_os_arch" - "test_os_version" - "test_os_mount" - "test_testbox" - ) - - for yaml_t in "${test_yaml_vars[@]}" - do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." - done + [ -n "${test_os_mount}" ] || die "cannot find value of var: test_os_mount." + [ -n "${test_testbox}" ] || die "cannot find value of var: test_testbox." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh @@ -499,9 +480,9 @@ submit_test_job_yaml() die "cannot find test yaml in LKP_SRC/jobs. test yaml: ${test_yaml}." - generate_submit_append_str - - "${LKP_SRC}/sbin/submit" ${SUBMIT_APPEND_STR} "${test_yaml}" + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" } log_info "submit ${test_yaml} finished" -- 2.23.0

1 0

2025

2024

2023

2022

2021

2020

Compass-ci