- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 2/3] git_mirror: fix that too few repos in upstream-repos cause memory leak
by Li Yuanchao 20 Nov '20

20 Nov '20

[why] when there is only a few repo files in upstream-repos, that means the same repo will be called very frequently. As the main loop is too fast, it seems the new flow would be called before the last flow of the same repo end completely. That cause a problem. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 72ca328..5b6b889 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,6 +203,7 @@ class MirrorMain push_git_queue handle_feedback Signal.trap(:SIGCHLD, 'SIG_IGN') + sleep(0.1) end end end -- 2.23.0

2 2

[PATCH v2 compass-ci 1/3] git_mirror: fix that README.md in upstream-repos cause a load error
by Li Yuanchao 20 Nov '20

20 Nov '20

[why] As README.md or other docs in upstream-repos, and they are not yaml file. But git-mirror load all files in upstream-repos as yaml file, so there comes out a load error. Now files whose dirname is 'xxx/upstream-repos' will not be loaded. [errmsg] in `load_repo_file': undefined method `[]=' for x:Integer (NoMethodError) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9648063..72ca328 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -135,6 +135,8 @@ class MirrorMain traverse_repodir("#{repodir}/#{entry}") end else + return if File.dirname(repodir) == REPO_DIR + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") fork_name = File.basename(repodir) load_repo_file(repodir, project, fork_name) -- 2.23.0

2 2

[PATCH lkp-tests] daemon/multi-qemu: add a default value for the queues field
by Zhang Yu 20 Nov '20

20 Nov '20

[why] For multi-qemu.yaml, if the queues field is not set, a default value is required. [how] If the multi-qemu.yaml not set queues field, there are two queues by default: "$tbox_group.$HOSTNAME, $tbox_group.$(arch)". Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- daemon/multi-qemu | 1 + 1 file changed, 1 insertion(+) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index 8d9dd38d..7cc625f4 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -6,6 +6,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +: ${queues:="$tbox_group.$HOSTNAME, $tbox_group.$(arch)"} multi_qemu() { export CCI_SRC=/c/compass-ci -- 2.23.0

3 3

[PATCH v2 lkp-tests] lib: fix request state
by Ren Wen 20 Nov '20

20 Nov '20

When run *start_daemon*, lkp reports a "finished" state to scheduler if daemon started. When run *run_test*, lkp also reports a "finished" state if tests finished. Now i want to stop machines after all tests finished, so alter state "finished" to "started" when running *start_daemon*. This makes it easier to handle request state at scheduler. Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.sh b/lib/job.sh index 81523a98..b84e6b18 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -278,7 +278,7 @@ start_daemon() run_program daemon "$@" fi - sync_cluster_state 'finished' + sync_cluster_state 'started' # If failed to start the daemon above, the job will abort. # LKP server on notice of the failed job will abort the other waiting nodes. -- 2.23.0

1 0

[PATCH lkp-tests] lib: fix request state
by Ren Wen 20 Nov '20

20 Nov '20

When run *start_daemon*, lkp reports a "finished" state to scheduler if daemon started. When run *run_test*, lkp also reports a "finished" state if tests finished. Now i want to stop machines after all tests finishing, so alter state "finished" to "started" when run *start_daemon*. This makes it easier to handle request state at scheduler. Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.sh b/lib/job.sh index 81523a98..b84e6b18 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -278,7 +278,7 @@ start_daemon() run_program daemon "$@" fi - sync_cluster_state 'finished' + sync_cluster_state 'started' # If failed to start the daemon above, the job will abort. # LKP server on notice of the failed job will abort the other waiting nodes. -- 2.23.0

2 2

[PATCH compass-ci] container/logging-es: fix failed to build images in x86 machine
by Liu Yinsi 20 Nov '20

20 Nov '20

[why] when build logging-es images in x86 machine, error message: libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error" The command '/bin/sh -c sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture In addition, 'declare', arrays, arrays references are undefined in POSIX sh by shellcheck, so change '#!/bin/sh' to '#!/bin/bash'. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 3 ++- container/logging-es/build | 13 +++++++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 707299f..6c01c43 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -1,7 +1,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM gagara/elasticsearch-oss-arm64:7.6.2 +ARG BASE_IMAGE +FROM $BASE_IMAGE # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 diff --git a/container/logging-es/build b/container/logging-es/build index 0ddecbf..451041b 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,5 +1,14 @@ -#!/bin/sh +#!/bin/bash # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t logging-es:7.6.2 . +declare -A BASE_IMAGE_DICT + +BASE_IMAGE_DICT=( + ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" + ["x86_64"]="elasticsearch:7.6.2" +) + +BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} + +docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . -- 2.23.0

1 0

[PATCH compass-ci] container/minio: fix build permission denied
by Liu Yinsi 20 Nov '20

20 Nov '20

lys@crystal ~/compass-ci/container/minio% ./build zsh: permission denied: ./build Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/minio/build | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 container/minio/build diff --git a/container/minio/build b/container/minio/build old mode 100644 new mode 100755 -- 2.23.0

1 0

[PATCH lkp-tests] stats: add new parsing parameters for performance test
by Bai Jing 20 Nov '20

20 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/lmbench3 | 62 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 2 deletions(-) diff --git a/stats/lmbench3 b/stats/lmbench3 index 211c3abc..750e8bfc 100755 --- a/stats/lmbench3 +++ b/stats/lmbench3 @@ -21,6 +21,27 @@ def largest_bandwidth bandwidth end +def designated_file_size + designated_hash = {} + result_str = '' + + $stdin.each_line do |line| + result_str += line + break if line =~ /(^\s|^\t)/ + end + + result_str.split("\n").each do |e| + designated_hash['L1_$'] = e.split[1] if e =~ /0.01758/ + designated_hash['Rand_mem'] = e.split[1] if e =~ /512.0000/ + designated_hash['Mmap_Latency'] = e.split[1] if e =~ /536.870912/ + designated_hash['L2_$'] = e.split[1] if e =~ /0.05078/ + end + + designated_hash +end + +null_io_array = [] + while (line = STDIN.gets) line = line.resolve_invalid_bytes case line @@ -32,14 +53,15 @@ while (line = STDIN.gets) # Simple fstat: 0.3517 microseconds # Simple open/close: 1.3696 microseconds when /^Simple (\S+): (\d+.\d+) microseconds$/ + null_io_array << $2.to_f if $1.to_s == 'read' || $1.to_s == 'write' puts "syscall.#{$1}.latency.us: #{$2}" # Extract select test result. # Select on 100 fd's: 1.2293 microseconds # Select on 100 tcp fd's: 5.0377 microseconds - when /^Select on 100 fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 fd's: (\d+.\d+) microseconds$/ puts "Select.100fd.latency.us: #{$1}" - when /^Select on 100 tcp fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 tcp fd's: (\d+.\d+) microseconds$/ puts "Select.100tcp.latency.us: #{$1}" # Extract proc test result. @@ -198,5 +220,41 @@ while (line = STDIN.gets) # ... when /^"Mmap read open2close bandwidth$/ puts "MMAP.read_open2close.bandwidth.MB/sec: #{largest_bandwidth}" + + # "Random Memory load latency" + # 416.00000 92.824 + # 448.00000 92.772 + # 512.00000 92.640 + # ... + when /^Random load latency$/ + result_rand_mem = designated_file_size + puts "Rand_mem: #{result_rand_mem['Rand_mem']}" + + # "Memory load latency" + # 0.00049 1.539 + # 0.00098 1.539 + # 512.00000 74.421 + # ... + when /^Memory load latency$/ + result_rand_mem = designated_file_size + puts "L1_$: #{result_rand_mem['L1_$']}" + puts "L2_$: #{result_rand_mem['L2_$']}" + + # "mappings" + # 67.108864 82 + # 134.217728 151 + # 268.435456 309 + # 536.870912 615 + # ... + when /^"mappings$/ + result_map = designated_file_size + puts "Mmap_Latency: #{result_map['Mmap_Latency']}" end end + +num = 0 +null_io_array.each do |v| + num += v +end +null_num = format('%.4f', num / 2).to_f +puts "null_io: #{null_num}" -- 2.23.0

3 4

[PATCH v3 lkp-tests 6/6] iso2rootfs: allow the use of job.yaml and test script from the internet
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] Not all the yaml and script of test job will be committed to the lkp-tests repo. [How] Support three optional fields for every test in job.yaml: - testX_git_url: the url of git repo which contains yaml and script; - testX_git_yaml: the relative path of the yaml file in the git repo; - testX_git_script: the relative path of the test script in the git repo; Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 5 +++++ tests/iso2rootfs | 26 ++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 1631aca6d23f..1a643b538ac1 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -26,3 +26,8 @@ iso2rootfs: test1_yaml: test1_os_mount: test1_testbox: + ## the following three fields is required when your job.yaml and + ## script for this test are from the internet. + test1_git_url: + test1_git_yaml: + test1_git_script: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 4135cdea7059..0353c045a4bc 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -457,10 +457,36 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } +prepare_submit() +{ + local submit_dir="/tmp/iso2rootfs/submit_tmp" + mkdir -p "${submit_dir}" + + local submit_git_dir="${submit_dir}/${test_git_url##*/}" + submit_git_dir="${submit_git_dir%.git}" + + [ -d "${submit_git_dir}" ] || { + cd "${submit_dir}" + git clone "${test_git_url}" + } + + local test_git_yaml="${submit_git_dir}/$(eval echo "\$${test_num}_git_yaml")" + local test_git_script="${submit_git_dir}/$(eval echo "\$${test_num}_git_script")" + + [ -f "${test_git_yaml}" ] || die "cannot find file: ${test_num}_git_yaml." + cp "${test_git_yaml}" "${LKP_SRC}/jobs/" + + [ -f "${test_git_script}" ] || die "cannot find file: ${test_num}_git_script." + cp "${test_git_script}" "${LKP_SRC}/tests/" +} + submit_one_yaml() { log_info "starting submit ${test_yaml} ..." + local test_git_url="$(eval echo "\$${test_num}_git_url")" + [ -z "${test_git_url}" ] || prepare_submit + cd "${LKP_SRC}/jobs" [ -f "${test_yaml}" ] || die "cannot find yaml in LKP_SRC/jobs: ${test_yaml}." -- 2.23.0

1 0

[PATCH v3 lkp-tests 5/6] iso2rootfs: allow submit multiple test job.yaml for result rootfs
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] Result rootfs of dailybuild requires multiple test jobs. [How] 1. jobs/iso2rootfs.yaml. Before: test_yaml, test_os_mount, test_testbox After: test1_yaml, test1_os_mount, test1_testbox test2_yaml, test2_os_mount, test2_testbox ... 2. tests/iso2rootfs. - Use while-loop to traverse if $testX_yaml exists. - [ -n "${test1_yaml}" ] || return 0. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 8 ++++--- tests/iso2rootfs | 50 ++++++++++++++++++++++++++++++-------------- 2 files changed, 39 insertions(+), 19 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 629b53febb2f..1631aca6d23f 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -21,6 +21,8 @@ iso2rootfs: dailybuild_iso_url_file: # submit test yaml related fields - test_yaml: - test_os_mount: - test_testbox: + ## 1. you can add as many jobs as you like. + ## 2. the following three fields is requied for every test job. + test1_yaml: + test1_os_mount: + test1_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 3015c6f62947..4135cdea7059 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -457,17 +457,33 @@ generate_submit_append_str() echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } -submit_test_job_yaml() +submit_one_yaml() { - [ -n ${test_yaml} ] || return 0 - log_info "starting submit ${test_yaml} ..." - [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + cd "${LKP_SRC}/jobs" + [ -f "${test_yaml}" ] || die "cannot find yaml in LKP_SRC/jobs: ${test_yaml}." + + local test_os_mount="$(eval echo "\$${test_num}_os_mount")" + local test_testbox="$(eval echo "\$${test_num}_testbox")" + + [ -n "${test_os_mount}" ] || die "cannot find value of var: ${test_num}_os_mount." + [ -n "${test_testbox}" ] || die "cannot find value of var: ${test_num}_testbox." - [ -n "${test_os_mount}" ] || die "cannot find value of var: test_os_mount." - [ -n "${test_testbox}" ] || die "cannot find value of var: test_testbox." + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" + log_info "submit ${test_yaml} finished" +} + +submit_yamls() +{ + [ -n "$test1_yaml" ] || return 0 + + log_info "starting submit test yamls ..." + + [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh ./sbin/install-dependencies.sh @@ -475,17 +491,19 @@ submit_test_job_yaml() generate_global_yaml } - cd "${LKP_SRC}/jobs" && { - [ -f "${test_yaml}" ] || - die "cannot find test yaml in LKP_SRC/jobs. - test yaml: ${test_yaml}." + local test_yaml_index=1 + local test_num="test${test_yaml_index}" + local test_yaml="$(eval echo "\$${test_num}_yaml")" - "${LKP_SRC}/sbin/submit" \ - $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ - "${test_yaml}" - } + while [ -n "${test_yaml}" ] + do + submit_one_yaml + test_yaml_index=$((${test_yaml_index} +1)) + test_num="test${test_yaml_index}" + test_yaml="$(eval echo "\$${test_num}_yaml")" + done - log_info "submit ${test_yaml} finished" + log_info "submit test yamls finished" } ############ post works ############ @@ -525,7 +543,7 @@ main() config_rootfs test_rootfs - submit_test_job_yaml + submit_yamls post_works } -- 2.23.0

1 0