- Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci] container/result-webdav: add symbolic link after uploading file
by Cui Yili 24 Nov '20

24 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..1085492 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -14,7 +14,7 @@ http { server_tokens off; client_max_body_size 100m; - location / { + location /result { allow all; root /srv/; @@ -39,6 +39,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + second_dir = string.match(uri, "/.+/(.*)/.+") + + if (second_dir == "pkg") + then + link_name = "latest.cgz" + else + link_name = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..link_name) + } + } } } -- 2.23.0

3 2

[PATCH lkp-tests] makepkg.conf: mv -Wl from LDFLAGS to C(CXX)FLAGS
by Xu Xijian 24 Nov '20

24 Nov '20

-Wl parameter should be used by CFLAGS and CXXFLAGS instead of LDFLAGS. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- etc/makepkg.conf | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/etc/makepkg.conf b/etc/makepkg.conf index 093789f5..1fffdc2d 100644 --- a/etc/makepkg.conf +++ b/etc/makepkg.conf @@ -36,9 +36,8 @@ VCSCLIENTS=('bzr::bzr' # -march (or -mcpu) builds exclusively for an architecture # -mtune optimizes for an architecture, but builds for whole processor family CPPFLAGS="-D_FORTIFY_SOURCE=2" -CFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt" -CXXFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt" -LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" +CFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" +CXXFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" #-- Make Flags: change this for DistCC/SMP systems #MAKEFLAGS="-j2" #-- Debugging flags -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/2] job: fix each_jobs to fit multi-args
by Bai Jing 24 Nov '20

24 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash={"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03","os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20","os_arch"=>"aarch64"} h2 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} Because "@job2=hash" will merge the h2 to @job after "each_job". We need to merge the h2 to @job before "each_job". So do the merge advance and set @job2={} Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index bc6c91ef..2062ba58 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -727,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH v4 lkp-tests 1/2] lib: add multi-args in job yaml
by Bai Jing 24 Nov '20

24 Nov '20

[original style] os: openeuler os_version: 20.03 os_arch: aarch64 os: openeuler os_version: 20 os_arch: aarch64 [now style] os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 [point] Currently, only one key of this style(xx xxx xx) can be contained in YAML files. Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 2e94316c..bc6c91ef 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -250,7 +250,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) - @jobs << hash + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e log_error "#{jobfile}: " + e.message @@ -280,6 +280,30 @@ class Job @jobfile = jobfile end + def multi_args(hash) + jobs_array = [] + hash.each { |key, value| + next unless key =~ /^\w.*\s+.*\w$/ + + next unless value.class == Array + + key_array = key.split + value.each do |v| + v_array = v.split + next unless key_array.size == v_array.size + + hash_job = hash.clone + key_array.size.times do |i| + hash_job[key_array[i]] = v_array[i] + end + hash_job.delete(key) + jobs_array << hash_job + end + } + jobs_array = [hash] if jobs_array.empty? + return jobs_array + end + def delete_keys_from_spec(spec_file) return unless File.exist? spec_file -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/build-pkg: fix syntax errors caused by replacement
by Lin Jiaxin 24 Nov '20

24 Nov '20

error: //PKGBUILD: line 13: syntax error near unexpected token `)' //PKGBUILD: line 13: ` "$_pkgname.desktop")' The original logic only adapts to the situation where source has only one value. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index c5d55eda..65c83e22 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -90,6 +90,11 @@ create_softlink() ln -sf "${bm_name}" "${soft_path}/latest.cgz" } +replace_source() +{ + sed -i "s|^source=[^)]*|$1|g" "$2" +} + build_source_pkg() { local repo_dir="" @@ -103,11 +108,9 @@ build_source_pkg() repo_dir="${repo_dir%.git*}::" fi - upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\")" - sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD - [ -n "$PKGBUILD_TAG" ] && { - sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" - } + upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + replace_source "${upstream_source}" PKGBUILD + [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" } cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" -- 2.23.0

1 0

[PATCH lkp-tests] lib/job2sh.rb: the job value obtaining logic is modified
by Hu Xuejiao 24 Nov '20

24 Nov '20

[why] the original logic is too round Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 14bf7159..7f9cf562 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -287,9 +287,7 @@ class Job2sh < Job @setups = available_programs(:setup) @programs = available_programs(:workload_elements) - if !job - job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job - end + job ||= (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job @cur_func = :extract_stats @@ -317,7 +315,7 @@ class Job2sh < Job @setups = available_programs(:setup) @programs = available_programs(:workload_elements) - if !job + unless job job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job job.delete_if { |key, val| parse_one([], key, val, :PASS_EXPORT_ENV) } end -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/wrapper support user define file to parse result
by Lu Weitao 23 Nov '20

23 Nov '20

[Why] User need debug their script which haven't commit to lkp-tests/stats [How] lkp-tests/stats/wrapper calls the program-script in result_root if lkp-tests/stats/ have no program-script [Dependence] Write job['define_files'] to file in result_root, job can query from ES DB Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/wrapper | 3 +++ 1 file changed, 3 insertions(+) diff --git a/stats/wrapper b/stats/wrapper index a44e9f31..d13249d2 100755 --- a/stats/wrapper +++ b/stats/wrapper @@ -140,6 +140,9 @@ check_empty_output check_binary_output tmpfile=$(mktemp -p /tmp -t lkp-stats.XXXXXXXX) + +[[ -f "$RESULT_ROOT/stats/$program" ]] && program_dir="$RESULT_ROOT/stats" + if [[ -f $log ]]; then $program_dir/$program $log < $log > $tmpfile || { log_error "$program_dir/$program $log < $log exit code $?, check $tmpfile"; exit 1; } else -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] doc/job/: add define_files.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing the define_files field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/define_files.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/define_files.md diff --git a/doc/job/define_files.md b/doc/job/define_files.md new file mode 100644 index 0000000..2c3457d --- /dev/null +++ b/doc/job/define_files.md @@ -0,0 +1,16 @@ +# define_files + +Meaning: +- We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: + distro/depends/$testcase + distro/depends/$testcase-dev + pkg/$testcase/PKGBUILD + stats/$testcase + tests/$testcase + jobs/$testcase.yaml +- When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. +- The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. + +Usage example: +- % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 +- % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] doc/job/: add initrd_deps.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing initrd_deps field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/initrd_deps.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/job/initrd_deps.md diff --git a/doc/job/initrd_deps.md b/doc/job/initrd_deps.md new file mode 100644 index 0000000..0c3d79c --- /dev/null +++ b/doc/job/initrd_deps.md @@ -0,0 +1,5 @@ +# initrd_deps + +Meaning: +- Most testcases depend on many other package to run. If user has already packed all the other package for running testcase. Then initrd_deps field would give the location of the package. +- The initrd_deps field do not need to be specified by user in the $testcase yaml. It is filled in by the scheduler. User can get the package through the wget command to check the rpm in compressed package. -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] doc/job/: add initrd_pkg.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing the initrd_pkg field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/initrd_pkg.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/job/initrd_pkg.md diff --git a/doc/job/initrd_pkg.md b/doc/job/initrd_pkg.md new file mode 100644 index 0000000..2ef3d03 --- /dev/null +++ b/doc/job/initrd_pkg.md @@ -0,0 +1,5 @@ +# initrd_pkg + +Meaning: +- Most testcases rely on many other files such as configuration files to run. If user has already packed all the files for running testcase. Then initrd_pkg field would give the location of the packed compression file. +- The initrd_pkg field do not need to be specified by user in the $testcase yaml. It is filled in by the scheduler. User can get the package through the wget command to check the files in compressed package. -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: return ips of all nodes
by Ren Wen 23 Nov '20

23 Nov '20

Return all nodes' ips when request state is "roles_ip". Example: If roles of all nodes are: "server", "client", ... ips of all nodes are: "<ip01>", "<ip02>", ... Then return: sever=<ip01> client=<ip02> ... All nodes will export the return content as environment variables line by line. --- src/scheduler/request_cluster_state.cr | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 5bbbad1..72e3810 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -52,11 +52,16 @@ class Sched "direct_macs" => direct_macs} update_cluster_state(cluster_id, job_id, job_info) when "roles_ip" - role = "server" - role_state = get_role_state(cluster_id, role) - raise "Missing #{role} state in cluster state" unless role_state - return "server=#{role_state["ip"]}\n" \ - "direct_server_ips=#{role_state["direct_ips"]}" + cluster_state = get_cluster_state(cluster_id) + roles_ip = [] of String + + cluster_state.each_value do |host_state| + roles = host_state["roles"] + direct_ips = host_state["direct_ips"] + roles_ip << "#{roles}=#{direct_ips}" + end + + return roles_ip.join('\n') end # show cluster state -- 2.23.0

2 2

[PATCH compass-ci] lib/job.cr: fix repeated setting job
by Wu Zhende 23 Nov '20

23 Nov '20

Only first time should full initialize the job. Next time function "initialized?" will return true, just to use job_content, don't need to set default values. But now, "kernel_version" will let "initialized?" return false. Fix this. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..512f256 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -63,6 +63,8 @@ class Job set_defaults() end + # Each key must exist after the job is initialized + # Otherwise "initialized"? will return false METHOD_KEYS = %w( id os @@ -91,7 +93,6 @@ class Job kernel_append_root kernel_params docker_image - kernel_version linux_vmlinuz_path linux_modules_initrd linux_headers_initrd @@ -312,7 +313,7 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? + suffix = "-#{self["kernel_version"]}" if self["kernel_version"]? self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") if "#{os_mount}" == "initramfs" self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") -- 2.23.0

2 1

[PATCH compass-ci 1/3] doc/job/: add define_files.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing the define_files field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/define_files.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/define_files.md diff --git a/doc/job/define_files.md b/doc/job/define_files.md new file mode 100644 index 0000000..dec74f1 --- /dev/null +++ b/doc/job/define_files.md @@ -0,0 +1,16 @@ +# define_files + +Meaning: +- We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: + distro/depends/$testcase + distro/depends/$testcase-dev + pkg/$testcase/PKGBUILD + stats/$testcase + tests/$testcase + jobs/$testcase.yaml +- When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. +- The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically by the scheduler when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. + +Usage example: +- % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 +- % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml -- 2.23.0

2 2

[PATCH compass-ci 2/2] lib/json_logger.cr: add a new class for json log
by Wu Zhende 23 Nov '20

23 Nov '20

[Why] The built-in log system prints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example1] use: log.info("test") output: {"level":"INFO","level_num":1,"time":"2020-11-19 16:57:17+0800","progname":"test","message":"test"} [Example2] use: log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":"test", "job_id":"1","job_state":"submit","caller":["./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 src/lib/json_logger.cr diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr new file mode 100755 index 0000000..371c042 --- /dev/null +++ b/src/lib/json_logger.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "logger" +require "json" +require "any_merge" + +class JSONLogger < Logger + FORMATTER = Logger::Formatter.new do | severity, datetime, progname, msg, io| + level_num = severity.to_i32 + logger_hash = JSON.parse(%({"level_num": #{level_num}, + "level": "#{severity}", + "time": "#{datetime}" + })).as_h + + logger_hash.any_merge!({"progname" => progname}) unless progname.empty? + logger_hash.merge!(JSON.parse(%({"caller": #{caller}})).as_h) if level_num >= 2 + + begin + message = JSON.parse(msg).as_h + rescue + message = {"message" => msg} + end + logger_hash.any_merge!(message) + + io << logger_hash.to_json + end + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

2 1

[PATCH v10 compass-ci 3/3] lib: fix group compare no output when dimension number less than 2
by Lu Kaiyi 23 Nov '20

23 Nov '20

[why] When do group compare, if returned groups of auto_group like below (the dimension number less than 2 for group_key): groups = { group_key_1 => { dimension_1 => matrix_1 }, group_key_2 => {} } the group_key will be deleted, then, group compare will have no output. [how] Reduce dimension number limit from 2 to 1. [example] lukaiyi@z9 ~% compare "suite=iperf" -d "os" in z9 [before] <null> [after] os_arch=aarch64/os_version=20/pp.iperf.protocol=tcp/pp.iperf.runtime=300/tbox_group=vm-hi1620-2p8g--liangt openeuler metric -------------------- ------------------------------ fails:runs | 18:18 stderr.EOF' 18:18 stderr.has_stderr openeuler metric -------------------- ------------------------------ %stddev \ 3.496630e+10 ± 19% iperf.tcp.receiver.bps 3.497117e+10 ± 19% iperf.tcp.sender.bps 300.23 iperf.time.elapsed_time 300.23 iperf.time.elapsed_time.max ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 2 +- lib/params_group.rb | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2065039..82cf033 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,7 +116,7 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| - if value.size < 2 + if value.empty? groups.delete(group_key) next end diff --git a/lib/params_group.rb b/lib/params_group.rb index 40191a9..48d651b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -34,8 +34,7 @@ def auto_group(jobs_list, dimensions) return [] if dimensions.empty? jobs_list = extract_jobs_list(jobs_list) - groups = group(jobs_list, dimensions) - return remove_singleton(groups) + group(jobs_list, dimensions) end def extract_jobs_list(jobs_list) -- 2.23.0

1 0

[PATCH v10 compass-ci 2/3] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 23 Nov '20

23 Nov '20

[why] Avoid to execute some unnecessary operations and improve performance. [how] Move the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ff4ea31..2065039 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

1 0

[PATCH v10 compass-ci 1/3] lib: keep suite result in top for compare output
by Lu Kaiyi 23 Nov '20

23 Nov '20

[why] Currently, the compare output result default ordered by alphabet. If there are too many lines in result, it will be difficult for user to find the lines that start with $suite in metric column. So, need reorder compare output result with keeping $suite item on the top of the table. [how] Divide the lines into two parts before printing output according to whether the lines start with $suite, then combine output result into a whole to highlight the important info. [example] lukaiyi@z9 ~% compare "id=z9.153094" "id=z9.153725" in z9 suites_list:["sysbench-mysql"] [before] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 58.80 +0.0% 58.82 boot-time.boot 5522.36 -0.1% 5515.17 boot-time.idle 21.22 -0.1% 21.19 boot-time.kernel_boot 9913.00 -2.1% 9703.00 interrupts.228:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 ... 396953.00 -0.6% 394693.00 softirqs.SCHED 26445.00 -5.8% 24902.00 softirqs.TASKLET 1364789.00 -0.2% 1362689.00 softirqs.TIMER 24925.67 +1.3% 25240.95 sysbench-mysql.transactions 2585.92 +0.2% 2592.20 uptime.boot 239274.99 +0.2% 239786.47 uptime.idle 9062055.00 -0.0% 9059563.00 vmstat.memory.cache 243316968.00 +0.0% 243352799.00 vmstat.memory.free ... [after] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 24925.67 +1.3% 25240.95 sysbench-mysql.transactions 58.80 +0.0% 58.82 boot-time.boot 5522.36 -0.1% 5515.17 boot-time.idle 21.22 -0.1% 21.19 boot-time.kernel_boot 9913.00 -2.1% 9703.00 interrupts.228:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 3526557.00 +5.1% 3707673.00 interrupts.4:GICv3.26.Level.arch_timer ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 16 ++++++++++------ lib/compare_matrixes.rb | 41 ++++++++++++++++++++++++++++------------- lib/matrix2.rb | 20 +++++++++++++++----- 3 files changed, 53 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..f34e239 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suite_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suite_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,16 @@ end def create_matrices_list(conditions) matrices_list = [] + suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suite_list.concat(suites) end - matrices_list + + return matrices_list, suite_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 762c087..14c14cb 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -128,7 +128,7 @@ def set_compare_values(index, values, field, success, options) end def get_values_by_field(matrixes_list, field, matrixes_size, success, options) - # get values by field, values struce example: values[0][:average] + # get values by field, values struct example: values[0][:average] # values = {} matrixes_list.length.times do |index| @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suite_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -808,26 +809,40 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suite_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suite_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suite_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suite_list, matrixes_titles, success, theme) return '' if values.empty? + suite_set = Set.new(suite_list) result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + ranked_str = get_ranked_str(values, suite_set, success, theme) + result_str += ranked_str + result_str +end + +def get_ranked_str(values, suite_set, success, theme) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suite_set.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end - result_str + suite_str + common_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..ff4ea31 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,23 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end - groups.delete(group_key) if value.size < 2 + if value.size < 2 + groups.delete(group_key) + next + end + suites_list << suite_list end - groups + + return groups, suites_list end # input: @@ -139,7 +149,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], _ = create_matrix(jobs) end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] sbin/my-submit: this script is used together with the my-qemu
by Xiao Shenwei 23 Nov '20

23 Nov '20

you can ignore the queue field, it will add default value which is "$tbox_group~$USER" usage: my-submit testbox=vm-2p8g jobs/borrow-1h.yaml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/my-submit | 3 +++ 1 file changed, 3 insertions(+) create mode 100755 sbin/my-submit diff --git a/sbin/my-submit b/sbin/my-submit new file mode 100755 index 00000000..57489948 --- /dev/null +++ b/sbin/my-submit @@ -0,0 +1,3 @@ +#!/bin/bash + +submit --my-queue $@ -- 2.23.0

3 2

[PATCH compass-ci 1/2] lib/json_logger.rb: refactor log.rb
by Wu Zhende 23 Nov '20

23 Nov '20

1.change log.rb to json_logger.rb 2.change Log to JSONLogger 3.change datetime to time 4.don't output empty values 5.msg use hash or string(originally json or string) Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/{log.rb => json_logger.rb} | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) rename lib/{log.rb => json_logger.rb} (65%) diff --git a/lib/log.rb b/lib/json_logger.rb similarity index 65% rename from lib/log.rb rename to lib/json_logger.rb index aad00ca..6d1173e 100644 --- a/lib/log.rb +++ b/lib/json_logger.rb @@ -6,7 +6,7 @@ require 'logger' require 'json' # print logs in JSON format -class Log < Logger +class JSONLogger < Logger LEVEL_INFO = { 'DEBUG' => 0, 'INFO' => 1, @@ -18,21 +18,19 @@ class Log < Logger FORMATTER = proc { |severity, datetime, progname, msg| level_num = LEVEL_INFO[severity] - begin - message = JSON.parse(msg) - rescue JSON::ParserError - message = { 'message' => msg } - end - h = { + logger_hash = { 'level' => severity.to_s, 'level_num' => level_num, - 'datetime' => datetime, - 'progname' => progname, - 'message' => '' + 'time' => datetime } - h.merge!(message) - h.merge!({ 'caller' => caller }) if level_num >= 2 - h.to_json + + logger_hash['progname'] = progname if progname + logger_hash['caller'] = caller if level_num >= 2 + + msg = { 'message' => msg } unless msg.is_a?(Hash) + logger_hash.merge!(msg) + + logger_hash.to_json } def initialize(logdev = STDOUT, formatter = FORMATTER) -- 2.23.0

2 1

[PATCH compass-ci] sched: save the upload dirs in redis
by Cao Xueliang 23 Nov '20

23 Nov '20

Before: Use access_key_file to limit job upload, need to create the file in upload dirs, that's not flexible enough. After: Set $job_id => "$dir1,$dir2,..." in "id2upload_dirs" hash of the redis. The dir of job upload must be equal or deeper than one of "$dir1,$dir2,...". Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 13 ++++--------- src/scheduler/close_job.cr | 6 ------ src/scheduler/find_job_boot.cr | 10 +++++----- 3 files changed, 9 insertions(+), 20 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..cb187a4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -83,8 +83,7 @@ class Job pp_params submit_date result_root - access_key - access_key_file + upload_dirs lkp_initrd_user user_lkp_src kernel_uri @@ -206,15 +205,11 @@ class Job def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group self["result_root"] = File.join("/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}") - - # access_key has information based on "result_root" - # so when set result_root, we need redo set_ to update it - set_access_key() + set_upload_dirs() end - private def set_access_key - self["access_key"] = "#{Random::Secure.hex(10)}-#{id}" - self["access_key_file"] = File.join("/srv/", "#{result_root}", ".#{access_key}") + def set_upload_dirs + self["upload_dirs"] = "#{result_root}" end private def set_result_service diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d071d69..b5aaa63 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -5,8 +5,6 @@ class Sched def close_job(job_id : String) job = @redis.get_job(job_id) - delete_access_key_file(job) if job - response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -24,8 +22,4 @@ class Sched return %({"job_id": "#{job_id}", "job_state": "complete"}) end - - def delete_access_key_file(job : Job) - File.delete(job.access_key_file) if File.exists?(job.access_key_file) - end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b5a23c5..968b9ed 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -128,10 +128,6 @@ class Sched return response end - def touch_access_key_file(job : Job) - FileUtils.touch(job.access_key_file) - end - private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" response += job.initrds_uri @@ -142,8 +138,12 @@ class Sched return response end + def set_id2upload_dirs(job) + @redis.hash_set("sched/id2upload_dirs",job.id, job.upload_dirs) + end + def boot_content(job : Job | Nil, boot_type : String) - touch_access_key_file(job) if job + set_id2upload_dirs(job) if job case boot_type when "ipxe" -- 2.23.0

1 1

[PATCH v3 compass-ci] sbin/es-jobs-mapping.sh: Support more fields in "jobs" index
by Zhang Yuhang 23 Nov '20

23 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..8e19662 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,9 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' \ + -d '{"index.mapping.total_fields.limit": 10000}' fi fi -- 2.23.0

1 0

[PATCH v1 compass-ci] sched: set the upload dirs to redis
by Cao Xueliang 23 Nov '20

23 Nov '20

Before: Use access_key_file to limit job upload, need to create the file in upload dirs, that's not flexible enough. After: Set $job_id => "$dir1,$dir2,..." in "id2upload_dirs" hash of the redis. The dir of job upload must be equal or deeper than one of "$dir1,$dir2,...". Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 13 ++++--------- src/scheduler/close_job.cr | 6 ------ src/scheduler/find_job_boot.cr | 10 +++++----- 3 files changed, 9 insertions(+), 20 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..cb187a4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -83,8 +83,7 @@ class Job pp_params submit_date result_root - access_key - access_key_file + upload_dirs lkp_initrd_user user_lkp_src kernel_uri @@ -206,15 +205,11 @@ class Job def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group self["result_root"] = File.join("/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}") - - # access_key has information based on "result_root" - # so when set result_root, we need redo set_ to update it - set_access_key() + set_upload_dirs() end - private def set_access_key - self["access_key"] = "#{Random::Secure.hex(10)}-#{id}" - self["access_key_file"] = File.join("/srv/", "#{result_root}", ".#{access_key}") + def set_upload_dirs + self["upload_dirs"] = "#{result_root}" end private def set_result_service diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d071d69..b5aaa63 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -5,8 +5,6 @@ class Sched def close_job(job_id : String) job = @redis.get_job(job_id) - delete_access_key_file(job) if job - response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -24,8 +22,4 @@ class Sched return %({"job_id": "#{job_id}", "job_state": "complete"}) end - - def delete_access_key_file(job : Job) - File.delete(job.access_key_file) if File.exists?(job.access_key_file) - end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b5a23c5..968b9ed 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -128,10 +128,6 @@ class Sched return response end - def touch_access_key_file(job : Job) - FileUtils.touch(job.access_key_file) - end - private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" response += job.initrds_uri @@ -142,8 +138,12 @@ class Sched return response end + def set_id2upload_dirs(job) + @redis.hash_set("sched/id2upload_dirs",job.id, job.upload_dirs) + end + def boot_content(job : Job | Nil, boot_type : String) - touch_access_key_file(job) if job + set_id2upload_dirs(job) if job case boot_type when "ipxe" -- 2.23.0

1 0

[PATCH v3 compass-ci] container/result-webdav: add symbolic link after uploading file
by Cui Yili 23 Nov '20

23 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..c9199ed 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -14,7 +14,7 @@ http { server_tokens off; client_max_body_size 100m; - location / { + location /result { allow all; root /srv/; @@ -39,6 +39,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + seconddir = string.match(uri, "/.+/(.*)/.+") + + if (seconddir == "pkg") + then + linkname = "latest.cgz" + else + linkname = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..linkname) + } + } } } -- 2.23.0

2 1

[PATCH compass-ci 2/2] container/result-webdav: add symbolic link after uploading file
by Cui Yili 23 Nov '20

23 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..0cb786c 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -13,8 +13,10 @@ http { server_name "result-webdav"; server_tokens off; client_max_body_size 100m; + access_log /tmp/access.log; + error_log /tmp/error.log; - location / { + location /result { allow all; root /srv/; @@ -39,6 +41,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + seconddir = string.match(uri, "/.+/(.*)/.+") + + if (seconddir == "pkg") + then + linkname = "latest.cgz" + else + linkname = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..linkname) + } + } } } -- 2.23.0

2 1

[PATCH v2 compass-ci] container/result-webdav: add symbolic link after uploading file
by Cui Yili 23 Nov '20

23 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..34df004 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -14,7 +14,7 @@ http { server_tokens off; client_max_body_size 100m; - location / { + location /result { allow all; root /srv/; @@ -39,6 +39,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + seconddir = string.match(uri, "/.+/(.*)/.+") + + if (seconddir == "pkg") + then + linkname = "latest.cgz" + else + linkname = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..linkname) + } + } } } -- 2.23.0

1 0