- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/minio: add minio container
by Cui Yili 16 Nov '20

16 Nov '20

Add minio container to store files. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/minio/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/minio/build diff --git a/container/minio/build b/container/minio/build new file mode 100755 index 0000000..d1ccd06 --- /dev/null +++ b/container/minio/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker pull jessestuart/minio -- 2.23.0

2 1

[PATCH v4 compass-ci] /lib: keep suite result in top for compare output
by Lu Kaiyi 16 Nov '20

16 Nov '20

[why] Currently, the compare output result default order by alphabet. If there are too many lines in result, it will be difficult for user to find the lines that starts with $suite in metric column. So, need reorder compare output result with keeping $suite item on the top of the table. [how] Divide the lines into two parts before print according to whether the line start with $suite or not, then combine output result into a whole to highlight the important info. [example] command: compare "id=z9.148096" "id=z9.148143" in z9 suites_list:["sysbench-mysql"] [before] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 ... 1266042.00 -18.3% 1034206.00 softirqs.RCU 547849.00 -27.4% 397593.00 softirqs.SCHED 159990.00 -80.5% 31181.00 softirqs.TASKLET 1617967.00 -23.0% 1245331.00 softirqs.TIMER 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 2149.23 -100.0% 0.00 time.elapsed_time 2149.23 -100.0% 0.00 time.elapsed_time.max 214458.00 -100.0% 0.00 time.involuntary_context_switches 2.00 -100.0% 0.00 time.major_page_faults 192832.00 -100.0% 0.00 time.maximum_resident_set_size ... [after] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 2894304.00 +12.8% 3265730.00 interrupts.4:GICv3.26.Level.arch_timer 11546998.00 -100.0% 0.00 interrupts.746:ITS-MSI.99092481.Edge.hns3-0000:bd:00.1-TxRx-0 7476064.00 -100.0% 0.00 interrupts.747:ITS-MSI.99092482.Edge.hns3-0000:bd:00.1-TxRx-2 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 16 ++++++++++------ lib/compare_matrixes.rb | 41 ++++++++++++++++++++++++++++------------- lib/matrix2.rb | 15 +++++++++++---- 3 files changed, 49 insertions(+), 23 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..813e08c 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suites_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suites_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,16 @@ end def create_matrices_list(conditions) matrices_list = [] + suites_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suites_list.concat(suites) end - matrices_list + + return matrices_list, suites_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..bd1e646 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -128,7 +128,7 @@ def set_compare_values(index, values, field, success, options) end def get_values_by_field(matrixes_list, field, matrixes_size, success, options) - # get values by field, values struce example: values[0][:average] + # get values by field, values struct example: values[0][:average] # values = {} matrixes_list.length.times do |index| @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suites_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suites_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -807,26 +808,40 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suites_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suites_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suites_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suites_list, matrixes_titles, success, theme) return '' if values.empty? + suites_list.uniq! result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + ranked_str = get_ranked_str(values, suites_list, success, theme) + result_str += ranked_str + result_str +end + +def get_ranked_str(values, suites_list, success, theme) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suites_list.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end - result_str + suite_str + common_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..479d280 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,20 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end groups.delete(group_key) if value.size < 2 + suites_list << suite_list unless value.size < 2 end - groups + + return groups, suites_list end # input: @@ -139,7 +146,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], = create_matrix(jobs) end end -- 2.23.0

3 4

[PATCH v4 compass-ci 4/4] container/git-mirrir: add Docker env GIT_ASKPASS
by Li Yuanchao 16 Nov '20

16 Nov '20

[why] some repositories need username and password, but we don't know. And the problem is, it will block there as we have no username and password. Add GIT_ASKPASS=/kill-parent.sh, will kill the block one, and go to next repository. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/Dockerfile | 3 +++ container/git-mirror/kill-parent.sh | 3 +++ container/git-mirror/start | 1 + 3 files changed, 7 insertions(+) create mode 100755 container/git-mirror/kill-parent.sh diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 390bbcc..1180a7b 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -2,6 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM alpine:3.8 +MAINTAINER Li Yuanchao <lyc163mail(a)163.com> + +ADD kill-parent.sh / RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories diff --git a/container/git-mirror/kill-parent.sh b/container/git-mirror/kill-parent.sh new file mode 100755 index 0000000..6418592 --- /dev/null +++ b/container/git-mirror/kill-parent.sh @@ -0,0 +1,3 @@ +#!/bin/sh + +kill $PPID diff --git a/container/git-mirror/start b/container/git-mirror/start index dd4fd63..ee79578 100755 --- a/container/git-mirror/start +++ b/container/git-mirror/start @@ -17,6 +17,7 @@ cmd = %W[ -d -e CCI_SRC=#{DOCKER_CCI} -e REPO_SRC=#{DOCKER_REPO} + -e GIT_ASKPASS=/kill-parent.sh -v #{ENV['CCI_SRC']}:#{DOCKER_CCI} -v /etc/localtime:/etc/localtime:ro -v #{ENV['REPO_SRC']}:#{DOCKER_REPO} -- 2.23.0

1 0

[PATCH v4 compass-ci 3/4] git_mirror: use "gitclone.com" to make "git clone" faster
by Li Yuanchao 16 Nov '20

16 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 8f0497e..9648063 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -31,12 +31,18 @@ class GitMirror @feedback_queue.push(@feedback_info) end - def git_clone(url, mirror_dir) - ret = false - url = Array(url)[0] + def get_url(url) if url.include?('gitee.com/') && File.exist?("/srv/git/#{url.delete_prefix('https://')}") url = "/srv/git/#{url.delete_prefix('https://')}" + elsif url.include?('://github.com') + url = "https://gitclone.com/#{url.split('://')[1]}" end + return url + end + + def git_clone(url, mirror_dir) + ret = false + url = get_url(Array(url)[0]) 10.times do stderr = %x(git clone --mirror #{url} #{mirror_dir} 2>&1) ret = !stderr.include?('fatal') -- 2.23.0

1 0

[PATCH v4 compass-ci 2/4] git_mirror: fix that worker thread die after system(git clone)
by Li Yuanchao 16 Nov '20

16 Nov '20

[why] As Signal.trap(:SIGCHLD, 'SIG_IGN') has been set, the init process will handle the return status of child process, so ret = system(git clone xxx) can't get return value, which cause worker thread die. Use %x() instead of system(), and use stderr to check if 'git clone' succeed. So return value is not needed. [errmsg] in `system': No child process - Another thread waited the process started by system(). (Errno::ECHILD) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 55ec67d..8f0497e 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -38,7 +38,8 @@ class GitMirror url = "/srv/git/#{url.delete_prefix('https://')}" end 10.times do - ret = system("git clone --mirror #{url} #{mirror_dir}") + stderr = %x(git clone --mirror #{url} #{mirror_dir} 2>&1) + ret = !stderr.include?('fatal') break if ret end FileUtils.rm_r(mirror_dir) unless ret -- 2.23.0

1 0

[PATCH lkp-tests] jobs: add optimization parameters to job.yaml
by Zhang Yu 16 Nov '20

16 Nov '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- jobs/sysbench-mysql.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml index 83795a18..12c83087 100644 --- a/jobs/sysbench-mysql.yaml +++ b/jobs/sysbench-mysql.yaml @@ -2,6 +2,8 @@ suite: sysbench-mysql category: benchmark cluster: cs-lkp-hsw-ep5 +kernel_version: 4.19.90-mysql +kernel_custom_params: "sched_steal_node_limit=8" if role server: mysql: -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: support more testbox formats
by Li Ping 16 Nov '20

16 Nov '20

If user wants to reproduce the result directly in the result directory. [problem] /srv/result/iperf/vm-2p8g/2020-11-09/z9.147520% submit job.yaml output: /home/liping/lkp-tests/lib/job.rb:344:in `get_hosts_file': hosts file not exist: vm-2p8g.taishan200-2280-2s64p-256g, maybe need check testbox field (ArgumentError) --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..6dd93f01 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -337,7 +337,7 @@ class Job lab_hosts_file = get_lab_hosts_file return lab_hosts_file if lab_hosts_file - hosts_file_name = @job['tbox_group'].split('--')[0] + hosts_file_name = @job['tbox_group'].split(/(\.|--)/)[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" return hosts_file if File.file?(hosts_file) -- 2.23.0

2 2

[PATCH v4 compass-ci] sparrow: fix 3-code/dev-env no such file error
by Liu Yinsi 16 Nov '20

16 Nov '20

when execute 3-code/dev-env, error message as follows: ./dev-env: line 45: ./os/openEuler: No such file or directory no file ./os/openEuler is normal, will not affect deploy CCI, so fix it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 77027d9..9fcc1b4 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -42,4 +42,4 @@ EOF source /etc/os-release -. $(dirname ${BASH_SOURCE[0]})/os/${ID} +. $(dirname ${BASH_SOURCE[0]})/os/${ID} 2> /dev/null -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/2] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 15 Nov '20

15 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe..1faa1e9f 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -37,6 +37,23 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +check_pkgfile() +{ + PKGBUILD_TAG="-$PKGBUILD_TAG" + PKGBUILD_TAG_FILE="PKGBUILD$PKGBUILD_TAG" + + get_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 @@ -69,6 +86,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && check_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

2 1

[PATCH compass-ci] lib/es-query: fix NoMethodError
by Lu Weitao 14 Nov '20

14 Nov '20

[Why] Traceback (most recent call last): 4: from /c/compass-ci/sbin/compare:70:in `<main>' 3: from /c/compass-ci/lib/compare.rb:97:in `compare_by_template' 2: from /c/compass-ci/lib/compare.rb:128:in `create_groups_matrices' 1: from /c/compass-ci/lib/es_query.rb:21:in `multi_field_query' /c/compass-ci/lib/es_query.rb:54:in `build_mutli_field_subquery_body': undefined method `each' for nil:NilClass (NoMethodError) [Root cause] when (sbin/compare) call es_query, may pass empty filter as input parameter [How] exit when the filter is empty Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 037213d..c027aa9 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -18,6 +18,10 @@ class ESQuery # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) def multi_field_query(items, size: 10_000) + unless items + warn 'empty filter!' + exit + end query_fields = build_mutli_field_subquery_body items query = { query: { -- 2.23.0

1 0