- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/2] result-webdav/start: add share dir to upload file
by Cui Yili 23 Nov '20

23 Nov '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index e0b6676..01141c3 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -14,6 +14,7 @@ cmd=( -p 3080:3080 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf -v /srv/result/:/srv/result/ + -v /srv/initrd/:/srv/initrd/ result-webdav ) -- 2.23.0

1 0

[PATCH v2 compass-ci] add api /compare_template
by Lu Weitao 23 Nov '20

23 Nov '20

[Why] user need to compare the specific metrics [how] add new api: /compare_template that can be access like: curl -X POST 'localhost:32767/compare_template' -H 'Content-Type: Application/json' -d '{ "compare_metrics": ["fio.write_iops", "fio.read_iops"], "filter": {"suite": ["fio-basic"], "os_arch": ["aarch64", "x86"]}, "compare_dimensions": [ {"os": "debian", "os_version": "sid"}, {"os": "openeuler", "os_version": 20.03} ], "x_params": ["bs", "test_size"], "title": "Hackbench Performance Testing", "unit": "KB/s"}' Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 32 +++++++++++++++++++++++++++++++ src/lib/web_backend.rb | 18 +++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 7db82ac..d68e9f2 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -71,3 +71,35 @@ end get '/get_repos' do get_repos(params) end + +# POST /compare_template +# - header: "Content-Type: Application/json" +# - body: json +# eg: +# { +# "compare_metrics": ["fio.write_iops", "fio.read_iops"], +# "filter":{ +# "suite": ["fio-basic"], +# "os_arch": ["aarch64", "x86"] +# }, +# "compare_dimensions": [ +# { +# "os": "debian", +# "os_version": "sid" +# }, +# { +# "os": "openeuler", +# "os_version: 20.03 +# } +# ], +# "x_params": ["bs", "test_size"], +# "title": "Hackbench Performance Testing", +# "unit": "KB/s" +# } +# +# Response: +# - compare result +post '/compare_template' do + request.body.rewind # in case someone already read it + compare_template(request.body.read) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 1f7224e..23716cb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -413,3 +414,20 @@ def get_repos(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def compare_template(data) + begin + body = template_body(JSON.parse(data)) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'compare error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def template_body(request_body) + groups_matrices = create_groups_matrices(request_body) + compare_results = compare_metrics_values(groups_matrices) + formatter = FormatEchartData.new(compare_results, request_body) + formatter.format_for_echart.to_json +end -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix syntax errors caused by replacement
by Lin Jiaxin 23 Nov '20

23 Nov '20

error: //PKGBUILD: line 13: syntax error near unexpected token `)' //PKGBUILD: line 13: ` "$_pkgname.desktop")' The original logic only adapts to the situation where source has only one value. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index c5d55eda..6ad825d7 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -103,10 +103,10 @@ build_source_pkg() repo_dir="${repo_dir%.git*}::" fi - upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\")" - sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD + upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + sed -i "s|^source=[^)]*|${upstream_source}|g" PKGBUILD [ -n "$PKGBUILD_TAG" ] && { - sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" + sed -i "s|^source=[^)]*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" } } -- 2.23.0

2 1

[PATCH lkp-tests] stats/wrapper support user define file to parse result
by Lu Weitao 23 Nov '20

23 Nov '20

[Why] User need debug their script which haven't commit to lkp-tests/stats [How] lkp-tests/stats/wrapper calls the program-script in result_root if lkp-tests/stats/ have no program-script [Dependence] Write job['define_files'] to file in result_root, job can query from ES DB Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/wrapper | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/stats/wrapper b/stats/wrapper index a44e9f31..03db5975 100755 --- a/stats/wrapper +++ b/stats/wrapper @@ -140,6 +140,13 @@ check_empty_output check_binary_output tmpfile=$(mktemp -p /tmp -t lkp-stats.XXXXXXXX) + +if [[ ! -f $program_dir/$program ]]; then + if [[ -f $RESULT_ROOT/"stats"/$program ]]; then + program_dir=$RESULT_ROOT/"stats" + fi +fi + if [[ -f $log ]]; then $program_dir/$program $log < $log > $tmpfile || { log_error "$program_dir/$program $log < $log exit code $?, check $tmpfile"; exit 1; } else -- 2.23.0

2 2

[PATCH compass-ci] create define_files in result_root
by Lu Weitao 23 Nov '20

23 Nov '20

[Why] User need debug their result-parse-script which haven't commit in lkp-tests/stats. [How] We already support user write script as "define_file" => {"stats/$script_name" => "$script_content"} in job.yaml, the job.yaml will be save as job to ES DB, write the job['define_file']['stats/$script_name'](value) at $result_root/stats/$script, extract-service will parse result by $result_root/stats/$script(user's result-parse-script) firstly Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index bcf6e52..ae5d703 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -22,6 +22,8 @@ class StatsWorker if job_id # will consume the job by post-processing job = @es.get_job_content(job_id) result_root = job["result_root"]? + define_files = job["define_files"]? + write_user_stats(result_root.to_s, define_files.as_h) if define_files.is_a?(JSON::Any) result_post_processing(job_id, result_root.to_s, queue_path) @tq.delete_task(queue_path + "/in_process", "#{job_id}") end @@ -139,4 +141,17 @@ class StatsWorker STDERR.puts e.message end end + + def write_user_stats(result_root : String, define_files : Hash(String, JSON::Any)) + return nil unless result_root && File.exists?(result_root) + + define_files.each do |k, v| + if k.match (/^stats/) + Dir.mkdir("#{result_root}/stats") unless File.exists?("#{result_root}/stats") + File.open("#{result_root}/#{k}", "w+", File::Permissions.new(0o775)) do |file| + file.puts v + end + end + end + end end -- 2.23.0

3 4

[PATCH lkp-tests] iso2rootfs: fix the os_version of test job.yaml is not dailybuild
by Yu Chuan 23 Nov '20

23 Nov '20

[Why] The target iso version we want to generate: {iso_os}, {iso_arch}, {iso_version} The target rootfs version we want to test: {iso_os}, {iso_arch}, {iso_version}-dailybuild But now, we test {iso_version} instead of {iso_version}-dailybuild. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 0353c045a4bc..fcaadaa1ba46 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -454,7 +454,7 @@ generate_global_yaml() generate_submit_append_str() { [ "$#" -eq 2 ] || die "generate_submit_append_str: please give 2 parameters." - echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" + echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version}-dailybuild os_mount=$1 testbox=$2" } prepare_submit() -- 2.23.0

1 0

[PATCH v9 compass-ci 1/3] lib: keep suite result in top for compare output
by Lu Kaiyi 23 Nov '20

23 Nov '20

[why] Currently, the compare output result default ordered by alphabet. If there are too many lines in result, it will be difficult for user to find the lines that start with $suite in metric column. So, need reorder compare output result with keeping $suite item on the top of the table. [how] Divide the lines into two parts before printing output according to whether the lines start with $suite, then combine output result into a whole to highlight the important info. [example] lukaiyi@z9 ~% compare "id=z9.148096" "id=z9.148143" in z9 suites_list:["sysbench-mysql"] [before] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 ... 1266042.00 -18.3% 1034206.00 softirqs.RCU 547849.00 -27.4% 397593.00 softirqs.SCHED 159990.00 -80.5% 31181.00 softirqs.TASKLET 1617967.00 -23.0% 1245331.00 softirqs.TIMER 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 2149.23 -100.0% 0.00 time.elapsed_time 2149.23 -100.0% 0.00 time.elapsed_time.max 214458.00 -100.0% 0.00 time.involuntary_context_switches 2.00 -100.0% 0.00 time.major_page_faults 192832.00 -100.0% 0.00 time.maximum_resident_set_size ... [after] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 2894304.00 +12.8% 3265730.00 interrupts.4:GICv3.26.Level.arch_timer 11546998.00 -100.0% 0.00 interrupts.746:ITS-MSI.99092481.Edge.hns3-0000:bd:00.1-TxRx-0 7476064.00 -100.0% 0.00 interrupts.747:ITS-MSI.99092482.Edge.hns3-0000:bd:00.1-TxRx-2 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 16 ++++++++++------ lib/compare_matrixes.rb | 41 ++++++++++++++++++++++++++++------------- lib/matrix2.rb | 20 +++++++++++++++----- 3 files changed, 53 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..f34e239 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suite_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suite_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,16 @@ end def create_matrices_list(conditions) matrices_list = [] + suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suite_list.concat(suites) end - matrices_list + + return matrices_list, suite_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 762c087..14c14cb 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -128,7 +128,7 @@ def set_compare_values(index, values, field, success, options) end def get_values_by_field(matrixes_list, field, matrixes_size, success, options) - # get values by field, values struce example: values[0][:average] + # get values by field, values struct example: values[0][:average] # values = {} matrixes_list.length.times do |index| @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suite_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -808,26 +809,40 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suite_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suite_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suite_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suite_list, matrixes_titles, success, theme) return '' if values.empty? + suite_set = Set.new(suite_list) result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + ranked_str = get_ranked_str(values, suite_set, success, theme) + result_str += ranked_str + result_str +end + +def get_ranked_str(values, suite_set, success, theme) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suite_set.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end - result_str + suite_str + common_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..ff4ea31 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,23 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end - groups.delete(group_key) if value.size < 2 + if value.size < 2 + groups.delete(group_key) + next + end + suites_list << suite_list end - groups + + return groups, suites_list end # input: @@ -139,7 +149,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], _ = create_matrix(jobs) end end -- 2.23.0

2 1

[PATCH compass-ci] result-webdav/start: add share dir to upload file
by Cui Yili 23 Nov '20

23 Nov '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index e0b6676..01141c3 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -14,6 +14,7 @@ cmd=( -p 3080:3080 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf -v /srv/result/:/srv/result/ + -v /srv/initrd/:/srv/initrd/ result-webdav ) -- 2.23.0

1 0

[PATCH v9 compass-ci 3/3] lib: fix group compare no output when dimension number less than 2
by Lu Kaiyi 23 Nov '20

23 Nov '20

[why] When do group compare, if returned groups of auto_group like below (the dimension number less than 2 for group_key): groups = { group_key_1 => { dimension_1 => matrix_1 }, group_key_2 => {} } the group_key will be deleted, then, group compare will have no output. [how] Reduce dimension number limit from 2 to 1. [example] lukaiyi@z9 ~% compare "suite=iperf" -d "os" in z9 [before] <null> [after] os_arch=aarch64/os_version=20/pp.iperf.protocol=tcp/pp.iperf.runtime=300/tbox_group=vm-hi1620-2p8g--liangt openeuler metric -------------------- ------------------------------ fails:runs | 18:18 stderr.EOF' 18:18 stderr.has_stderr openeuler metric -------------------- ------------------------------ %stddev \ 3.496630e+10 ± 19% iperf.tcp.receiver.bps 3.497117e+10 ± 19% iperf.tcp.sender.bps 300.23 iperf.time.elapsed_time 300.23 iperf.time.elapsed_time.max ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 2 +- lib/params_group.rb | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2065039..9753a6f 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,7 +116,7 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| - if value.size < 2 + if value.size < 1 groups.delete(group_key) next end diff --git a/lib/params_group.rb b/lib/params_group.rb index 40191a9..48d651b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -34,8 +34,7 @@ def auto_group(jobs_list, dimensions) return [] if dimensions.empty? jobs_list = extract_jobs_list(jobs_list) - groups = group(jobs_list, dimensions) - return remove_singleton(groups) + group(jobs_list, dimensions) end def extract_jobs_list(jobs_list) -- 2.23.0

2 2

[PATCH lkp-tests] create merge_yaml and submit merge_yaml/*.yaml
by Hu Xuejiao 23 Nov '20

23 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 3 +++ spec/submit/merge_yaml/yaml_merge_included_files.yaml | 1 + spec/submit_spec.rb | 4 ++++ 3 files changed, 8 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..60e3df16 --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,3 @@ +suite: test +category: benchmark +test: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..b56a860c 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH compass-ci 3/3] doc/job/: add initrd_deps.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing initrd_deps field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/initrd_deps.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/job/initrd_deps.md diff --git a/doc/job/initrd_deps.md b/doc/job/initrd_deps.md new file mode 100644 index 0000000..0c3d79c --- /dev/null +++ b/doc/job/initrd_deps.md @@ -0,0 +1,5 @@ +# initrd_deps + +Meaning: +- Most testcases depend on many other package to run. If user has already packed all the other package for running testcase. Then initrd_deps field would give the location of the package. +- The initrd_deps field do not need to be specified by user in the $testcase yaml. It is filled in by the scheduler. User can get the package through the wget command to check the rpm in compressed package. -- 2.23.0

1 0

[PATCH compass-ci 2/3] doc/job/: add initrd_pkg.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing the initrd_pkg field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/initrd_pkg.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/job/initrd_pkg.md diff --git a/doc/job/initrd_pkg.md b/doc/job/initrd_pkg.md new file mode 100644 index 0000000..2ef3d03 --- /dev/null +++ b/doc/job/initrd_pkg.md @@ -0,0 +1,5 @@ +# initrd_pkg + +Meaning: +- Most testcases rely on many other files such as configuration files to run. If user has already packed all the files for running testcase. Then initrd_pkg field would give the location of the packed compression file. +- The initrd_pkg field do not need to be specified by user in the $testcase yaml. It is filled in by the scheduler. User can get the package through the wget command to check the files in compressed package. -- 2.23.0

1 0

[PATCH lkp-tests 2/2] bin/my-submit: this script is used together with the my-qemu
by Xiao Shenwei 23 Nov '20

23 Nov '20

ignore the queue field, it will add default value which is "$tbox_group~$USER" usage: my-submit testbox=vm-2p8g jobs/borrow-1h.yaml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/my-submit | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 sbin/my-submit diff --git a/sbin/my-submit b/sbin/my-submit new file mode 100755 index 00000000..3a7c741e --- /dev/null +++ b/sbin/my-submit @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +submit --my-queue $@ -- 2.23.0

1 1

[PATCH lkp-tests 1/2] sbin/submit: add job into my queue
by Xiao Shenwei 23 Nov '20

23 Nov '20

add --my-queue parameter the job will be add into your own queue usage: submit --my-queue testbox=vm-2p8g ../jobs/borrow-1h.yaml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/submit | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/submit b/sbin/submit index 9bba1018..6af86496 100755 --- a/sbin/submit +++ b/sbin/submit @@ -16,6 +16,7 @@ opt_output_dir = nil opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} +opt_my_queue = false actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -54,6 +55,11 @@ options = OptionParser.new do |opts| ARGV.shift end end + + opts.on('--my-queue', 'add to my queue') do + opt_my_queue = true + end + end options.parse!(ARGV) @@ -93,6 +99,11 @@ def find_jobfile(jobfile) exit 1 end +def add_my_queue(job) + queue = "#{job['tbox_group']}~#{ENV['USER']}" + job['queue'] = queue +end + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -134,6 +145,7 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) + add_my_queue(job_hash) if opt_my_queue # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/job.cr: check the account
by Wu Zhende 23 Nov '20

23 Nov '20

When a job submit, check it's account info. Must key:my_email, my_name, my_uuid Check whether the information is the same as that stored in the ES database Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..05c5462 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "../scheduler/elasticsearch_client" struct JSON::Any def []=(key : String, value : String) @@ -58,8 +59,10 @@ class Job return if @hash["id"] == "#{id}" end + @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") check_required_keys() + check_account_info() set_defaults() end @@ -271,6 +274,9 @@ class Job id suite testbox + my_email + my_uuid + my_name ] private def check_required_keys @@ -281,6 +287,17 @@ class Job end end + private def check_account_info + error_msg = "Failed to verify the account. Please check your configuration" + account_info = @es.get_account(self["my_email"]) + + raise error_msg if account_info["found"]? == false + raise error_msg unless self["my_name"] == account_info["my_name"].to_s + raise error_msg unless self["my_uuid"] == account_info["my_uuid"] + + @hash.delete("my_uuid") + end + private def initialized? initialized_keys = [] of String @@ -302,6 +319,8 @@ class Job "SCHED_HOST", "SCHED_PORT"] + initialized_keys -= ["my_uuid"] + initialized_keys.each do |key| return false unless @hash.has_key?(key) end @@ -441,7 +460,7 @@ class Job else program_version = "latest" end - + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" -- 2.23.0

2 1

[PATCH v3 compass-ci 1/3] git_mirror: fix that README.md in upstream-repos cause a load error
by Li Yuanchao 23 Nov '20

23 Nov '20

[why] As README.md or other docs in upstream-repos, and they are not yaml file. But git-mirror load all files in upstream-repos as yaml file, so there comes out a load error. Now files whose dirname is 'xxx/upstream-repos' will not be loaded. [errmsg] ...in `load_repo_file': undefined method `[]=' for x:Integer (NoMethodError) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9648063..72ca328 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -135,6 +135,8 @@ class MirrorMain traverse_repodir("#{repodir}/#{entry}") end else + return if File.dirname(repodir) == REPO_DIR + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") fork_name = File.basename(repodir) load_repo_file(repodir, project, fork_name) -- 2.23.0

2 1

[PATCH v3 compass-ci 3/3] container/git-mirror: fix that can't get group id of committer in z9
by Li Yuanchao 23 Nov '20

23 Nov '20

In z9, 'committer' is only a group, not a user, so command 'id -g committer' can't get id of committer group Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/start | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/git-mirror/start b/container/git-mirror/start index e1d7d8f..a618ec3 100755 --- a/container/git-mirror/start +++ b/container/git-mirror/start @@ -10,13 +10,13 @@ DOCKER_REPO = '/c/upstream-repos' docker_rm "git-mirror" lkp = %x(id -u lkp) -committer = %x(id -g committer) +group_id = %x(cat /etc/group | grep '^committer' | awk -F ':' '{print $3}') cmd = %W[ docker run --restart=always --name git-mirror - -u #{lkp.to_i}:#{committer.to_i} + -u #{lkp.to_i}:#{group_id.to_i} -d -e CCI_SRC=#{DOCKER_CCI} -e REPO_SRC=#{DOCKER_REPO} -- 2.23.0

2 1

[PATCH lkp-tests 2/2] sbin/my-submit: submit a task to personal queue
by Xiao Shenwei 23 Nov '20

23 Nov '20

usage: my-submit testbox=vm-2p8g jobs/borrow-1h.yaml use this script to submit job, your job will be add to persoanl queue which is $tbox_group~$USER Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/my-submit | 3 +++ 1 file changed, 3 insertions(+) create mode 100755 sbin/my-submit diff --git a/sbin/my-submit b/sbin/my-submit new file mode 100755 index 00000000..57489948 --- /dev/null +++ b/sbin/my-submit @@ -0,0 +1,3 @@ +#!/bin/bash + +submit --my-queue $@ -- 2.23.0

2 2

[PATCH v3 compass-ci 2/3] git_mirror: fix that too few repos in upstream-repos cause memory leak
by Li Yuanchao 21 Nov '20

21 Nov '20

When there are only a few repo files in upstream-repos, that means the same repo will be called very frequently. As the main loop is too fast, it seems the new flow would be called before the last flow of the same repo end completely. That cause a problem. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 72ca328..5b6b889 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,6 +203,7 @@ class MirrorMain push_git_queue handle_feedback Signal.trap(:SIGCHLD, 'SIG_IGN') + sleep(0.1) end end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] scheduler/elasticsearch_client.cr: add function get_account
by Wu Zhende 21 Nov '20

21 Nov '20

Add a new function "get_account" to get account info by "my_email". Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index ec6de0d..6ea8e2d 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -69,6 +69,16 @@ class Elasticsearch::Client return job end + def get_account(my_email : String) + if @client.exists({:index => "accounts", :type => "_doc", :id => my_email}) + response = @client.get_source({:index => "accounts", :type => "_doc", :id => my_email}) + else + response = {"_id" => my_email, "found" => false} + end + + return response + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

1 0

[PATCH lkp-tests] lkp-tests/*: add archlinux install pp support
by Wang Yong 21 Nov '20

21 Nov '20

archlinux use pacman to install pp Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- bin/lkp-setup-rootfs | 1 + lib/bootstrap.sh | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 98c63bae9..0eabea755 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -261,6 +261,7 @@ while true; do fi install_rpms install_deb + install_pkgs echo $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-lkp $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-lkp $job diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index a7776d687..ce34a6329 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -705,3 +705,10 @@ boot_init() mount_rootfs } + +install_pkgs() +{ + [ -d /opt/pkgs ] || return + echo "install $(ls /opt/pkgs/*.pkg.tar*)" + pacman -U --needed --noconfirm /opt/pkgs/*.pkg.tar* +} -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/iozone: explicit support test=all for iozone
by Lu Kaiyi 21 Nov '20

21 Nov '20

[example] iozone-bs.yaml ... suite: iozone category: benchmark file_size: 4g test: all block_size: - 64k ... [output] ... Run began: Sat Nov 21 16:48:22 2020 Record Size 64 kB File size set to 4194304 kB Command line used: iozone -r 64k -s 4g -f /fs/sdb/iozonetest Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 4194304 64 13650392 11529057 7878373 7763936 7166639 10774159 7520609 16188351 7362482 9751567 9453804 7981606 7932187 iozone test complete. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/iozone b/tests/iozone index b6a1e206..bafc0087 100755 --- a/tests/iozone +++ b/tests/iozone @@ -19,6 +19,8 @@ if [ -n "$block_size" ]; then for ele in ${array[@]} do case $ele in + "all") args+=" -a" + ;; "write") args+=" -i 0" ;; "read") args+=" -i 1" -- 2.23.0

1 0

[PATCH lkp-tests] tests/iozone: explicit support test=all for iozone
by Lu Kaiyi 21 Nov '20

21 Nov '20

[example] iozone-bs.yaml ... suite: iozone category: benchmark file_size: 4g test: all block_size: - 64k ... [output] ... Run began: Sat Nov 21 16:10:59 2020 Record Size 64 kB File size set to 4194304 kB Command line used: iozone -r 64k -s 4g -f /fs/sda/iozonetest Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 4194304 64 13566379 14323483 8305849 8185793 7770078 13206535 4562592 15656107 4448821 4868781 4908219 4736366 4720413 iozone test complete. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/iozone b/tests/iozone index b6a1e206..48688ebb 100755 --- a/tests/iozone +++ b/tests/iozone @@ -19,6 +19,8 @@ if [ -n "$block_size" ]; then for ele in ${array[@]} do case $ele in + "all") args+= " -a" + ;; "write") args+=" -i 0" ;; "read") args+=" -i 1" -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] container/git-mirror: fix that can't get group id of committer in z9
by Li Yuanchao 21 Nov '20

21 Nov '20

[why] In z9, 'committer' is only a group, not a user, so command 'id -g committer' can't get id of committer group Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/git-mirror/start b/container/git-mirror/start index e1d7d8f..f5321a1 100755 --- a/container/git-mirror/start +++ b/container/git-mirror/start @@ -10,7 +10,7 @@ DOCKER_REPO = '/c/upstream-repos' docker_rm "git-mirror" lkp = %x(id -u lkp) -committer = %x(id -g committer) +committer = %x(cat /etc/group | grep '^committer' | awk -F ':' '{print $3}') cmd = %W[ docker run -- 2.23.0

3 4

[PATCH v9 compass-ci 2/3] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 21 Nov '20

21 Nov '20

[why] Avoid to execute some unnecessary operations and improve performance. [how] Move the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ff4ea31..2065039 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

1 0