- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] stats/netperf: fix rubocop offenses
by Zhang Yale 20 Nov '20

20 Nov '20

[Why] Inspecting 1 file C Offenses: netperf:51:29: C: Style/NumericPredicate: Use iterations.positive? instead of iterations > 0. throughtput /= iteration if iterations > 0 ^^^^^^^^^^^^^^ 1 file inspectef, 1 offense detected Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/netperf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/netperf b/stats/netperf index 7965b1ed..80b07ae5 100755 --- a/stats/netperf +++ b/stats/netperf @@ -48,7 +48,7 @@ avg_throughput = throughput / num # We only cares for the average total throughput/workload # within each iteration. -throughput /= iterations if iterations > 0 +throughput /= iterations if iterations.positive? workload = throughput * time workload = workload * 10**6 / 8.0 / pkt_size if unit == 'Mbps' -- 2.23.0

2 2

[PATCH v2 lkp-tests] create merge_yaml and submit merge_yaml/*.yaml
by Hu Xuejiao 20 Nov '20

20 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories [how] Create merge_yaml and submit merge_yaml/*.yaml Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit_spec.rb | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 983d050e..5c72447a 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -34,6 +34,22 @@ describe 'submit job spec' do ] system(*submit_cmd) traverse_file(output_dir) + end + end + + it 'merge yaml spec' do + Dir.glob("#{LKP_SRC}/spec/submit/merge_yaml/*.yaml").each do |yaml_file| + job_name = File.basename(yaml_file, '.yaml') + output_dir = "#{LKP_SRC}/spec/submit/merge_yaml/#{job_name}/output" + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) + end end - end end -- 2.23.0

2 1

[PATCH v8 compass-ci] container/initrd-cifs: add the container to share /srv/initrd
by Xiao Shenwei 20 Nov '20

20 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result in order to reduce the coupling, need split them [how] one folder mount by one container [usage] two ways for the client to mount initrd: 1. initrd mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage: for lkp compatibility mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 22 ++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 89 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..c2ec808 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +RUN adduser -u 1090 -D lkp +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash +RUN rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..4b93647 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..77558a1 --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

2 2

[PATCH v2 lkp-tests] add the case of yaml_merge_included_files.yaml
by Hu Xuejiao 20 Nov '20

20 Nov '20

[description] The content of template.yaml will be import in yaml_merge_included_files Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 3 +++ spec/submit/merge_yaml/yaml_merge_included_files.yaml | 1 + 2 files changed, 4 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..0415606a --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,3 @@ +suite: test +category: functional +test: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml -- 2.23.0

1 0

[PATCH v8 compass-ci 1/2] lib: keep suite result in top for compare output
by Lu Kaiyi 20 Nov '20

20 Nov '20

[why] Currently, the compare output result default ordered by alphabet. If there are too many lines in result, it will be difficult for user to find the lines that start with $suite in metric column. So, need reorder compare output result with keeping $suite item on the top of the table. [how] Divide the lines into two parts before printing output according to whether the lines start with $suite, then combine output result into a whole to highlight the important info. [example] lukaiyi@z9 ~% compare "id=z9.148096" "id=z9.148143" in z9 suites_list:["sysbench-mysql"] [before] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 ... 1266042.00 -18.3% 1034206.00 softirqs.RCU 547849.00 -27.4% 397593.00 softirqs.SCHED 159990.00 -80.5% 31181.00 softirqs.TASKLET 1617967.00 -23.0% 1245331.00 softirqs.TIMER 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 2149.23 -100.0% 0.00 time.elapsed_time 2149.23 -100.0% 0.00 time.elapsed_time.max 214458.00 -100.0% 0.00 time.involuntary_context_switches 2.00 -100.0% 0.00 time.major_page_faults 192832.00 -100.0% 0.00 time.maximum_resident_set_size ... [after] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 2894304.00 +12.8% 3265730.00 interrupts.4:GICv3.26.Level.arch_timer 11546998.00 -100.0% 0.00 interrupts.746:ITS-MSI.99092481.Edge.hns3-0000:bd:00.1-TxRx-0 7476064.00 -100.0% 0.00 interrupts.747:ITS-MSI.99092482.Edge.hns3-0000:bd:00.1-TxRx-2 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 16 ++++++++++------ lib/compare_matrixes.rb | 41 ++++++++++++++++++++++++++++------------- lib/matrix2.rb | 20 +++++++++++++++----- 3 files changed, 53 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..f34e239 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suite_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suite_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,16 @@ end def create_matrices_list(conditions) matrices_list = [] + suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suite_list.concat(suites) end - matrices_list + + return matrices_list, suite_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..5039b01 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -128,7 +128,7 @@ def set_compare_values(index, values, field, success, options) end def get_values_by_field(matrixes_list, field, matrixes_size, success, options) - # get values by field, values struce example: values[0][:average] + # get values by field, values struct example: values[0][:average] # values = {} matrixes_list.length.times do |index| @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suite_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -807,26 +808,40 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suite_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suite_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suite_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suite_list, matrixes_titles, success, theme) return '' if values.empty? + suite_set = Set.new(suite_list) result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + ranked_str = get_ranked_str(values, suite_set, success, theme) + result_str += ranked_str + result_str +end + +def get_ranked_str(values, suite_set, success, theme) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suite_set.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end - result_str + suite_str + common_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..ff4ea31 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,23 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end - groups.delete(group_key) if value.size < 2 + if value.size < 2 + groups.delete(group_key) + next + end + suites_list << suite_list end - groups + + return groups, suites_list end # input: @@ -139,7 +149,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], _ = create_matrix(jobs) end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 4/6] iso2rootfs: delete the useless fields
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] The 'os', 'os_arch', 'os_version' fields to be use by test job are the same as the 'iso_os', 'iso_arch', 'iso_version' of the destination iso this time. So delete the 'test_os', 'test_os_arch', 'test_os_version'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 3 --- tests/iso2rootfs | 33 +++++++-------------------------- 2 files changed, 7 insertions(+), 29 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 55bf71517174..222ac309d33e 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -22,8 +22,5 @@ iso2rootfs: # submit test yaml related fields test_yaml: - test_os: - test_os_arch: - test_os_version: test_os_mount: test_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index c2e823c93ff9..4f6e3e578ce4 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -447,17 +447,8 @@ generate_global_yaml() generate_submit_append_str() { - local wait_add_vars=( - "os" - "os_arch" - "os_version" - "os_mount" - "testbox" - ) - for vt in "${wait_add_vars[@]}" - do - SUBMIT_APPEND_STR="${SUBMIT_APPEND_STR} ${vt}=$(eval echo "\$test_${vt}")" - done + [ "$#" -eq 2 ] || die "generate_submit_append_str: please give 2 parameters." + echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" } submit_test_job_yaml() @@ -468,18 +459,8 @@ submit_test_job_yaml() [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." - local test_yaml_vars=( - "test_os" - "test_os_arch" - "test_os_version" - "test_os_mount" - "test_testbox" - ) - - for yaml_t in "${test_yaml_vars[@]}" - do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." - done + [ -n "${test_os_mount}" ] || die "cannot fine value of var: test_os_mount." + [ -n "${test_testbox}" ] || die "cannot fine value of var: test_testbox." cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh @@ -493,9 +474,9 @@ submit_test_job_yaml() die "cannot find test yaml in LKP_SRC/jobs. test yaml: ${test_yaml}." - generate_submit_append_str - - "${LKP_SRC}/sbin/submit" ${SUBMIT_APPEND_STR} "${test_yaml}" + "${LKP_SRC}/sbin/submit" \ + $(generate_submit_append_str "${test_os_mount}" "${test_testbox}") \ + "${test_yaml}" } log_info "submit ${test_yaml} finished" -- 2.23.0

2 2

[PATCH v2 lkp-tests 2/6] iso2rootfs: allow non-dailybuild jobs to be submitted
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] 1. Decouple the code related to dailybuild. The 'iso_url_file' field is only relate to the dailybuild function, and this field is unnecessary when user only want generate a rootfs from offical distribution. But now if this field is none, this job will failed. 2. Modify the field name. 'iso_url_file' -> 'dailybuild_iso_url_file'. [Addition] if the `iso_url_file` field exists, this job will generate a soft link which point to the result rootfs. - soft link name: {iso_version}-dailybuild. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 4 +++- tests/iso2rootfs | 41 +++++++++++++++++++++++++---------------- 2 files changed, 28 insertions(+), 17 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index d4f0ffc0e80c..8796874ce79d 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -2,7 +2,6 @@ suite: iso2rootfs category: benchmark iso2rootfs: iso2rootfs_cgz: - iso_url_file: iso_os: iso_arch: iso_version: @@ -12,6 +11,9 @@ iso2rootfs: initrd_http_host: initrd_http_port: + # dailybuild related fields + dailybuild_iso_url_file: + # submit test yaml related fields test_yaml: test_os: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 9c71b8427d90..5f85cc9db01e 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -44,7 +44,6 @@ check_yaml_vars() "os_arch" "os_version" "iso2rootfs_cgz" - "iso_url_file" "iso_os" "iso_arch" "iso_version" @@ -105,10 +104,10 @@ check_iso_name() get_daily_iso_checksum() { - ISO_URL="$(curl "${iso_url_file}")" + ISO_URL="$(curl "${dailybuild_iso_url_file}")" [ "${iso_os}" == "openeuler" ] && { - local pub_ip=$(echo "${iso_url_file}" | grep -oEw "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}") - ISO_URL="$(curl "${iso_url_file}" |sed -r "s/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/${pub_ip}/g")" + local pub_ip=$(echo "${dailybuild_iso_url_file}" | grep -oEw "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}") + ISO_URL="$(curl "${dailybuild_iso_url_file}" |sed -r "s/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/${pub_ip}/g")" } ISO_NAME=$(basename "$ISO_URL") @@ -149,15 +148,15 @@ mount_rootfs() get_cache_iso_checksum() { - mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-latest.sha256sum" - [ ! -f "${CHECKSUM_FILE_CACHE}" ] || - SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") + [ -f "${CHECKSUM_FILE_CACHE}" ] || return 0 + SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } check_sha256sum_update() { + [ -n "${dailybuild_iso_url_file}" ] || return 0 + get_cache_iso_checksum get_daily_iso_checksum @@ -243,7 +242,10 @@ config_iso2rootfs() pre_works() { check_yaml_vars + + mount_rootfs check_sha256sum_update + check_os config_iso2rootfs @@ -274,9 +276,10 @@ config_iso_conf() config_iso_url() { - local iso_url_file="${I2Q_SRC}/conf/iso/net-iso/${iso_os}/${iso_arch}/${iso_version}" - sed -i "s|^iso_url=.*|iso_url=\"${ISO_URL}\"|g" "${iso_url_file}" - sed -i "s|^iso_checksum_url=.*|iso_checksum_url=\"${ISO_CHECKSUM_URL}\"|g" "${iso_url_file}" + [ -n "${dailybuild_iso_url_file}" ] || return 0 + local i2q_iso_url_file="${I2Q_SRC}/conf/iso/net-iso/${iso_os}/${iso_arch}/${iso_version}" + sed -i "s|^iso_url=.*|iso_url=\"${ISO_URL}\"|g" "${i2q_iso_url_file}" + sed -i "s|^iso_checksum_url=.*|iso_checksum_url=\"${ISO_CHECKSUM_URL}\"|g" "${i2q_iso_url_file}" } config_iso2qcow2() @@ -332,7 +335,7 @@ config_qcow2rootfs() run_qcow2rootfs() { log_info "starting run qcow2rootfs ..." - local qcow2_path="${I2Q_SRC}/output/${ISO_NAME%.iso}.qcow2" + local qcow2_path="$(ls ${I2Q_SRC}/output/*.qcow2)" cd "${CCI_SRC}/container/qcow2rootfs" ./run "${qcow2_path}" "${ROOTFS_DES_DIR}" @@ -497,16 +500,22 @@ submit_test_job_yaml() } ############ post works ############ -post_works() +update_dailybuild_soft_link() { - log_info "starting post works ..." + [ -n "${dailybuild_iso_url_file}" ] || return 0 mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && - rm -f "${soft_link}" && - ln -s "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" + ln -sfT "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" +} + +post_works() +{ + log_info "starting post works ..." + + update_dailybuild_soft_link cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/6] iso2rootfs: change how to get the kernel and initrd.lkp of rootfs
by Yu Chuan 20 Nov '20

20 Nov '20

[Why] We use the container in compass-ci repo to generate rootfs from a qcow2, it will generate the follow two soft links: - {rootfs_dir}/boot/vmlinuz # for vmlinuz file - {rootfs_dir}/initrd.lkp # for initrd file So we can use the below two soft links to get the absolute path of vmlinuz and initrd. [Addition] URL of qcow2rootfs: - https://gitee.com/wu_fengguang/compass-ci/tree/master/container/qcow2rootfs Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 05274164f8c1..9c71b8427d90 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -375,8 +375,12 @@ test_rootfs() log_info "starting test rootfs ..." local test_script="${CGZ_PATH}/kvm.sh" - local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz*|grep -v rescue) - local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/boot/initramfs.lkp*) + local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz) + local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/initrd.lkp) + + [ -f "$kernel_path" ] || die "cannot find kernel of rootfs: $kernel_path" + [ -f "$initrd_lkp_path" ] || die "cannot find initrd_lkp of rootfs: $initrd_lkp_path" + local root_path case ${rootfs_protocol} in "nfs") -- 2.23.0

2 2

[PATCH lkp-tests] common.sh: a new function for count input cpu number
by Wei Jihui 20 Nov '20

20 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/common.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/common.sh b/lib/common.sh index 2a93aafa..b1ca1b13 100755 --- a/lib/common.sh +++ b/lib/common.sh @@ -85,6 +85,18 @@ cpu_list_ref() echo $cpu_list | cut -d ' ' -f $((n+1)) } + +# input: 1-3 output: 3 +# input: 1,3 output: 2 +cpu_list_num() +{ + cpu_list="" + for c in $(expand_cpu_list $1); do + cpu_list+="$c " + done + cpu_list_count "$cpu_list" +} + # if str starts with prefix, output remaining part, otherwise output empty string remove_prefix() { -- 2.23.0

2 1

[PATCH v3 compass-ci] lib/log.rb: add new class for json log
by Wu Zhende 19 Nov '20

19 Nov '20

[Why] The built-in log system prints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example] require './log' log = Log.new log.info("test") output: {"level":"INFO","level_num":1,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"test"} log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"", "job_id":"1","job_state":"submit","caller":["/usr/share/ruby/logger.rb:4:in `format_message'", "/usr/share/ruby/logger.rb:472:in`add'","/usr/share/ruby/logger.rb:545:in `error'","./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/log.rb | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..0ffc0f9 --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' +require 'json' + +# log class to print logs in JSON format +class Log < Logger + LEVEL_INFO = { + 'DEBUG' => 0, + 'INFO' => 1, + 'WARN' => 2, + 'ERROR' => 3, + 'FATAL' => 4, + 'UNKNOWN' => 5 + }.freeze + + FORMATTER = proc { |severity, datetime, progname, msg| + puts + level_num = LEVEL_INFO[severity] + begin + message = JSON.parse(msg) + rescue JSON::ParserError + message = { 'message' => msg } + end + h = { + 'level' => severity.to_s, + 'level_num' => level_num, + 'datetime' => datetime, + 'progname' => progname, + 'message' => '' + } + h.merge!(message) + h.merge!({ 'caller' => caller }) if level_num >= 2 + h.to_json + } + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

2 2