November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] spec/submit: encapsulate function for submit job yaml
by Wei Jihui 20 Nov '20

20 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit_spec.rb | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 983d050e..f365f35b 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -20,20 +20,24 @@ def traverse_file(output_dir) end end +def submit_job(submit_item) + submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) + Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| + output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) + end +end + describe 'submit job spec' do - it 'link jobs spec' do - Dir.glob("#{LKP_SRC}/spec/submit/link_jobs/*.yaml").each do |yaml_file| - job_name = File.basename(yaml_file, '.yaml') - output_dir = "#{LKP_SRC}/spec/submit/link_jobs/#{job_name}" - submit_cmd = [ - "#{LKP_SRC}/sbin/submit", - '-o', output_dir, - '-s', 'lab: spec_lab', - '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file - ] - system(*submit_cmd) - traverse_file(output_dir) - end + it 'link jobs spec' do + submit_job('link_jobs') end end -- 2.23.0

2 1

[PATCH v8 compass-ci 2/2] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 20 Nov '20

20 Nov '20

[why] Avoid to execute some unnecessary operations and improve performance. [how] Keep the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ff4ea31..2065039 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

3 4

[PATCH compass-ci] stats_worker.cr: add blank line after next cmd
by Xu Xijian 20 Nov '20

20 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/extract-stats/stats_worker.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 2af07d4..bcf6e52 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -103,6 +103,7 @@ class StatsWorker next end next if is_exists + new_error_ids << error_id @rc.store_error_info error_id, job_id end @@ -114,6 +115,7 @@ class StatsWorker ERROR_ID_FILES.each do |filename| filepath = File.join(result_root, filename) next unless File.exists?(filepath) + content = File.open(filepath) do |file| JSON.parse(file) end -- 2.23.0

1 0

[PATCH v4 compass-ci] lib/log.rb: add new class for json log
by Wu Zhende 20 Nov '20

20 Nov '20

[Why] The built-in log system prints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example1] use: log.info("test") output: {"level":"INFO","level_num":1,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"test"} [Example2] use: log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"", "job_id":"1","job_state":"submit","caller":["/usr/share/ruby/logger.rb:545:in `error'","./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/log.rb | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..0ffc0f9 --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' +require 'json' + +# log class to print logs in JSON format +class Log < Logger + LEVEL_INFO = { + 'DEBUG' => 0, + 'INFO' => 1, + 'WARN' => 2, + 'ERROR' => 3, + 'FATAL' => 4, + 'UNKNOWN' => 5 + }.freeze + + FORMATTER = proc { |severity, datetime, progname, msg| + puts + level_num = LEVEL_INFO[severity] + begin + message = JSON.parse(msg) + rescue JSON::ParserError + message = { 'message' => msg } + end + h = { + 'level' => severity.to_s, + 'level_num' => level_num, + 'datetime' => datetime, + 'progname' => progname, + 'message' => '' + } + h.merge!(message) + h.merge!({ 'caller' => caller }) if level_num >= 2 + h.to_json + } + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

3 2

[PATCH v2 lkp-tests] common.sh: add function for count input cpu number
by Wei Jihui 20 Nov '20

20 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/common.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/common.sh b/lib/common.sh index 2a93aafa..95d7953c 100755 --- a/lib/common.sh +++ b/lib/common.sh @@ -85,6 +85,18 @@ cpu_list_ref() echo $cpu_list | cut -d ' ' -f $((n+1)) } + +# input: 1-3 output: 3 +# input: 1,3 output: 2 +cpu_list_num() +{ + cpu_list="" + for cpu in $(expand_cpu_list $1); do + cpu_list+="$cpu " + done + cpu_list_count "$cpu_list" +} + # if str starts with prefix, output remaining part, otherwise output empty string remove_prefix() { -- 2.23.0

1 0

[PATCH lkp-tests] sshd: fix error when execute in docker
by Zhang Yale 20 Nov '20

20 Nov '20

error: bash: 1605766611473034209%""+"": syntax error: operand expected (error token is """+""") Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daemon/sshd b/daemon/sshd index 1a1d59b4..a3fdf954 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -77,7 +77,7 @@ set_port() for i in $(seq 1 10) do - port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) + port=$(($(date +%s%N)%${sshr_port_len}+${sshr_port_base})) ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null [ $? -eq 0 ] && return -- 2.23.0

2 2

[PATCH lkp-tests] stats/netperf: fix rubocop offenses
by Zhang Yale 20 Nov '20

20 Nov '20

[Why] Inspecting 1 file C Offenses: netperf:51:29: C: Style/NumericPredicate: Use iterations.positive? instead of iterations > 0. throughtput /= iteration if iterations > 0 ^^^^^^^^^^^^^^ 1 file inspectef, 1 offense detected Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/netperf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/netperf b/stats/netperf index 7965b1ed..80b07ae5 100755 --- a/stats/netperf +++ b/stats/netperf @@ -48,7 +48,7 @@ avg_throughput = throughput / num # We only cares for the average total throughput/workload # within each iteration. -throughput /= iterations if iterations > 0 +throughput /= iterations if iterations.positive? workload = throughput * time workload = workload * 10**6 / 8.0 / pkt_size if unit == 'Mbps' -- 2.23.0

2 2

[PATCH v2 lkp-tests] create merge_yaml and submit merge_yaml/*.yaml
by Hu Xuejiao 20 Nov '20

20 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories [how] Create merge_yaml and submit merge_yaml/*.yaml Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit_spec.rb | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 983d050e..5c72447a 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -34,6 +34,22 @@ describe 'submit job spec' do ] system(*submit_cmd) traverse_file(output_dir) + end + end + + it 'merge yaml spec' do + Dir.glob("#{LKP_SRC}/spec/submit/merge_yaml/*.yaml").each do |yaml_file| + job_name = File.basename(yaml_file, '.yaml') + output_dir = "#{LKP_SRC}/spec/submit/merge_yaml/#{job_name}/output" + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) + end end - end end -- 2.23.0

2 1

[PATCH v8 compass-ci] container/initrd-cifs: add the container to share /srv/initrd
by Xiao Shenwei 20 Nov '20

20 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result in order to reduce the coupling, need split them [how] one folder mount by one container [usage] two ways for the client to mount initrd: 1. initrd mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage: for lkp compatibility mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 22 ++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 89 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..c2ec808 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +RUN adduser -u 1090 -D lkp +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash +RUN rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..4b93647 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..77558a1 --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

2 2

[PATCH v2 lkp-tests] add the case of yaml_merge_included_files.yaml
by Hu Xuejiao 20 Nov '20

20 Nov '20

[description] The content of template.yaml will be import in yaml_merge_included_files Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 3 +++ spec/submit/merge_yaml/yaml_merge_included_files.yaml | 1 + 2 files changed, 4 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..0415606a --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,3 @@ +suite: test +category: functional +test: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml -- 2.23.0

1 0