- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] fix: idle job for subqueue: "idle" is a value of subqueue not a field
by Ren Wen 17 Dec '20

17 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c322d0f..c09d145 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -114,13 +114,6 @@ class Sched def submit_single_job(job) queue = "#{job.queue}/#{job.subqueue}" - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - job_id = add_task(queue, job.lab) return [{ "job_id" => "0", -- 2.23.0

1 0

[PATCH compass-ci] fix: idle job for subqueue: subqueue can be "idle"
by Ren Wen 17 Dec '20

17 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1d44559..9fc3bf7 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -274,7 +274,7 @@ class Job end private def set_subqueue - self["subqueue"] = self["my_email"] + self["subqueue"] = self["my_email"] unless self["subqueue"] == "idle" end # if not assign tbox_group, set it to a match result from testbox -- 2.23.0

1 0

[PATCH compass-ci] lib: optimize es query_by_id to adapt to change the es mapping
by Cao Xueliang 17 Dec '20

17 Dec '20

When we change es mapping, we can't use es.get_source, because the jobs is a alias for new jobs index, the old data at the old index. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/es_query.rb | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 68c70b5..3bea624 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -54,9 +54,12 @@ class ESQuery end def query_by_id(id) - @client.get_source({ index: @index, type: '_doc', id: id }) - rescue Elasticsearch::Transport::Transport::Errors::NotFound - nil + result = @client.search(index: @index + '*', + body: { query: { bool: { must: { term: { '_id' => id } } } }, + size: 1 })['hits']['hits'] + return nil unless result.size == 1 + + return result[0]['_source'] end end -- 2.23.0

1 0

[PATCH compass-ci] doc: and runtime field in borrow-machine.en.md and borrow-machine.zh.md
by Zhang Yale 17 Dec '20

17 Dec '20

User can specify the sleep time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 11 ++++++----- doc/manual/borrow-machine.zh.md | 11 ++++++----- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index b9c2bc4..0277984 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -202,7 +202,8 @@ Ensure that you have performed the following operations according to the [apply- hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 06d4db6..5bc951d 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -206,7 +206,8 @@ hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom - sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + runtime: 1h + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml + runtime: 1h + # ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends: add sshd depends file
by Wang Yong 17 Dec '20

17 Dec '20

add depends file for sshd Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/sshd | 1 + 1 file changed, 1 insertion(+) create mode 100644 distro/depends/sshd diff --git a/distro/depends/sshd b/distro/depends/sshd new file mode 100644 index 000000000..dd21b5aa9 --- /dev/null +++ b/distro/depends/sshd @@ -0,0 +1 @@ +openssh-server -- 2.23.0

2 2

[PATCH v2 compass-ci] scheduler: submit cluster job to "queue/subqueue"
by Ren Wen 17 Dec '20

17 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 4303b1d..c322d0f 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -37,6 +37,7 @@ class Sched def submit_cluster_job(job, cluster_config) job_messages = Array(Hash(String, String)).new lab = job.lab + subqueue = job.subqueue # collect all job ids job_ids = [] of String @@ -52,7 +53,7 @@ class Sched # steps for each host cluster_config["nodes"].as_h.each do |host, config| queue = host.to_s - job_id = add_task(queue, lab) + job_id = add_task("#{queue}/#{subqueue}", lab) # return when job_id is '0' # 2 Questions: -- 2.23.0

1 0

[PATCH lkp-tests] jobs: add runtime field in borrow-*.yaml
by Zhang Yale 17 Dec '20

17 Dec '20

[why] 1. We need "$runtime" field to determine "Due time" value in email template received by users. 2. And the "$runtime" field is used in sleep script too. Therefore, sleep can be left blank, and pp.sleep can be parse successfully now. 3. User can specify the sleep time in "runtime" field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-10d.yaml | 3 ++- jobs/borrow-1d.yaml | 3 ++- jobs/borrow-1h.yaml | 3 ++- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/jobs/borrow-10d.yaml b/jobs/borrow-10d.yaml index e0e35e17..88a137e0 100644 --- a/jobs/borrow-10d.yaml +++ b/jobs/borrow-10d.yaml @@ -1,6 +1,7 @@ suite: borrow testcase: borrow +runtime: 10d ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -10,4 +11,4 @@ ssh_pub_key: <%= %> sshd: # sleep at the bottom -sleep: 10d +sleep: diff --git a/jobs/borrow-1d.yaml b/jobs/borrow-1d.yaml index c31d7ebd..f79b440b 100644 --- a/jobs/borrow-1d.yaml +++ b/jobs/borrow-1d.yaml @@ -3,6 +3,7 @@ testcase: borrow timeout: 1d +runtime: 1d ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -12,4 +13,4 @@ ssh_pub_key: <%= %> sshd: # sleep at the bottom -sleep: 1d +sleep: diff --git a/jobs/borrow-1h.yaml b/jobs/borrow-1h.yaml index d6b72c43..d13af910 100644 --- a/jobs/borrow-1h.yaml +++ b/jobs/borrow-1h.yaml @@ -1,6 +1,7 @@ suite: borrow testcase: borrow +runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -10,4 +11,4 @@ ssh_pub_key: <%= %> sshd: # sleep at the bottom -sleep: 1h +sleep: -- 2.23.0

2 2

[PATCH compass-ci] result-webdav/nginx.conf: error 500 at the second request
by Cui Yili 17 Dec '20

17 Dec '20

[why] the table in the function split is used as a variable elsewhere. error: Attempt to call field 'insert' (a nil value) [how] table => dirs Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index af27653..8b54314 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -61,13 +61,13 @@ http { ngx.exit(ngx.HTTP_FORBIDDEN) end - if string.find(res, ",") then - table = split(res,",") - for i = 1, #table do - if string.find(path, table[i]) then + if string.find(res, ",", 1, true) then + dirs = split(res, ",") + for i = 1, #dirs do + if string.find(path, dirs[i], 1, true) then return end - end + end else if string.find(path, res, 1, true) then return -- 2.23.0

2 1

[PATCH v1 compass-ci] delimiter: optimize delimiter utils function
by Cao Xueliang 17 Dec '20

17 Dec '20

1. Delete redundant bad job fields. 2. Add to make /tmp/lkp/result dir. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index b60ada0..e87bbbe 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -108,8 +108,11 @@ module Utils job.delete('error_ids') job.delete('start_time') + job.delete('job_state') job.delete('end_time') job.delete('loadavg') + job.delete('stats') + job.delete('id') return job end @@ -123,6 +126,7 @@ module Utils end def create_bisect_log(git_dir) + FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT log_file = File.join(TMP_RESULT_ROOT, "bisect.log") log_content = parse_bisect_log(git_dir) File.open(log_file, 'w') do |f| -- 2.23.0

1 0

[PATCH compass-ci] delimiter: optimize delimiter utils function
by Cao Xueliang 17 Dec '20

17 Dec '20

1. Delete redundant bad job fields. 2. Add to make /tmp/lkp/result dir. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index b60ada0..e87bbbe 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -108,8 +108,11 @@ module Utils job.delete('error_ids') job.delete('start_time') + job.delete('job_state') job.delete('end_time') job.delete('loadavg') + job.delete('stats') + job.delete('id') return job end @@ -123,6 +126,7 @@ module Utils end def create_bisect_log(git_dir) + FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT log_file = File.join(TMP_RESULT_ROOT, "bisect.log") log_content = parse_bisect_log(git_dir) File.open(log_file, 'w') do |f| -- 2.23.0

1 1

[PATCH v2 lkp-tests] tests: fix error when running host-info in vm
by Zhang Yu 17 Dec '20

17 Dec '20

Error: Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tests/host-info b/tests/host-info index 0ab8e6c97..d1d0df449 100755 --- a/tests/host-info +++ b/tests/host-info @@ -102,11 +102,9 @@ get_ipmi_ip() # 802.1q VLAN ID : Disabled # RMCP+ Cipher Suites : 0,1,2,3,17 # Cipher Suite Priv Max : XuuaXXXXXXXXXXX - is_virt && return - is_docker && return + [ -e /dev/ipmi0 ] || [ -e /dev/ipmi/0 ] || [ -e /dev/ipmidev/0 ] && return local ip=$(ipmitool lan print | awk 'NR==3{print $4}') [ -n "$ip" ] && echo "ipmi_ip: $ip" - } main() -- 2.23.0

2 2

[PATCH v1 compass-ci] delimiter: error id use inspect to avoid special characters
by Cao Xueliang 17 Dec '20

17 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- src/delimiter/delimiter.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 97ad467..bc1e035 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -79,7 +79,7 @@ class GitBisect result = `git -C #{@work_dir} bisect start #{@bad_commit} #{@good_commit}` temp = result.split(/\n/) if temp[0].include? 'Bisecting' - result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} "#{@error_id}" #{@work_dir}` + result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} #{(a)error_id.inspect} #{@work_dir}` end Utils.create_bisect_log(@work_dir) FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index c1922e2..bf36db3 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -25,7 +25,7 @@ class Delimiter next end - %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id']} bisect.yaml) + %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id'].inspect} bisect.yaml) rescue StandardError => e puts e sleep(60) -- 2.23.0

1 0

[PATCH compass-ci] delimiter: error id use inspect to avoid special characters
by Cao Xueliang 17 Dec '20

17 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- src/delimiter/delimiter.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 97ad467..bc1e035 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -79,7 +79,7 @@ class GitBisect result = `git -C #{@work_dir} bisect start #{@bad_commit} #{@good_commit}` temp = result.split(/\n/) if temp[0].include? 'Bisecting' - result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} "#{@error_id}" #{@work_dir}` + result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} #{(a)error_id.inspect} #{@work_dir}` end Utils.create_bisect_log(@work_dir) FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index c1922e2..cb2ed6c 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -25,7 +25,7 @@ class Delimiter next end - %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id']} bisect.yaml) + %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=\'#{task['error_id']}\' bisect.yaml) rescue StandardError => e puts e sleep(60) -- 2.23.0

1 1

Re: [PATCH lkp-tests] daemon: add the local execution permission
by Bai Jing 17 Dec '20

17 Dec '20

title can be : add the execute permission Thanks, Baijing Reply-To: In-Reply-To: <20201217061232.1480336-1-1034502035(a)qq.com> On Thu, Dec 17, 2020 at 02:12:32PM +0800, Hu Xuejiao wrote: >Signed-off-by: Hu XueJiao <1034502035(a)qq.com> >--- > daemon/multi-docker | 2 +- > daemon/multi-qemu | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > >diff --git a/daemon/multi-docker b/daemon/multi-docker >index b39506b6..257645b8 100755 >--- a/daemon/multi-docker >+++ b/daemon/multi-docker >@@ -8,7 +8,7 @@ > > multi_docker() > { >- export CCI_SRC=/c/compass-ci >+ export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} > local hostname=$tbox_group.$HOSTNAME > $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues > } >diff --git a/daemon/multi-qemu b/daemon/multi-qemu >index aa2348b4..55b96c8e 100755 >--- a/daemon/multi-qemu >+++ b/daemon/multi-qemu >@@ -10,7 +10,7 @@ > > multi_qemu() > { >- export CCI_SRC=/c/compass-ci >+ export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} > local hostname=$tbox_group.$HOSTNAME > > cd "$CACHE_DIR" >-- >2.23.0 >

1 0

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: update pp.sleep mapping
by Lu Weitao 17 Dec '20

17 Dec '20

[Why] bellow doc will be store to ES { ... "pp":{ "sleep": 3600 } } or { .. "pp": { "sleep": {"runtime": 3600} } } both type doc will cause mapper-parsing-exceptions Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-jobs-mapping.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 5cb9b9c..b43625e 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -79,6 +79,10 @@ else "dynamic": true, "properties": {} }, + "pp.sleep":{ + "type": "object", + "enabled": false + }, "suite": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-jobs-mapping.sh: update
by Lu Weitao 17 Dec '20

17 Dec '20

[Why] bellow doc will be store to ES { ... "pp":{ "sleep": 3600 } } or { .. "pp": { "sleep": {"runtime": 3600} } } both type doc will cause mapper-parsing-exceptions Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-jobs-mapping.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 5cb9b9c..5713cbf 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -79,6 +79,10 @@ else "dynamic": true, "properties": {} }, + "pp.sleep":{ + "type": "object", + "enabled": false + }, "suite": { "type": "keyword" }, -- 2.23.0

3 4

[PATCH v2 lkp-tests] daemon: add the local execution permission
by Hu Xuejiao 17 Dec '20

17 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-docker | 2 +- daemon/multi-qemu | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/daemon/multi-docker b/daemon/multi-docker index b39506b6..257645b8 100755 --- a/daemon/multi-docker +++ b/daemon/multi-docker @@ -8,7 +8,7 @@ multi_docker() { - export CCI_SRC=/c/compass-ci + export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} local hostname=$tbox_group.$HOSTNAME $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues } diff --git a/daemon/multi-qemu b/daemon/multi-qemu index aa2348b4..55b96c8e 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -10,7 +10,7 @@ multi_qemu() { - export CCI_SRC=/c/compass-ci + export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} local hostname=$tbox_group.$HOSTNAME cd "$CACHE_DIR" -- 2.23.0

1 0

[PATCH lkp-tests] daemon: add the local execution permission
by Hu Xuejiao 17 Dec '20

17 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-docker | 2 +- daemon/multi-qemu | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/daemon/multi-docker b/daemon/multi-docker index b39506b6..257645b8 100755 --- a/daemon/multi-docker +++ b/daemon/multi-docker @@ -8,7 +8,7 @@ multi_docker() { - export CCI_SRC=/c/compass-ci + export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} local hostname=$tbox_group.$HOSTNAME $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues } diff --git a/daemon/multi-qemu b/daemon/multi-qemu index aa2348b4..55b96c8e 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -10,7 +10,7 @@ multi_qemu() { - export CCI_SRC=/c/compass-ci + export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} local hostname=$tbox_group.$HOSTNAME cd "$CACHE_DIR" -- 2.23.0

1 0

[PATCH v5 compass-ci 6/6] fetch-mail: start file
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/start | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/fetch-mail/start diff --git a/container/fetch-mail/start b/container/fetch-mail/start new file mode 100755 index 0000000..98cbd8f --- /dev/null +++ b/container/fetch-mail/start @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'fetch-mail' + +names = Set.new %w[ + FETCH_MAIL_ADDRESS + FETCH_MAIL_AUTH_CODE + FETCH_MAIL_DIR + FETCH_MAIL_BOX +] + +defaults = relevant_defaults(names) + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + -u mailer + --name=fetch-mail + -d +] + env + %W[ + -v #{defaults['FETCH_MAIL_DIR']}:#{defaults['FETCH_MAIL_DIR']}:rw + -w /home/mailer/ + fetch-mail +] + +cmd += ['sh', '-c', './run.sh'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v5 compass-ci 5/6] fetch-mail: build file
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/fetch-mail/build diff --git a/container/fetch-mail/build b/container/fetch-mail/build new file mode 100755 index 0000000..e72bed8 --- /dev/null +++ b/container/fetch-mail/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t fetch-mail:latest . -- 2.23.0

1 0

[PATCH v5 compass-ci 4/6] fetch-mail: Dockerfile
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/Dockerfile | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 container/fetch-mail/Dockerfile diff --git a/container/fetch-mail/Dockerfile b/container/fetch-mail/Dockerfile new file mode 100644 index 0000000..0434e09 --- /dev/null +++ b/container/fetch-mail/Dockerfile @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM centos:latest +MAINTAINER Luan Shengde <shdluan(a)163.com> +ENV DEBIAN_FRONTEND noninteractive + +RUN useradd -u 1144 -g 1147 mailer +RUN yum update -y +RUN yum install -y procmail fetchmail + +COPY fetchmailrc.template /home/mailer/.fetchmailrc +COPY procmailrc.template /home/mailer/.procmailrc +COPY run.sh /home/mailer/ + +RUN chmod 600 /home/mailer/.fetchmailrc +RUN chmod 600 /home/mailer/.procmailrc +RUN chown -R mailer:mailer /home/mailer +RUN chmod +x /home/mailer/run.sh -- 2.23.0

1 0

[PATCH v5 compass-ci 3/6] fetch-mail: runner for fetch-mail service
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/run.sh | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 container/fetch-mail/run.sh diff --git a/container/fetch-mail/run.sh b/container/fetch-mail/run.sh new file mode 100755 index 0000000..c79b054 --- /dev/null +++ b/container/fetch-mail/run.sh @@ -0,0 +1,14 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +mail_conf () +{ + sed -i "s|FETCH_MAIL_ADDRESS|${FETCH_MAIL_ADDRESS}|g" '.fetchmailrc' + sed -i "s|FETCH_MAIL_AUTH_CODE|${FETCH_MAIL_AUTH_CODE}|g" '.fetchmailrc' + sed -i "s|FETCH_MAIL_DIR|${FETCH_MAIL_DIR}|g" '.procmailrc' + sed -i "s|FETCH_MAIL_BOX|${FETCH_MAIL_BOX}|g" '.procmailrc' +} + +mail_conf +fetchmail -d 100 -- 2.23.0

1 0

[PATCH v5 compass-ci 2/6] fetch-mail: config file for procmail
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/procmailrc.template | 25 ++++++++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 container/fetch-mail/procmailrc.template diff --git a/container/fetch-mail/procmailrc.template b/container/fetch-mail/procmailrc.template new file mode 100644 index 0000000..9cea4bc --- /dev/null +++ b/container/fetch-mail/procmailrc.template @@ -0,0 +1,25 @@ +DROPPRIVS=yes +PATH=$HOME/bin:/usr/local/bin:/usr/bin:/bin +MAILDIR=FETCH_MAIL_DIR +MAILLIST=${MAILDIR} +DEFAULT=${MAILDIR}/FETCH_MAIL_BOX/ +ORGMAIL=${DEFAULT}/FETCH_MAIL_BOX/ +LOGFILE=${MAILDIR}/log/procmail # recommended + +UMASK=002 + +DISPLAY=:0 +XAUTHORITY=$HOME/.Xauthority + +# One often receives duplicate messages for cross-postings to mailing +# lists, or explicit replies to a post from you in addition to the +# mailing list message. +# +# The following recipe eliminates all duplicate messages. It tells +# formail(1) to keep a cache file of the Message-IDs of the most recent +# messages you've received. Since Message-IDs are guaranteed unique for +# distinct messages, they are ideally suited to weed out duplicates. + +# LOCKFILE=$MAILDIR/.msgid.lock +:0 Wh: $MAILDIR/.msgid.lock +| formail -D 655360 $MAILDIR/log/msgid -- 2.23.0

1 0

[PATCH v5 compass-ci 1/6] fetch-mail: config file for fetchmail
by Luan Shengde 17 Dec '20

17 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/fetch-mail/fetchmailrc.template | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 container/fetch-mail/fetchmailrc.template diff --git a/container/fetch-mail/fetchmailrc.template b/container/fetch-mail/fetchmailrc.template new file mode 100644 index 0000000..8c7e82b --- /dev/null +++ b/container/fetch-mail/fetchmailrc.template @@ -0,0 +1,5 @@ +poll imap.qq.com with proto IMAP +username "FETCH_MAIL_ADDRESS" +password "FETCH_MAIL_AUTH_CODE" +mda "/usr/bin/procmail -f %F -d %T" +nokeep -- 2.23.0

1 0

[PATCH lkp-tests] daemon: add the local execution permission to the script
by Hu Xuejiao 17 Dec '20

17 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-docker | 2 +- daemon/multi-qemu | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/daemon/multi-docker b/daemon/multi-docker index b39506b6..49c8c968 100755 --- a/daemon/multi-docker +++ b/daemon/multi-docker @@ -8,7 +8,7 @@ multi_docker() { - export CCI_SRC=/c/compass-ci + [ -n "$CCI_SRC" ] || export CCI_SRC=/c/compass-ci local hostname=$tbox_group.$HOSTNAME $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues } diff --git a/daemon/multi-qemu b/daemon/multi-qemu index aa2348b4..99f63583 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -10,9 +10,9 @@ multi_qemu() { - export CCI_SRC=/c/compass-ci + [ -n "$CCI_SRC" ] || export CCI_SRC=~/compass-ci local hostname=$tbox_group.$HOSTNAME - + cd "$CACHE_DIR" $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" } -- 2.23.0

4 3

[PATCH v2 compass-ci] git_mirror.rb: improve how to get filenames
by Ren Wen 17 Dec '20

17 Dec '20

Dir.children returns filenames except for “.” and “..”. Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/git_mirror.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 389b185..156d5bb 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -128,7 +128,7 @@ class MirrorMain def traverse_repodir(repodir) if File.directory? repodir load_defaults(repodir) - entry_list = Dir.entries(repodir) - Array['.', '..', 'DEFAULTS', '.ignore', '.git'] + entry_list = Dir.children(repodir) - ['DEFAULTS'] entry_list.each do |entry| next if entry.start_with? '.' -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: import h5ai into container
by Lu Weitao 17 Dec '20

17 Dec '20

[Why] Optimize result page style to make it more modern Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/build | 17 +++++++++++++++++ .../srv-http/root/etc/nginx/conf.d/default.conf | 5 ++--- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/container/srv-http/build b/container/srv-http/build index 3a500ea..6580c55 100755 --- a/container/srv-http/build +++ b/container/srv-http/build @@ -2,4 +2,21 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[[ $H5AI_SRC ]] || H5AI_SRC=/c/h5ai + +WORKSPACE=$(dirname $(realpath $0)) + +# npm build h5ai package +cd $H5AI_SRC +npm install +npm run build + +# add h5ai package in ./root/srv to add it into image +cd $WORKSPACE +mkdir ./root/srv +cp -r $H5AI_SRC/build/_h5ai ./root/srv + docker build -t srv-http . + +# clean h5ai package +rm -rf ./root/srv diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 676418a..646eee3 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -3,8 +3,8 @@ server { server_name "srv-http"; server_tokens off; - root /usr/share/nginx/html; - index index.html; + root /srv; + index index.html index.php /_h5ai/public/index.php; charset utf-8,gbk; @@ -24,7 +24,6 @@ server { autoindex on; autoindex_localtime on; allow all; - root /srv; location ~* /[-_a-z]+$ { add_header Content-Type text/plain; } -- 2.23.0

1 1

[PATCH compass-ci] providers: mount "/srv/result" for docker
by Cao Xueliang 17 Dec '20

17 Dec '20

Now use mount "/srv/result" to get error messages for bisect job by container, next will get the error messages from the assistant container API. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- providers/docker/run.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index d532a56..127cf5b 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -29,6 +29,7 @@ cmd=( -v ${DIR}/bin:/root/bin:ro -v $CCI_SRC:/c/compass-ci -v /srv/git:/srv/git:ro + -v /srv/result:/srv/result:ro -v ${busybox_path}:/usr/local/bin/busybox --oom-score-adj="-1000" ${docker_image} -- 2.23.0

1 0

[PATCH compass-ci] code-spec.md: adjust markdown document format
by Xu Xijian 17 Dec '20

17 Dec '20

In markdown documents, text for bash should be contained in block as belows: ```SHELL content SHELL Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/code-spec.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index d44c6fe..a717f64 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -21,10 +21,12 @@ code注意事项 - puts/printf字符串应当容易搜索(grep)。特别注意包含足够长的固定内容substring. 例如： +```SHELL die "test yaml$${test_yaml}$ not exist in ${LKP_SRC}/jobs !!!" ==> die "cannot find test yaml ($test_yaml) in $LKP_SRC/jobs" ^^^^^^^^^^^^^^^^^^^^^ grep-friendly fixed substring +``` 重构友好编码 ============ -- 2.23.0

2 2

[PATCH compass-ci] lib: add the add_task function to taskqueue client
by Cao Xueliang 17 Dec '20

17 Dec '20

1. Modify the default host to 172.17.0.1. 2. Add the add_task function to taskqueue client. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/taskqueue_client.rb | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/lib/taskqueue_client.rb b/lib/taskqueue_client.rb index 91b72dc..2c22990 100644 --- a/lib/taskqueue_client.rb +++ b/lib/taskqueue_client.rb @@ -6,7 +6,7 @@ require 'rest-client' # taskqueue client class TaskQueueClient - HOST = (ENV.key?('TASKQUEUE_HOST') ? ENV['TASKQUEUE_HOST'] : '127.0.0.1') + HOST = (ENV.key?('TASKQUEUE_HOST') ? ENV['TASKQUEUE_HOST'] : '172.17.0.1') PORT = (ENV.key?('TASKQUEUE_PORT') ? ENV['TASKQUEUE_PORT'] : 3060).to_i def initialize(host = HOST, port = PORT) @host = host @@ -20,4 +20,14 @@ class TaskQueueClient url: url ) end + + def add_task(queue_path, json_data) + url = "http://#{@host}:#{@port}/add?queue=#{queue_path}" + RestClient::Request.execute( + method: :post, + url: url, + payload: json_data, + headers: { content_type: 'application/json' } + ) + end end -- 2.23.0

1 0

[PATCH compass-ci] container: add curl to delimiter container for uploading files
by Cao Xueliang 17 Dec '20

17 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/delimiter/Dockerfile b/container/delimiter/Dockerfile index 4163cea..3db5890 100644 --- a/container/delimiter/Dockerfile +++ b/container/delimiter/Dockerfile @@ -6,7 +6,7 @@ FROM alpine:3.11 RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' RUN umask 002 && \ echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ -- 2.23.0

1 0

[PATCH v1 compass-ci] lib: fix string literal in condition
by Cao Xueliang 17 Dec '20

17 Dec '20

1. Fix warning string literal in condition 2. Display the first error message in email subject Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index c0a2731..090855e 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -19,8 +19,8 @@ class MailBisectResult end def compose_mail - subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{@error_messages[0]}" - job_url = "job url: #{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}/#{ENV['result_root']}\n" ? ENV['result_root'] : '' + subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{(a)error_messages[0].split("\n")[0]}" + job_url = ENV['result_root'] ? "job url: http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}#{ENV['result_root']}\n" : '' body = <<~BODY Hi #{(a)git_commit.author_name}, @@ -28,6 +28,7 @@ class MailBisectResult git commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") gcc version: 7.3.0 + error_messages: #{(a)error_messages.join("\n")} -- 2.23.0

1 0

[PATCH compass-ci] lib: fix string literal in condition
by Cao Xueliang 17 Dec '20

17 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index c0a2731..090855e 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -19,8 +19,8 @@ class MailBisectResult end def compose_mail - subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{@error_messages[0]}" - job_url = "job url: #{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}/#{ENV['result_root']}\n" ? ENV['result_root'] : '' + subject = "[Compass-CI][#{(a)repo.split('/')[1]}]: #{(a)error_messages[0].split("\n")[0]}" + job_url = ENV['result_root'] ? "job url: http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}#{ENV['result_root']}\n" : '' body = <<~BODY Hi #{(a)git_commit.author_name}, @@ -28,6 +28,7 @@ class MailBisectResult git commit: #{@commit_id[0..11]} ("#{(a)git_commit.subject}") gcc version: 7.3.0 + error_messages: #{(a)error_messages.join("\n")} -- 2.23.0

1 0

[PATCH v6 compass-ci 4/4] container/assign-account/get_account_info.rb: refactor sort for files
by Lu Kaiyi 17 Dec '20

17 Dec '20

files.sort make no difference for files itself Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- container/assign-account/get_account_info.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 67c4c46..af08d91 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -76,11 +76,11 @@ class AccountStorage end def build_account_name(available_dir, files) - files.sort + files.sort! account_info = [] - account_info.push files[0] - source_file = File.join(available_dir, files[0]) - account_info.push File.readlines(source_file)[0].chomp + account_info.push files.first + source_file = File.join(available_dir, files.first) + account_info.push File.readlines(source_file).first.chomp dest_dir = File.join(@account_dir, 'assigned-users') FileUtils.mv(source_file, dest_dir) -- 2.23.0

1 0

[PATCH v6 compass-ci 3/4] lib/compare_data_format.rb: limit the number of table columns in a single row
by Lu Kaiyi 17 Dec '20

17 Dec '20

[example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - meminfo.Active - meminfo.Memused filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 4k|1G | 4k|10G | 4k|80G | 4k|100G | 16k|1G | 16k|10G | 16k|100G | 32k|1G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1051645.34 | 704958.00 | 706798.00 | 657669.27 | 1030484.85 | 979289.00 | 948576.00 | 1262591.17 | | standard_deviation openeuler 20.03 | 74.00 | 0.00 | 0.00 | 8.00 | 46.00 | 0.00 | 2.00 | 42.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 64k|1G | 64k|100G | 128k|1G | 128k|100G | 256k|1G | 256k|100G | 512k|1G | 512k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1418731.53 | 1854231.33 | 2004321.76 | 2960171.75 | 2979326.50 | 4775340.00 | 5003555.53 | 5630845.75 | | standard_deviation openeuler 20.03 | 47.00 | 10.00 | 65.00 | 18.00 | 88.00 | 8.00 | 108.00 | 15.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 1024k|1G | 1024k|100G | 2048k|100G | 4096k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 8974190.91 | 9641039.00 | 5480763.67 | 7302509.67 | | standard_deviation openeuler 20.03 | 118.00 | 36.00 | 86.00 | 67.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 41 +++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index c1f7023..0f6bdc0 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -72,19 +72,20 @@ end # format compare template results into a table format # class FormatTableData - def initialize(result_hash) + def initialize(result_hash, row_size = 8) @title = result_hash['title'] @tables = result_hash['tables'] @unit = result_hash['unit'] @x_name = result_hash['x_name'] + @row_size = row_size end def show_table @tables.each do |table_title, table| @tb = Terminal::Table.new set_table_title - set_field_names(table_title, table) - add_rows(table) + row_num = get_row_num(table) + split_data_column(table_title, table, row_num) set_align_column print_table end @@ -94,14 +95,36 @@ class FormatTableData @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" end - def set_field_names(table_title, table) + def get_row_num(table) + data_column_size = table['average']['source'][0].size + unless @row_size.positive? + warn('row size must be positive!') + exit + end + (data_column_size / @row_size.to_f).ceil + end + + def split_data_column(table_title, table, row_num) + row_num.times do |row| + starts = 1 + row * @row_size + ends = starts + @row_size + set_field_names(table_title, table, starts, ends) + add_rows(table, starts, ends) + break if row == row_num - 1 + + @tb.add_separator + @tb.add_separator + end + end + + def set_field_names(table_title, table, starts, ends) field_names = [table_title] - field_names.concat(table['average']['source'][0]) + field_names.concat(table['average']['source'][0][starts - 1...ends - 1]) @tb.add_row(field_names) @tb.add_separator end - def add_rows(table) + def add_rows(table, starts, ends) row_names = %w[average standard_deviation change] max_size = row_names.map(&:size).max row_names.each do |row_name| @@ -109,15 +132,15 @@ class FormatTableData dimensions_size = table[row_name]['dimensions'].size (1...dimensions_size).each do |index| - add_row(table, row_name, index, max_size) + add_row(table, row_name, index, max_size, starts, ends) end end end - def add_row(table, row_name, index, max_size) + def add_row(table, row_name, index, max_size, starts, ends) row = table[row_name]['source'][index] row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') - format_data_row = row[1..-1] + format_data_row = row[starts...ends] if row_name == 'change' format_data_row.map! { |data| format('%.1f%%', data) } else -- 2.23.0

1 0

[PATCH v6 compass-ci 2/4] add options for compare_by_template
by Lu Kaiyi 17 Dec '20

17 Dec '20

if want json format results, use command like below: compare -t compare_template.yaml --theme json otherwise, print table format results by default: compare -t compare_template.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 10 +++++++--- sbin/compare | 2 +- 3 files changed, 10 insertions(+), 6 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index f34e239..db264f8 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -96,11 +96,11 @@ end # unit: KB/s # -def compare_by_template(template) +def compare_by_template(template, options) template_params = load_template(template) groups_matrices = create_groups_matrices(template_params) compare_results = compare_metrics_values(groups_matrices) - show_compare_result(compare_results, template_params) + show_compare_result(compare_results, template_params, options) end def load_template(template) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 9bf676f..b7ea5a4 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -508,11 +508,15 @@ def get_dimensions_combination(dimension_list) dims end -def show_compare_result(metrics_compare_results, template_params) +def show_compare_result(metrics_compare_results, template_params, options) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - table_results = FormatTableData.new(echart_results) - table_results.show_table + if options[:theme] == 'json' + print JSON.pretty_generate(echart_results) + else + table_results = FormatTableData.new(echart_results) + table_results.show_table + end end # Format Fields diff --git a/sbin/compare b/sbin/compare index c13ec1a..e8d1ac4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -67,7 +67,7 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful if template - compare_by_template(template) + compare_by_template(template, options) elsif is_group compare_group(argv, dimensions, options) else -- 2.23.0

1 0

[PATCH v6 compass-ci 1/4] lib: friendly show compare template result in command line
by Lu Kaiyi 17 Dec '20

17 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the results of "compare -t <compare_template.yaml>" with terminal-table in Ruby. [example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | average openeuler 20.03 | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | standard_deviation openeuler 20.03 | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | average centos 7.6 | 345243.03 | 142698.79 | 62108.35 | 34747.73 | 26330.19 | 10317.85 | 7471.71 | 3558.30 | | average openeuler 20.03 | 122003.54 | 33528.53 | 31469.06 | 13870.14 | 8249.71 | 4329.45 | 1976.54 | 1141.00 | | standard_deviation centos 7.6 | 97.00 | 95.00 | 122.00 | 125.00 | 100.00 | 130.00 | 101.00 | 103.00 | | standard_deviation openeuler 20.03 | 174.00 | 188.00 | 171.00 | 197.00 | 181.00 | 175.00 | 170.00 | 176.00 | | change centos 7.6 vs openeuler 20.03 | 183.0% | 325.6% | 97.4% | 150.5% | 219.2% | 138.3% | 278.0% | 211.9% | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 73 ++++++++++++++++++++++++++++++++++++++ lib/compare_matrixes.rb | 3 +- 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index bf4013b..c1f7023 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -2,6 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'terminal-table' + # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # @@ -66,6 +68,77 @@ class FormatEchartData end end +# ---------------------------------------------------------------------------------------------------- +# format compare template results into a table format +# +class FormatTableData + def initialize(result_hash) + @title = result_hash['title'] + @tables = result_hash['tables'] + @unit = result_hash['unit'] + @x_name = result_hash['x_name'] + end + + def show_table + @tables.each do |table_title, table| + @tb = Terminal::Table.new + set_table_title + set_field_names(table_title, table) + add_rows(table) + set_align_column + print_table + end + end + + def set_table_title + @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" + end + + def set_field_names(table_title, table) + field_names = [table_title] + field_names.concat(table['average']['source'][0]) + @tb.add_row(field_names) + @tb.add_separator + end + + def add_rows(table) + row_names = %w[average standard_deviation change] + max_size = row_names.map(&:size).max + row_names.each do |row_name| + next unless table[row_name] + + dimensions_size = table[row_name]['dimensions'].size + (1...dimensions_size).each do |index| + add_row(table, row_name, index, max_size) + end + end + end + + def add_row(table, row_name, index, max_size) + row = table[row_name]['source'][index] + row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') + format_data_row = row[1..-1] + if row_name == 'change' + format_data_row.map! { |data| format('%.1f%%', data) } + else + format_data_row.map! { |data| format('%.2f', data) } + end + @tb.add_row([row_title, *format_data_row]) + end + + def set_align_column + @tb.number_of_columns.times do |index| + @tb.align_column(index + 1, :right) + end + @tb.align_column(0, :left) + end + + def print_table + puts @tb + puts + end +end + # input: x_params_list # eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] # output: diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 5069939..9bf676f 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -511,7 +511,8 @@ end def show_compare_result(metrics_compare_results, template_params) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - print JSON.pretty_generate(echart_results) + table_results = FormatTableData.new(echart_results) + table_results.show_table end # Format Fields -- 2.23.0

1 0

[PATCH compass-ci] lib: change the file permission
by Zhang Yu 17 Dec '20

17 Dec '20

The file permission should be consistent with the other file in this directory Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/mail-post.rb | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100755 new mode 100644 -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: install hostname
by Liu Yinsi 17 Dec '20

17 Dec '20

sparrow/3-code/dev-env will use hostname command, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 6e3ed42..8df36c0 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -21,6 +21,7 @@ pkgs=( lockfile-progs busybox iproute + hostname ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 8ee6d7e..6a58a0c 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -19,6 +19,7 @@ pkgs=( bridge-utils busybox iproute2 + hostname ) apt-get install -y --fix-missing "${pkgs[@]}" diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 13e0875..a63964f 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -20,6 +20,7 @@ pkgs=( wget busybox iproute + hostname ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests: fix error when running host-info in vm
by Zhang Yu 17 Dec '20

17 Dec '20

Error: Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/tests/host-info b/tests/host-info index 0ab8e6c97..3984727b4 100755 --- a/tests/host-info +++ b/tests/host-info @@ -102,11 +102,9 @@ get_ipmi_ip() # 802.1q VLAN ID : Disabled # RMCP+ Cipher Suites : 0,1,2,3,17 # Cipher Suite Priv Max : XuuaXXXXXXXXXXX - is_virt && return - is_docker && return + [ -e /dev/ipmi0 ] || [ -e /dev/ipmi/0 ] || [ -e /dev/ipmidev/0 ] || return local ip=$(ipmitool lan print | awk 'NR==3{print $4}') [ -n "$ip" ] && echo "ipmi_ip: $ip" - } main() -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow/3-code: use defaults/* to be loaded as job defaults
by Liu Yinsi 16 Dec '20

16 Dec '20

before: /etc/compass-ci/account/autosubmit.yaml after: /etc/compass-ci/defaults/admin-account.yaml Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 750738f..7eb6166 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -41,8 +41,7 @@ LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci EOF -mkdir -p /etc/compass-ci/account -cat > /etc/compass-ci/account/autosubmit.yaml <<EOF +cat > /etc/compass-ci/defaults/admin-account.yaml <<EOF my_name: Auto Submit Robot my_email: autosubmit@localhost my_token: $(uuidgen) -- 2.23.0

2 1

[PATCH compass-ci] container/assign-account/get_account_info.rb: refactor sort for files
by Lu Kaiyi 16 Dec '20

16 Dec '20

files.sort make no difference for files itself Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- container/assign-account/get_account_info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 9a7acda..0ce8cad 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -76,7 +76,7 @@ class AccountStorage end def build_account_name(available_dir, files) - files.sort + files.sort! account_info = [] account_info.push files[0] source_file = File.join(available_dir, files[0]) -- 2.23.0

2 1

[PATCH v5 compass-ci 3/3] lib/compare_data_format.rb: limit the number of table columns in a single row
by Lu Kaiyi 16 Dec '20

16 Dec '20

[example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - meminfo.Active - meminfo.Memused filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 4k|1G | 4k|10G | 4k|80G | 4k|100G | 16k|1G | 16k|10G | 16k|100G | 32k|1G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1051645.34 | 704958.00 | 706798.00 | 657669.27 | 1030484.85 | 979289.00 | 948576.00 | 1262591.17 | | standard_deviation openeuler 20.03 | 74.00 | 0.00 | 0.00 | 8.00 | 46.00 | 0.00 | 2.00 | 42.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 64k|1G | 64k|100G | 128k|1G | 128k|100G | 256k|1G | 256k|100G | 512k|1G | 512k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1418731.53 | 1854231.33 | 2004321.76 | 2960171.75 | 2979326.50 | 4775340.00 | 5003555.53 | 5630845.75 | | standard_deviation openeuler 20.03 | 47.00 | 10.00 | 65.00 | 18.00 | 88.00 | 8.00 | 108.00 | 15.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 1024k|1G | 1024k|100G | 2048k|100G | 4096k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 8974190.91 | 9641039.00 | 5480763.67 | 7302509.67 | | standard_deviation openeuler 20.03 | 118.00 | 36.00 | 86.00 | 67.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 41 +++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index c1f7023..0f6bdc0 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -72,19 +72,20 @@ end # format compare template results into a table format # class FormatTableData - def initialize(result_hash) + def initialize(result_hash, row_size = 8) @title = result_hash['title'] @tables = result_hash['tables'] @unit = result_hash['unit'] @x_name = result_hash['x_name'] + @row_size = row_size end def show_table @tables.each do |table_title, table| @tb = Terminal::Table.new set_table_title - set_field_names(table_title, table) - add_rows(table) + row_num = get_row_num(table) + split_data_column(table_title, table, row_num) set_align_column print_table end @@ -94,14 +95,36 @@ class FormatTableData @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" end - def set_field_names(table_title, table) + def get_row_num(table) + data_column_size = table['average']['source'][0].size + unless @row_size.positive? + warn('row size must be positive!') + exit + end + (data_column_size / @row_size.to_f).ceil + end + + def split_data_column(table_title, table, row_num) + row_num.times do |row| + starts = 1 + row * @row_size + ends = starts + @row_size + set_field_names(table_title, table, starts, ends) + add_rows(table, starts, ends) + break if row == row_num - 1 + + @tb.add_separator + @tb.add_separator + end + end + + def set_field_names(table_title, table, starts, ends) field_names = [table_title] - field_names.concat(table['average']['source'][0]) + field_names.concat(table['average']['source'][0][starts - 1...ends - 1]) @tb.add_row(field_names) @tb.add_separator end - def add_rows(table) + def add_rows(table, starts, ends) row_names = %w[average standard_deviation change] max_size = row_names.map(&:size).max row_names.each do |row_name| @@ -109,15 +132,15 @@ class FormatTableData dimensions_size = table[row_name]['dimensions'].size (1...dimensions_size).each do |index| - add_row(table, row_name, index, max_size) + add_row(table, row_name, index, max_size, starts, ends) end end end - def add_row(table, row_name, index, max_size) + def add_row(table, row_name, index, max_size, starts, ends) row = table[row_name]['source'][index] row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') - format_data_row = row[1..-1] + format_data_row = row[starts...ends] if row_name == 'change' format_data_row.map! { |data| format('%.1f%%', data) } else -- 2.23.0

2 1

[PATCH v5 compass-ci 2/3] add options for compare_by_template
by Lu Kaiyi 16 Dec '20

16 Dec '20

if want json format results, use command like below: compare -t compare_template.yaml --theme json otherwise, print table format results by default: compare -t compare_template.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 11 ++++++++--- sbin/compare | 2 +- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index f34e239..db264f8 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -96,11 +96,11 @@ end # unit: KB/s # -def compare_by_template(template) +def compare_by_template(template, options) template_params = load_template(template) groups_matrices = create_groups_matrices(template_params) compare_results = compare_metrics_values(groups_matrices) - show_compare_result(compare_results, template_params) + show_compare_result(compare_results, template_params, options) end def load_template(template) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 9bf676f..e8f52f1 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -508,11 +508,16 @@ def get_dimensions_combination(dimension_list) dims end -def show_compare_result(metrics_compare_results, template_params) +def show_compare_result(metrics_compare_results, template_params, options) + theme = options[:theme] formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - table_results = FormatTableData.new(echart_results) - table_results.show_table + if theme == 'json' + print JSON.pretty_generate(echart_results) + else + table_results = FormatTableData.new(echart_results) + table_results.show_table + end end # Format Fields diff --git a/sbin/compare b/sbin/compare index c13ec1a..e8d1ac4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -67,7 +67,7 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful if template - compare_by_template(template) + compare_by_template(template, options) elsif is_group compare_group(argv, dimensions, options) else -- 2.23.0

2 1

[PATCH lkp-tests] tests/build-pkg: build different version software packages
by Liu Shaofei 16 Dec '20

16 Dec '20

[why] When building different version software, we need to pack out different *.cgz. The field of PKGBUILD_PKGVE can be appended job.yaml file, which specifies the version number. The parts of contents for job.yaml file is as follow: job.yaml: ... pkgbuild_repo: archlinux/packages//fcgi/trunk PKGBUILD_PKGVER: 2.4.2 ... [how] -rw-rw-r--. 1 lkp lkp 46K 2020-12-15 17:17 2.4.2.cgz lrwxrwxrwx. 1 lkp lkp 9 2020-12-15 17:17 latest.cgz -> 2.4.2.cgz Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index aafc120ed..c3909d1b9 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -3,6 +3,7 @@ # - upstream_repo # - upstream_commit # - PKGBUILD_TAG +# - PKGBUILD_PKGVER # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -60,18 +61,6 @@ request_pkg() do get_pkgfile "${pkgrepo}" "$pkgfile" done - - # download PKGBUILD-$tag - benchmark="${upstream_repo##*/}" - benchmark="${benchmark%.git}" - [ -n "$PKGBUILD_TAG" ] && { - get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" - [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" - [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" - } - - [ -f PKGBUILD ] || die "PKGBUILD not exist" - [ -s PKGBUILD ] || die "PKGBUILD is empty" } create_softlink() @@ -109,6 +98,7 @@ replace_source() build_source_pkg() { + [ -n "$PKGBUILD_PKGVER" ] && sed -i "s|^pkgver=.*|pkgver=${PKGBUILD_PKGVER}|g" PKGBUILD repo_dir="" source PKGBUILD [ -n "$upstream_repo" ] && [ -n "$upstream_url" ] && { @@ -122,11 +112,16 @@ build_source_pkg() } replace_source PKGBUILD.src - [ -n "$PKGBUILD_TAG" ] && replace_source "PKGBUILD-$PKGBUILD_TAG" - cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" + pkg_args="-A --check --skippgpcheck" + + [ -n ${PKGBUILD_PKGVER} ] && { + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${pkgver}.cgz" + pkg_args="${pkg_args} --skipchecksums --skipinteg" + } + PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" STDERR_REDIR=y\ - $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf [ "$?" == 0 ] || exit 1 create_softlink -- 2.23.0

2 3

[PATCH lkp-tests 1/3] tests/build-pkg: fix format
by Lin Jiaxin 16 Dec '20

16 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index aafc120e..dbde3e54 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,8 +2,8 @@ # - pkgbuild_repo # - upstream_repo # - upstream_commit +# - upstream_url # - PKGBUILD_TAG -# upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ -- 2.23.0

2 1

Re: [PATCH lkp-tests] lib/upload: change the key of upload results
by Cao Xueliang 16 Dec '20

16 Dec '20

On Wed, Dec 16, 2020 at 07:56:34PM +0800, Li Ping wrote: >On Wed, Dec 16, 2020 at 03:37:09PM +0800, Wei Jihui wrote: >>the key of upload results is changed from access_key to job id >> >>Signed-off-by: Wei Jihui <weijihuiall(a)163.com> >>--- >> lib/job-init.sh | 2 +- >> lib/upload.sh | 18 +++++++++--------- >> 2 files changed, 10 insertions(+), 10 deletions(-) >> >>diff --git a/lib/job-init.sh b/lib/job-init.sh >>index 40534af3..dc4a65d9 100755 >>--- a/lib/job-init.sh >>+++ b/lib/job-init.sh >>@@ -55,7 +55,7 @@ setup_result_service() >> >> supports_raw_upload && result_service=raw_upload && return >> >>- is_docker || [ -n "$access_key" ] || { >>+ is_docker || [ -n "$id" ] || { >> if [ "$os_mount" = 'nfs' ] || [ -z "$os_mount" ]; then >> supports_netfs 'nfs' && result_service=$RESULT_SERVER:/result && return >> elif [ "$os_mount" = 'cifs' ] || [ -z "$os_mount" ]; then >>diff --git a/lib/upload.sh b/lib/upload.sh >>index a5daceda..0115e350 100755 >>--- a/lib/upload.sh >>+++ b/lib/upload.sh >>@@ -78,9 +78,9 @@ upload_one_curl() >> ( >> cd $(dirname "$1") >> dir=$(basename "$1") >>- if [ -n "$access_key" ]; then >>- find "$dir" -type d -exec curl -sSf -X MKCOL "$http_url/{}" --cookie "ACCESSKEY=$access_key" \; >>- find "$dir" -type f -size +0 -exec curl -sSf -T '{}' "$http_url/{}" --cookie "ACCESSKEY=$access_key" \; >>+ if [ -n "$id" ]; then >>+ find "$dir" -type d -exec curl -sSf -X MKCOL "$http_url/{}" --cookie "JOBID=$id" \; >>+ find "$dir" -type f -size +0 -exec curl -sSf -T '{}' "$http_url/{}" --cookie "JOBID=$id" \; > > users can upload files or folders regardless > of whether the job id exists. what is the > function of determing whether the job id > exist? Get the id from the env var, when we run a job lkp will export all the job yaml as the env var. Thanks, Xueliang > > Thanks, > Ping > >> else >> find "$dir" -type d -exec curl -sSf -X MKCOL "$http_url/{}" \; >> find "$dir" -type f -size +0 -exec curl -sSf -T '{}' "$http_url/{}" \; >>@@ -88,8 +88,8 @@ upload_one_curl() >> ) >> else >> [ -s "$src" ] || return >>- if [ -n "$$access_key" ]; then >>- curl -sSf -T "$src" $http_url/ --cookie "ACCESSKEY=$access_key" >>+ if [ -n "$id" ]; then >>+ curl -sSf -T "$src" $http_url/ --cookie "JOBID=$id" >> else >> curl -sSf -T "$src" $http_url/ >> fi >>@@ -111,8 +111,8 @@ upload_files_curl() >> for dir in $(echo $target_directory | tr '/' ' ') >> do >> job_result_root=$job_result_root/$dir/ >>- if [ -n "$$access_key" ]; then >>- curl -sSf -X MKCOL "$http_host$job_result_root" --cookie "ACCESSKEY=$access_key" >/dev/null >>+ if [ -n "$id" ]; then >>+ curl -sSf -X MKCOL "$http_host$job_result_root" --cookie "JOBID=$id" >/dev/null >> else >> curl -sSf -X MKCOL "$http_host$job_result_root" >/dev/null >> fi >>@@ -178,12 +178,12 @@ upload_files() >> upload_files_copy "$@" >> return >> } >>- if has_cmd rsync && has_rsync_server && [ -z "$access_key" ]; then >>+ if has_cmd rsync && has_rsync_server && [ -z "$id" ]; then >> upload_files_rsync "$@" >> return >> fi >> >>- if has_cmd lftp && [ -z "$access_key" ]; then >>+ if has_cmd lftp && [ -z "$id" ]; then >> upload_files_lftp "$@" >> return >> fi >>-- >>2.23.0 >>

1 0

[PATCH v2 lkp-tests] tests/build-pkg: build different version software packages
by Liu Shaofei 16 Dec '20

16 Dec '20

[why] When building different version software, we need to pack out different *.cgz. The field of "git_tag" and "pkg_ver" can be appended job.yaml file, which specifies the version number. The parts of contents for job.yaml file is as follow: job.yaml: ... pkgbuild_repo: archlinux/packages//fcgi/trunk pkg_ver: 2.4.2 ... [how] -rw-rw-r--. 1 lkp lkp 46K 2020-12-15 17:17 2.4.2.cgz lrwxrwxrwx. 1 lkp lkp 9 2020-12-15 17:17 latest.cgz -> 2.4.2.cgz Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 43 +++++++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 20 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index aafc120ed..fd2697e25 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,7 +2,8 @@ # - pkgbuild_repo # - upstream_repo # - upstream_commit -# - PKGBUILD_TAG +# - git_tag +# - pkg_ver # upstream_url # We'll use the below global env vars, but not treat them as parameter to avoid @@ -60,34 +61,32 @@ request_pkg() do get_pkgfile "${pkgrepo}" "$pkgfile" done +} + +create_cgz_name() +{ + cgz_path="$PKG_MNT/${pack_to}/${pkgname}" + cgz_name="${cgz_path}/${upstream_commit}.cgz" + pkg_args="-A --check --skippgpcheck" - # download PKGBUILD-$tag - benchmark="${upstream_repo##*/}" - benchmark="${benchmark%.git}" - [ -n "$PKGBUILD_TAG" ] && { - get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" - [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" - [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" + [ -n "$git_tag" ] && { + cgz_name="${cgz_path}/${git_tag}.cgz" } - [ -f PKGBUILD ] || die "PKGBUILD not exist" - [ -s PKGBUILD ] || die "PKGBUILD is empty" + [ -n "$pkg_ver" ] && { + cgz_name="${cgz_path}/${pkg_ver}.cgz" + pkg_args="${pkg_args} --skipchecksums --skipinteg" + } } create_softlink() { [ -e "$cgz_name" ] || return 0 - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" local bm_name=$(basename $(realpath ${cgz_name})) - if [ -n "$PKGBUILD_TAG" ]; then - echo "create soft link: ${PKGBUILD_TAG}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${PKGBUILD_TAG}.cgz" - fi - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" + ln -sf "${bm_name}" "${cgz_path}/latest.cgz" } replace_source() @@ -98,6 +97,9 @@ replace_source() echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" + [ -n ${git_tag} ] && { + url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#tag=${git_tag}" + } } } url=$(echo "$url" | sed 's|https://|http://|g') @@ -109,6 +111,8 @@ replace_source() build_source_pkg() { + [ -n "$pkg_ver" ] && sed -i "s|^pkgver=.*|pkgver=${pkg_ver}|g" PKGBUILD + repo_dir="" source PKGBUILD [ -n "$upstream_repo" ] && [ -n "$upstream_url" ] && { @@ -122,11 +126,10 @@ build_source_pkg() } replace_source PKGBUILD.src - [ -n "$PKGBUILD_TAG" ] && replace_source "PKGBUILD-$PKGBUILD_TAG" + create_cgz_name - cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" STDERR_REDIR=y\ - $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf [ "$?" == 0 ] || exit 1 create_softlink -- 2.23.0

2 2

[PATCH compass-ci] sched.cr: put testbox state to es
by Li Yuanchao 16 Dec '20

16 Dec '20

History of testbox would be kept. [sample] { "_index" : "testbox", "_type" : "_doc", "_id" : "vm-2p16g.liyuanchao-969257", "_score" : 1.0, "_source" : { "state" : "rebooting", "job_id" : "crystal.190886", "history" : [ { "time" : "2020-12-15 19:52:14", "mac" : "0a-f5-c0-1f-f0-dd", "ip" : "172.18.202.96", "job_id" : "crystal.190886", "state" : "running" }, { "time" : "2020-12-15 19:57:22", "mac" : "0a-f5-c0-1f-f0-dd", "ip" : "172.18.202.96", "job_id" : "crystal.190886", "state" : "rebooting" } ] } } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/sched.cr | 1 + src/scheduler/elasticsearch_client.cr | 21 +++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 10a51ab..7cb0dbf 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -104,6 +104,7 @@ class Sched end @redis.update_wtmp(testbox, hash) + @es.update_tbox(testbox, hash) # json log hash["testbox"] = testbox diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 7c92204..b17d7f8 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -87,6 +87,27 @@ class Elasticsearch::Client ) end + def update_tbox(testbox : String, wtmp_hash : Hash) + query = {:index => "testbox", :type => "_doc", :id => testbox} + result = @client.get_source(query) if @client.exists(query) + history = result["history"].as_a? if result.is_a?(JSON::Any) + history ||= [] of JSON::Any + history << JSON.parse(wtmp_hash.to_json) + + @client.create( + { + :index => "testbox", + :type => "_doc", + :id => testbox, + :body => { + :state => wtmp_hash["state"], + :job_id => wtmp_hash["job_id"], + :history => history + } + } + ) + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

2 1

[PATCH compass-ci] update-os-docker: add support sshd to docker images
by Xu Xijian 16 Dec '20

16 Dec '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/update-os-docker/sbin/setup | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/container/update-os-docker/sbin/setup b/container/update-os-docker/sbin/setup index a38e2df..e054a29 100755 --- a/container/update-os-docker/sbin/setup +++ b/container/update-os-docker/sbin/setup @@ -4,3 +4,9 @@ yum update -y yum install -y --skip-broken "$@" + +command -v sshd || yum install -y openssh-server +echo -e "y\n" | ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key +echo -e "y\n" | ssh-keygen -t rsa -f /etc/ssh/ssh_host_ecdsa_key +echo -e "y\n" | ssh-keygen -t rsa -f /etc/ssh/ssh_host_ed25519_key + -- 2.23.0

3 2