- Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests] lib/bootstrap.sh: 'run-ipconfig' step should be before the check step
by Yu Chuan 03 Jan '21

03 Jan '21

[Why] The `run-ipconfig` step is the IP configuration step for initramfs, which should be executed immediately after checking the initramfs-tools file for existence. [Error Msg] + [ ! -e /usr/share/initramfs-tools/scripts/functions ] + get_net_devices + local net_devices + local i + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + [ p0s1/ != /sys/class/net/enp0s1/ ] + basename /sys/class/net/enp0s1/ + net_devices= enp0s1 + [ != /sys/class/net/lo/ ] + continue + echo enp0s1 + local net_devices= enp0s1 + [ -z ] + warn_no_eth0 + [ -f /proc/config.gz ] + return + echo ls /sys/class/net + ls /sys/class/net + is_virt + [ -n ] + [ -n ] + has_cmd virt-what + command -v virt-what + grep -q -w hypervisor /proc/cpuinfo + return 0 + export NO_NETWORK=1 + echo export NO_NETWORK=1 due to no net devices + return Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7d15bb370fa9..770d550724f1 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -160,6 +160,9 @@ setup_network() return } + $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig + network_ok && return + local net_devices=$(get_net_devices) if [ -z "$net_devices" ]; then @@ -181,9 +184,6 @@ setup_network() fi fi - $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig - network_ok && return - local err_msg='IP-Config: Auto-configuration of network failed' dmesg | grep -q -F "$err_msg" || { # Include $err_msg in the error message so that it matches -- 2.23.0

1 0

[PATCH lkp-tests] lib/bootstrap.sh: 'run-ipconfig' step should be before the check step
by Yu Chuan 03 Jan '21

03 Jan '21

[Before] test job: result/borrow/2020-12-31/vm-2p8g/debian-sid-aarch64/3600/z9.275666 key log: + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + cat /sys/class/net/enp0s1//operstate + [ down = up ] [After] test job: result/borrow/2020-12-31/vm-2p8g/debian-sid-aarch64/3600/z9.275673 key log: + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + cat /sys/class/net/enp0s1//operstate + [ up = up ] Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7d15bb370fa9..770d550724f1 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -160,6 +160,9 @@ setup_network() return } + $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig + network_ok && return + local net_devices=$(get_net_devices) if [ -z "$net_devices" ]; then @@ -181,9 +184,6 @@ setup_network() fi fi - $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig - network_ok && return - local err_msg='IP-Config: Auto-configuration of network failed' dmesg | grep -q -F "$err_msg" || { # Include $err_msg in the error message so that it matches -- 2.23.0

2 2

git count
by Wu Fengguang 31 Dec '20

31 Dec '20

Cao Xueliang is our code star again, thanks xueliang! 22 Cao Xueliang 21 Luan Shengde 16 Wu Zhende 13 Wei Jihui 13 Li Yuanchao 11 Hu Xuejiao 11 Ren Wen 10 Lu Weitao 9 Liu Shaofei 7 Lin Jiaxin 7 Lu Kaiyi 6 Liu Yinsi 6 Cui Yili 6 Wang Yong 5 Li Ping 4 Xiao Shenwei 3 Wu Fengguang 3 Zhang Yu 3 Yu Chuan 2 Xu Xijian 2 Wang Chenglong 1 Zhang Yale 0 Bai Jing 0 Tong Qunfeng 0 Du Kaitian 0 Sun Yukui 0 Zhang Dewan 0 Shi Zhichao

1 0

[PATCH v2 compass-ci 2/2] mail-robot: optimize assign-account-email
by Luan Shengde 31 Dec '20

31 Dec '20

set different greetings for mail-robot and bisect Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index ee3b574..b71b3fe 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -4,13 +4,19 @@ # frozen_string_literal: true def build_apply_account_email(my_info) + email_greeting = if my_info['bisect'] + 'We have automatically created the following information for you.' + else + 'Thank you for joining us.' + end + email_msg = <<~EMAIL_MESSAGE To: #{my_info['my_email']} Subject: [compass-ci] Account Ready Dear #{my_info['my_name']}, - Thank you for joining us. + #{email_greeting} You can use the following info to submit jobs: -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/assign_account_client: optimize
by Luan Shengde 31 Dec '20

31 Dec '20

add new key for bisect way to apply account use rest-client instead of linux command 'curl' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb index d7e0623..4831545 100755 --- a/lib/assign_account_client.rb +++ b/lib/assign_account_client.rb @@ -7,6 +7,7 @@ require 'json' require 'mail' require 'set' require 'optparse' +require 'rest-client' require_relative '../container/defconfig' require_relative 'es_client' require_relative '../container/mail-robot/lib/assign-account-email' @@ -52,7 +53,8 @@ class AutoAssignAccount apply_info['is_update_account'] = true unless apply_info['my_login_name'].nil? apply_info['lab'] = LAB - account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" + account_info_str = RestClient.post assign_account_url, apply_info.to_json JSON.parse account_info_str end @@ -69,6 +71,7 @@ class AutoAssignAccount end def send_mail + @my_info['bisect'] = true message = build_apply_account_email(@my_info) %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") end -- 2.23.0

1 0

[PATCH compass-ci 2/2] mail-robot: optimize assign-account-email
by Luan Shengde 31 Dec '20

31 Dec '20

set different greetings for mail-robot and bisect Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index ee3b574..b71b3fe 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -4,13 +4,19 @@ # frozen_string_literal: true def build_apply_account_email(my_info) + email_greeting = if my_info['bisect'] + 'We have automatically created the following information for you.' + else + 'Thank you for joining us.' + end + email_msg = <<~EMAIL_MESSAGE To: #{my_info['my_email']} Subject: [compass-ci] Account Ready Dear #{my_info['my_name']}, - Thank you for joining us. + #{email_greeting} You can use the following info to submit jobs: -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/assign_account_client: optimize
by Luan Shengde 31 Dec '20

31 Dec '20

add new key for bisect way to apply account use rest-client instead of linux command 'curl' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign_account_client.rb | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb index d7e0623..15d3f35 100755 --- a/lib/assign_account_client.rb +++ b/lib/assign_account_client.rb @@ -7,6 +7,7 @@ require 'json' require 'mail' require 'set' require 'optparse' +require 'rest-client' require_relative '../container/defconfig' require_relative 'es_client' require_relative '../container/mail-robot/lib/assign-account-email' @@ -52,7 +53,8 @@ class AutoAssignAccount apply_info['is_update_account'] = true unless apply_info['my_login_name'].nil? apply_info['lab'] = LAB - account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" + account_info_str = RestClient.post assign_account_url, apply_info.to_json JSON.parse account_info_str end @@ -69,6 +71,7 @@ class AutoAssignAccount end def send_mail + @my_info['bisect'] = true message = build_apply_account_email(@my_info) %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") end @@ -81,3 +84,12 @@ class AutoAssignAccount send_mail end end + +my_info = { + 'my_email' => 'shdluan(a)163.com', + 'my_name' => 'lsd', + 'my_commit_url' => '' +} + +test = AutoAssignAccount.new(my_info) +test.send_account -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: send bisect report to author email
by Cao Xueliang 31 Dec '20

31 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index f0f0194..209a28d 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -18,8 +18,9 @@ class MailBisectResult @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) + @to = @git_commit.author_email # now send mail to review - @to = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com' + @bcc = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' end def create_send_email @@ -37,6 +38,7 @@ class MailBisectResult data = <<~BODY To: #{@to} + Bcc: #{@bcc} Subject: #{subject} Hi #{(a)git_commit.author_name}, -- 2.23.0

1 0

[PATCH compass-ci] delimiter: use docker to run build-pkg job
by Cao Xueliang 31 Dec '20

31 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/delimiter.rb | 2 +- src/delimiter/utils.rb | 10 ++++++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index bf36db3..1afffbc 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -25,7 +25,7 @@ class Delimiter next end - %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id'].inspect} bisect.yaml) + %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id'].inspect} bisect.yaml queue='dc-bisect') rescue StandardError => e puts e sleep(60) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 49d48ee..de5a302 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -135,10 +135,16 @@ module Utils job['my_email'] = account_info['my_email'] job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id - job['testbox'] = job['config'] ? 'vm-2p32g' : 'vm-2p16g' + job['testbox'] = job['config'] ? 'dc-16g' : 'dc-8g' + job['os'] = 'openeuler' + job['os_arch'] = 'aarch64' + job['os_mount'] = 'container' + job['os_version'] = '20.03-pre' + job['docker_image'] = 'openeuler:20.03-pre' - job.delete('error_ids') job.delete('start_time') + job.delete('tbox_group') + job.delete('error_ids') job.delete('job_state') job.delete('end_time') job.delete('loadavg') -- 2.23.0

1 0

[PATCH v4 compass-ci] doc/tutorial.md: fix abnormal display in markdown
by Lu Kaiyi 31 Dec '20

31 Dec '20

the middle part in * and * is italics for markdown document, if want show '*' in this condition, add ` in both sides of path. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- doc/tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..bf72630 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -258,7 +258,7 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM 4) 备注目前提供 archlinux 下已有 PKGBUILD 项目构建测试，暂不支持自主导入 PKGBUILD 文件。 - PKGBUILD 存放路径： /srv/git/archlinux/*/*/PKGBUILD。 + PKGBUILD 存放路径： `/srv/git/archlinux/*/*/PKGBUILD`。 ### depends -- 2.23.0

1 0

[PATCH v2 compass-ci] doc: add runtime field in borrow-*.yml and update borrow-machine.*.md
by Zhang Yale 31 Dec '20

31 Dec '20

User can specify the borrow time in "runtime" field Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 7 ++++--- doc/manual/borrow-machine.zh.md | 9 +++++---- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index bed9fcc..b598bec 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -203,6 +203,7 @@ Ensure that you have performed the following operations according to the [apply- suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 3f500e2..5288377 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -206,7 +206,8 @@ hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # 使用vim来修改你的sleep字段的值 + # 使用vim来修改你的runtime字段的值 hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # 修改完毕后重新submit即可 hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

2 1

[PATCH v3 compass-ci] sparrow/4-docker/buildall: build dist image when it doesn't exist
by Xu Xijian 31 Dec '20

31 Dec '20

If the dist docker image doesn't exist for some reason, run build script to create it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 32801d1..ca4406b 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -32,7 +32,8 @@ do_one() ( cd $container [ "$container_name" == 'ssh-r' ] && exit - if [ "$action" != "reboot" ]; then + docker images | grep -wq "$container_name" + if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/4-docker/buildall: build dist image when it doesn't exist
by Xu Xijian 31 Dec '20

31 Dec '20

If the dist docker image doesn't exist for some reason, run build script to create it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 32801d1..9e66ac9 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -32,7 +32,8 @@ do_one() ( cd $container [ "$container_name" == 'ssh-r' ] && exit - if [ "$action" != "reboot" ]; then + docker images | grep -w -q "$container_name" + if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 1

[PATCH lkp-tests] tests/build-pkg: fix to generate the directory of *.cgz file
by Liu Shaofei 31 Dec '20

31 Dec '20

[why] Since the file upload path saved by ES is inconsistent with the file path generated by the build-pkg script, the *.cgz file upload failed. [error] ==> /tmp/stdout <== renamed '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz.tmp' -> '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz' ==> Leaving fakeroot environment. ==> Finished making: kubernetes 1.18.5-1 (Wed Dec 30 16:50:17 CST 2020) create soft link: latest.cgz -> HEAD.cgz ==> /tmp/stderr <== curl: (22) The requested URL returned error: 403 Forbidden Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 801649cf4..e21397ee6 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -88,6 +88,9 @@ request_pkg() create_cgz_name() { + pkgname="${pkgname}" + [ -n "${pkgbase}" ] && pkgname="${pkgbase}" + cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" pkg_args="-A --check --skippgpcheck" -- 2.23.0

3 5

[PATCH compass-ci] doc: add runtime field in borrow-*.yml and update borrow-machine.*.md
by Zhang Yale 31 Dec '20

31 Dec '20

User can specify the borrow time in "runtime" field Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/borrow-machine.en.md | 7 ++++--- doc/manual/borrow-machine.zh.md | 9 +++++---- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/doc/manual/borrow-machine.en.md b/doc/manual/borrow-machine.en.md index bed9fcc..b598bec 100644 --- a/doc/manual/borrow-machine.en.md +++ b/doc/manual/borrow-machine.en.md @@ -203,6 +203,7 @@ Ensure that you have performed the following operations according to the [apply- suite: borrow testcase: borrow + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -212,10 +213,10 @@ Ensure that you have performed the following operations according to the [apply- %> sshd: # sleep at the bottom + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # Use the VIM editor to change the value of the sleep field. + # Use the VIM editor to change the value of the runtime field. hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # After changing the value, submit the request again. hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml diff --git a/doc/manual/borrow-machine.zh.md b/doc/manual/borrow-machine.zh.md index 3f500e2..5288377 100644 --- a/doc/manual/borrow-machine.zh.md +++ b/doc/manual/borrow-machine.zh.md @@ -206,7 +206,8 @@ hi684@account-vm ~/lkp-tests/jobs% cat borrow-1h.yaml suite: borrow testcase: borrow - + + runtime: 1h ssh_pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -216,10 +217,10 @@ %> sshd: # sleep at the bottom + sleep: + hi684@account-vm ~/lkp-tests/jobs% grep runtime: borrow-1h.yaml sleep: 1h - hi684@account-vm ~/lkp-tests/jobs% grep sleep: borrow-1h.yaml - sleep: 1h - # ��vim��sleep�� + # ��vim��runtime�� hi684@account-vm ~/lkp-tests/jobs% vim borrow-1h.yaml # ��submit�� hi684@account-vm ~/lkp-tests/jobs% submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0

[PATCH v3 lkp-tests] optimize the deploy-cci script
by Hu Xuejiao 31 Dec '20

31 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 52 ++++++++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 55835544..8e6acbac 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -2,30 +2,38 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# redirect remote-repository address to "/srv/git/gitee.com" -if [ -n "$GIT_SERVER" ]; then - cat >> /etc/gitconfig <<-EOF - [url "git://$GIT_SERVER/gitee.com"] - insteadOf=https://gitee.com - EOF -fi +export_env() +{ + export initrd_http_host=$INITRD_HTTP_HOST + export initrd_http_port=$INITRD_HTTP_PORT + export gitcache_host=$GITCACHE_HOST + export gitcache_port=$GITCACHE_PORT +} -export initrd_http_host=$INITRD_HTTP_HOST -export initrd_http_port=$INITRD_HTTP_PORT -export gitcache_host=$GITCACHE_HOST -export gitcache_port=$GITCACHE_PORT +deploy() +{ + # redirect remote-repository address to "/srv/git/gitee.com" + if [ -n "$GIT_SERVER" ]; then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF + fi -mkdir -p /c -cd /c || exit -git clone https://gitee.com/wu_fengguang/compass-ci.git + mkdir -p /c && cd /c + git clone https://gitee.com/wu_fengguang/compass-ci.git -cd /c/compass-ci || exit -[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + cd /c/compass-ci || exit + [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" -# Need mount tmpfs to fix compass-ci/container/os-nfs/start error -# "exportfs: /exports/result does not support NFS export" -mount -t tmpfs none /srv + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error + # "exportfs: /exports/result does not support NFS export" + mount -t tmpfs none /srv -# test the script for one-click deployment -cd /c/compass-ci/sparrow || exit -./install-tiny + # test the script for one-click deployment + cd /c/compass-ci/sparrow || exit + ./install-tiny +} + +export_env +deploy -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] pkg: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/httpress/PKGBUILD | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 pkg/httpress/PKGBUILD diff --git a/pkg/httpress/PKGBUILD b/pkg/httpress/PKGBUILD new file mode 100644 index 00000000..8891d6c7 --- /dev/null +++ b/pkg/httpress/PKGBUILD @@ -0,0 +1,17 @@ +pkgname=httpress +pkgver=1 +pkgrel=1 +url='http://ftp.us2.freshrpms.net/linux/RPM/epel/7/aarch64/Packages/h/httpress-1…' +arch=('aarch64') +license=('BSD') +source=("https://download.fedoraproject.org/pub/epel/7/SRPMS/Packages/h/httpress-1.1…") +md5sums=('SKIP') + +package() +{ + # The rpm from source maybe not be available + # httpress-1.1.0-13.fc31.aarch64.rpm has been validated to be available + rpmpkg_path="${pkgdir}/opt/rpms/" + mkdir -p "${rpmpkg_path}" + cp "${srcdir}/"*.rpm "${rpmpkg_path}" +} -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] distro/depends: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/httpress | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/httpress diff --git a/distro/depends/httpress b/distro/depends/httpress new file mode 100644 index 00000000..73fc57c6 --- /dev/null +++ b/distro/depends/httpress @@ -0,0 +1,2 @@ +numactl +ethtool -- 2.23.0

1 0

[PATCH compass-ci] lib/es_query.rb: parse es_result to a Hash result
by Lu Kaiyi 31 Dec '20

31 Dec '20

[why] make es result more easy access. [example] input: result_hash = {} es_result = [{"key"=>"build-pkg", "doc_count"=>354526, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>5708}, {"key"=>"finished", "doc_count"=>1033}, {"key"=>"incomplete", "doc_count"=>204}, {"key"=>"submit", "doc_count"=>136}, ... output: result_hash = {"build-pkg"=> {"failed"=>5708, "finished"=>1033, "incomplete"=>204, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1785, "failed"=>675, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 47 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 46 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 29d50e9..dcd735b 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -129,7 +129,9 @@ class ESQuery result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] return nil if result.empty? - result + result_hash = {} + parse_fields(result, result_hash) + result_hash end end @@ -217,3 +219,46 @@ def build_aggs_from_fields(fields, aggs_hash) build_aggs_from_fields(fields, aggs_hash['aggs']["all_#{field}"]) end + +# input: +# result_hash = {} +# es_result = +# [{"key"=>"build-pkg", +# "doc_count"=>354526, +# "all_job_state"=> +# {"doc_count_error_upper_bound"=>0, +# "sum_other_doc_count"=>0, +# "buckets"=> +# [{"key"=>"failed", "doc_count"=>5708}, +# {"key"=>"finished", "doc_count"=>1033}, +# {"key"=>"incomplete", "doc_count"=>204}, +# {"key"=>"submit", "doc_count"=>136}, +# ... +# +# output: +# result_hash = +# {"build-pkg"=> +# {"failed"=>5708, +# "finished"=>1033, +# "incomplete"=>204, +# "submit"=>136, +# "OOM"=>11, +# "post_run"=>2}, +# "cci-depends"=> +# {"finished"=>1785, +# "failed"=>675, +# ... +def parse_fields(es_result, result_hash) + es_result.each do |result| + key = result['key'] + sub_field = result.keys.detect { |field| field.start_with?('all_') } + + if sub_field + result_hash[key] = {} + all_field = result[sub_field]['buckets'] + parse_fields(all_field, result_hash[key]) + else + result_hash[key] = result['doc_count'] + end + end +end -- 2.23.0

2 1

[PATCH compass-ci] matrix2.rb: filter groups to make compare faster
by Li Yuanchao 31 Dec '20

31 Dec '20

When do compare, after group jobs, there are many one_size group, which can not find another job to make a group. Delete such groups to save time. But, if there is no size > 1 group, 3 one_size groups will be kept to avoid 'No Data'. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/matrix2.rb | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 14e9558..0e80858 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -123,18 +123,37 @@ end def combine_group_query_data(job_list, dims) suites_hash = {} groups = auto_group(job_list, dims) + have_multi_member = multi_member?(groups) + one_size_count = 0 groups.each do |group_key, value| - suite_list = [] - value.each do |dimension_key, jobs| - groups[group_key][dimension_key], suites = create_matrix(jobs) - suite_list.concat(suites) + if value.size < 2 + one_size_count += 1 + if have_multi_member || one_size_count > 3 + groups.delete(group_key) + next + end end - suites_hash[group_key] = suite_list + get_groups_matrix(groups, group_key, value, suites_hash) end - return groups, suites_hash end +def get_groups_matrix(groups, group_key, value, suites_hash) + suite_list = [] + value.each do |dimension_key, jobs| + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) + end + suites_hash[group_key] = suite_list +end + +def multi_member?(groups) + groups.each do |_key, value| + return true if value.size > 1 + end + return false +end + # input: # 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() # 2. params from user's template include: -- 2.23.0

2 3

[PATCH v4 compass-ci] providers/qemu: refactor kvm script
by Liu Yinsi 31 Dec '20

31 Dec '20

encapsulating into functions. test: /srv/result/host-info/2020-12-30/vm-2p8g/openeuler-20.09-aarch64/z9.275381 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 160 +++++++++++++++++++++++++----------------- 1 file changed, 97 insertions(+), 63 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fd921dc..0845099 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,78 +7,112 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + [ ! -f "$log_file" ] || { + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + } +} + +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm +} -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} +parse_ipxe_script() +{ + append= + initrds= + while read a b c + do + case "$a" in + '#') + ;; + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + *) + ;; + esac + done < $ipxe_script +} -append= -initrds= -while read a b c -do - case "$a" in - '#') - ;; - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - *) - ;; - esac -done < $ipxe_script +check_option_value() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || exit +} -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +get_initrd() +{ + initrd=initrd + cat $initrds > $initrd } -[ -n "$initrds" ] || { - exit +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 } -initrd=initrd -cat $initrds > $initrd +run_qemu() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) + + "${kvm[@]}" --append "${append}" +} -echo $SCHED_PORT -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file +check_logfile +write_logfile -sleep 5 +qemu_command +parse_ipxe_script +check_option_value +get_initrd -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) +print_message -"${kvm[@]}" --append "${append}" +run_qemu -- 2.23.0

2 2

[PATCH v2 lkp-tests] tests/build-pkg: fix to generate the directory of *.cgz file
by Liu Shaofei 31 Dec '20

31 Dec '20

[why] Since the file upload path saved by ES is inconsistent with the file path generated by the build-pkg script, the *.cgz file upload failed. [error] ==> /tmp/stdout <== renamed '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz.tmp' -> '/initrd/build-pkg/nfs/openeuler/aarch64/20.03-SP1/kubelet/HEAD.cgz' ==> Leaving fakeroot environment. ==> Finished making: kubernetes 1.18.5-1 (Wed Dec 30 16:50:17 CST 2020) create soft link: latest.cgz -> HEAD.cgz ==> /tmp/stderr <== curl: (22) The requested URL returned error: 403 Forbidden Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 801649cf4..436d43840 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -88,6 +88,8 @@ request_pkg() create_cgz_name() { + [ -n "${pkgbase}" ] && pkgname="${pkgbase}" + cgz_path="$PKG_MNT/${pack_to}/${pkgname}" cgz_name="${cgz_path}/${upstream_commit}.cgz" pkg_args="-A --check --skippgpcheck" -- 2.23.0

1 0

[PATCH lkp-tests] api-avx2neon/PKGBUILD: add multi arch build
by Wang Yong 31 Dec '20

31 Dec '20

build packages through $(arch) Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- pkg/api-avx2neon/PKGBUILD | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/api-avx2neon/PKGBUILD b/pkg/api-avx2neon/PKGBUILD index 3ccebca86..fac047add 100644 --- a/pkg/api-avx2neon/PKGBUILD +++ b/pkg/api-avx2neon/PKGBUILD @@ -9,7 +9,7 @@ md5sums=('SKIP') build() { cd "$srcdir/AvxToNeon/tests" - make + make ARCH=$(arch) } package() { -- 2.23.0

1 0

[PATCH lkp-tests 2/2] pkg: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/httpress/PKGBUILD | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 pkg/httpress/PKGBUILD diff --git a/pkg/httpress/PKGBUILD b/pkg/httpress/PKGBUILD new file mode 100644 index 00000000..b906303d --- /dev/null +++ b/pkg/httpress/PKGBUILD @@ -0,0 +1,17 @@ +pkgname=httpress +pkgver=1 +pkgrel=1 +url='' +arch=('aarch64') +license=('BSD') +source=("https://download.fedoraproject.org/pub/epel/7/SRPMS/Packages/h/httpress-1.1…") +md5sums=('SKIP') + +package() +{ + # The rpm which from source maybe unavailable + # httpress-1.1.0-13.fc31.aarch64.rpm is validated to be available + rpmpkg_path="${pkgdir}/opt/rpms/" + mkdir -p "${rpmpkg_path}" + cp "${srcdir}/"*.rpm "${rpmpkg_path}" +} -- 2.23.0

2 1

[PATCH v2 lkp-tests 2/2] pkg: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/httpress/PKGBUILD | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 pkg/httpress/PKGBUILD diff --git a/pkg/httpress/PKGBUILD b/pkg/httpress/PKGBUILD new file mode 100644 index 00000000..44af00ea --- /dev/null +++ b/pkg/httpress/PKGBUILD @@ -0,0 +1,17 @@ +pkgname=httpress +pkgver=1 +pkgrel=1 +url='' +arch=('aarch64') +license=('BSD') +source=("https://download.fedoraproject.org/pub/epel/7/SRPMS/Packages/h/httpress-1.1…") +md5sums=('SKIP') + +package() +{ + # The rpm from source maybe bot be available + # httpress-1.1.0-13.fc31.aarch64.rpm has been validated to be available + rpmpkg_path="${pkgdir}/opt/rpms/" + mkdir -p "${rpmpkg_path}" + cp "${srcdir}/"*.rpm "${rpmpkg_path}" +} -- 2.23.0

2 1

[PATCH v2 lkp-tests 1/2] distro/depends: add httpress
by Wei Jihui 31 Dec '20

31 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/httpress | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/httpress diff --git a/distro/depends/httpress b/distro/depends/httpress new file mode 100644 index 00000000..73fc57c6 --- /dev/null +++ b/distro/depends/httpress @@ -0,0 +1,2 @@ +numactl +ethtool -- 2.23.0

1 0

[PATCH v2 lkp-tests] optimize the deploy-cci script
by Hu Xuejiao 31 Dec '20

31 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 52 ++++++++++++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 22 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 55835544..8e6acbac 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -2,30 +2,38 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# redirect remote-repository address to "/srv/git/gitee.com" -if [ -n "$GIT_SERVER" ]; then - cat >> /etc/gitconfig <<-EOF - [url "git://$GIT_SERVER/gitee.com"] - insteadOf=https://gitee.com - EOF -fi +export_env() +{ + export initrd_http_host=$INITRD_HTTP_HOST + export initrd_http_port=$INITRD_HTTP_PORT + export gitcache_host=$GITCACHE_HOST + export gitcache_port=$GITCACHE_PORT +} -export initrd_http_host=$INITRD_HTTP_HOST -export initrd_http_port=$INITRD_HTTP_PORT -export gitcache_host=$GITCACHE_HOST -export gitcache_port=$GITCACHE_PORT +deploy() +{ + # redirect remote-repository address to "/srv/git/gitee.com" + if [ -n "$GIT_SERVER" ]; then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF + fi -mkdir -p /c -cd /c || exit -git clone https://gitee.com/wu_fengguang/compass-ci.git + mkdir -p /c && cd /c + git clone https://gitee.com/wu_fengguang/compass-ci.git -cd /c/compass-ci || exit -[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + cd /c/compass-ci || exit + [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" -# Need mount tmpfs to fix compass-ci/container/os-nfs/start error -# "exportfs: /exports/result does not support NFS export" -mount -t tmpfs none /srv + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error + # "exportfs: /exports/result does not support NFS export" + mount -t tmpfs none /srv -# test the script for one-click deployment -cd /c/compass-ci/sparrow || exit -./install-tiny + # test the script for one-click deployment + cd /c/compass-ci/sparrow || exit + ./install-tiny +} + +export_env +deploy -- 2.23.0

2 1

[PATCH lkp-tests 1/2] distro/depends: add httpress
by Wei Jihui 30 Dec '20

30 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/httpress | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 distro/depends/httpress diff --git a/distro/depends/httpress b/distro/depends/httpress new file mode 100644 index 00000000..73fc57c6 --- /dev/null +++ b/distro/depends/httpress @@ -0,0 +1,2 @@ +numactl +ethtool -- 2.23.0

1 0

[PATCH compass-ci] 3-code/dev-env: add variables
by Hu Xuejiao 30 Dec '20

30 Dec '20

[why] $GIT_SERVER will be defined in job.yaml $LKP_SERVER is ip in test-machine for upload_files Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 750738f..ac38f56 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -23,6 +23,8 @@ mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port +LKP_SERVER: $server_ip +GIT_SERVER: $GIT_SERVER TASKQUEUE_HOST: $sched_host OS_HTTP_HOST: $os_http_host INITRD_HTTP_HOST: $initrd_http_host -- 2.23.0

1 0

[PATCH v4 compass-ci] sbin/run_job: Automatic task execution
by Hu Xuejiao 30 Dec '20

30 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 0000000..c84df0d --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,74 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + docker_images=${single_arg[4]} + testbox=$testbox + queue="${tbox_group}.${HOSTNAME}" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/mail_bisect_result: optimize bisect report
by Cao Xueliang 30 Dec '20

30 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- doc/bisect_email.en.md | 28 ++++++++++++++++++++++++++++ lib/constants.rb | 2 ++ lib/mail_bisect_result.rb | 28 ++++++++++++++++++---------- 3 files changed, 48 insertions(+), 10 deletions(-) create mode 100644 doc/bisect_email.en.md diff --git a/doc/bisect_email.en.md b/doc/bisect_email.en.md new file mode 100644 index 0000000..fee7ee1 --- /dev/null +++ b/doc/bisect_email.en.md @@ -0,0 +1,28 @@ +# Who are we? + +Compass-CI runs a number of tests for open source projects for free. +We aim to improve OSS quality in a number of hardware and OS environment. +We are part of the openEuler community. + +# Why do you get the email? + +Whenever you git push, our test robot will pull and test the new code. +When new errors are found, Compass-CI will auto run git bisect to find out the first bad commit, +and send email report to the commit author. + +# About the email + +The first bad commit job result directory: +you can read the below file to understand the content of each file. +- https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-res… + +Notice: +You may need to read the output file of the first bad commit job result directory, +it is the build log about the first bad commit. + +# how to confirm bug fix + +- git push your fix patch to your project. +- download the job.yaml from the first bad commit job directory. +- submit -m job.yaml upstream_commit=$your_fix_commit. +- check result from the new result directory which is in output above command and key is result_root. diff --git a/lib/constants.rb b/lib/constants.rb index f0938f2..c3c7543 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -13,3 +13,5 @@ MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 + +SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5f88c36..f0f0194 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -3,6 +3,8 @@ require 'json' require 'base64' require_relative 'git' +require_relative 'es_query' +require_relative 'constants' require_relative 'mail_client' require_relative 'assign_account_client' @@ -27,10 +29,11 @@ class MailBisectResult def compose_mail subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" - prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" - bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' - pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" - first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" + prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" + bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' + bisect_report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" + pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/#{@pkgbuild_repo}\n" + first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY To: #{@to} @@ -38,17 +41,19 @@ class MailBisectResult Hi #{(a)git_commit.author_name}, - url: #{(a)git_commit.url} - commit: #{@commit_id} ("#{(a)git_commit.subject}") - compiler: gcc (GCC) 7.3.0 + We found some error/warning(s) and the first bad commit in the below project: + git url: #{(a)git_commit.url} + git commit: #{@commit_id} ("#{(a)git_commit.subject}") - all errors/warnings (new ones prefixed by >>): + All error/warning(s) (new ones prefixed by >>): #{@all_errors} - reference information: + Reference information: + compiler: gcc (GCC) 7.3.0 #{pkgbuild_repo_url} - #{bisect_job_url} #{first_bad_commit_job_url} + #{bisect_job_url} + #{bisect_report_doc} Regards, Compass CI team BODY @@ -67,6 +72,9 @@ class MailBisectResult 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" } + account_info = ESQuery.new(index: 'accounts').query_by_id(@to) + return if account_info + apply_account = AutoAssignAccount.new(user_info) apply_account.send_account end -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: optimize bisect report
by Cao Xueliang 30 Dec '20

30 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5f88c36..4883072 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -3,6 +3,7 @@ require 'json' require 'base64' require_relative 'git' +require_relative 'es_query' require_relative 'mail_client' require_relative 'assign_account_client' @@ -28,9 +29,9 @@ class MailBisectResult def compose_mail subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" - bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' - pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" - first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" + bisect_job_url = ENV['result_root'] ? "bisect job result directory: #{prefix_srv}#{ENV['result_root']}\n" : '' + pkgbuild_repo_url = "PKGBUILD file directory: #{prefix_srv}/#{@pkgbuild_repo}\n" + first_bad_commit_job_url = "first bad commit job result directory: #{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY To: #{@to} @@ -46,9 +47,9 @@ class MailBisectResult #{@all_errors} reference information: + #{first_bad_commit_job_url} #{pkgbuild_repo_url} #{bisect_job_url} - #{first_bad_commit_job_url} Regards, Compass CI team BODY @@ -67,6 +68,9 @@ class MailBisectResult 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" } + account_info = ESQuery.new(index: 'accounts').query_by_id(@to) + return if account_info + apply_account = AutoAssignAccount.new(user_info) apply_account.send_account end -- 2.23.0

1 1

[PATCH v1 compass-ci] lib/mail_bisect_result: optimize bisect report
by Cao Xueliang 30 Dec '20

30 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/constants.rb | 2 ++ lib/mail_bisect_result.rb | 28 ++++++++++++++++++---------- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index f0938f2..c3c7543 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -13,3 +13,5 @@ MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 + +SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5f88c36..bc3c74a 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -3,6 +3,8 @@ require 'json' require 'base64' require_relative 'git' +require_relative 'es_query' +require_relative 'constants' require_relative 'mail_client' require_relative 'assign_account_client' @@ -27,10 +29,11 @@ class MailBisectResult def compose_mail subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" - prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" - bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' - pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" - first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" + prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" + bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' + bisect_report_doc = "bisect report:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_r…" + pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/#{@pkgbuild_repo}\n" + first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY To: #{@to} @@ -38,17 +41,19 @@ class MailBisectResult Hi #{(a)git_commit.author_name}, - url: #{(a)git_commit.url} - commit: #{@commit_id} ("#{(a)git_commit.subject}") - compiler: gcc (GCC) 7.3.0 + We found some error/warning(s) and the first bad commit in the below project: + git url: #{(a)git_commit.url} + git commit: #{@commit_id} ("#{(a)git_commit.subject}") - all errors/warnings (new ones prefixed by >>): + All error/warning(s) (new ones prefixed by >>): #{@all_errors} - reference information: + Reference information: + compiler: gcc (GCC) 7.3.0 #{pkgbuild_repo_url} - #{bisect_job_url} #{first_bad_commit_job_url} + #{bisect_job_url} + #{bisect_report_doc} Regards, Compass CI team BODY @@ -67,6 +72,9 @@ class MailBisectResult 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" } + account_info = ESQuery.new(index: 'accounts').query_by_id(@to) + return if account_info + apply_account = AutoAssignAccount.new(user_info) apply_account.send_account end -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: add ${config} judge in common job
by Wang Yong 30 Dec '20

30 Dec '20

if not exist ${config} in general build-pkg job, ${cci_dir} should not be judeged and do not need upload file Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 801649cf4..d356b2f14 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -49,7 +49,9 @@ get_config() upload_config() { local upload_dir="${cci_dir}/${pkgname}" - [ -d "${upload_dir}" ] || return + + [ -n "${config}" ] || return + [ -d "${upload_dir}" ] || return upload_to_target_dir "${upload_dir}" } -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: import h5ai into container
by Lu Weitao 30 Dec '20

30 Dec '20

[Why] Optimize result page style to make it more modern Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/Dockerfile | 16 +++++++++++++--- container/srv-http/build | 7 +++++++ .../srv-http/root/etc/nginx/conf.d/default.conf | 5 ++--- 3 files changed, 22 insertions(+), 6 deletions(-) diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index 8f9a36d..987e40e 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -1,15 +1,25 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM nginx:alpine +FROM alpine:3.9 + ADD root / -RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo +RUN apk add --no-cache nginx php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo nodejs npm \ + && mkdir /run/nginx/ \ + && mkdir /h5ai RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf RUN sed -i "s/user = nobody/user = nginx/g" /etc/php7/php-fpm.d/www.conf RUN sed -i "s/group = nobody/group = nginx/g" /etc/php7/php-fpm.d/www.conf -RUN chown -R nginx:nginx /srv/_h5ai +WORKDIR /h5ai +COPY h5ai /h5ai + +# build h5ai package +RUN npm install \ + && npm run build \ + && cp -r ./build/_h5ai /srv \ + && chown -R nginx:nginx /srv/_h5ai ENTRYPOINT ["/sbin/entrypoint.sh"] diff --git a/container/srv-http/build b/container/srv-http/build index 3a500ea..bdda319 100755 --- a/container/srv-http/build +++ b/container/srv-http/build @@ -2,4 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[[ $H5AI_SRC ]] || H5AI_SRC=/c/h5ai + +cp -r $H5AI_SRC ./ + docker build -t srv-http . + +# clean h5ai foler +rm -rf ./h5ai diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 676418a..646eee3 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -3,8 +3,8 @@ server { server_name "srv-http"; server_tokens off; - root /usr/share/nginx/html; - index index.html; + root /srv; + index index.html index.php /_h5ai/public/index.php; charset utf-8,gbk; @@ -24,7 +24,6 @@ server { autoindex on; autoindex_localtime on; allow all; - root /srv; location ~* /[-_a-z]+$ { add_header Content-Type text/plain; } -- 2.23.0

1 1

Re: [PATCH v3 compass-ci] providers/qemu: refactor kvm script
by Ren Wen 30 Dec '20

30 Dec '20

>>+ if [ ! -f "$log_file" ]; then >>>+ touch $log_file >>>+ # fluentd refresh time is 1s >>>+ # let fluentd to monitor this file first >>>+ sleep 2 >>>+ fi >> >>how about: [ -f "$log_file" ] || { ... } ? > >there are 2 line codes, if just 1 line code use like you say is good. {} is a way to group a list of commands to be executed as a unit, you can also use like this: [ ] || { expression1 expression2 } Thanks, RenWen > >Thanks, >Yinsi > > >> >>-------- >>Thanks >>Yu Chuan >> >>>+} >>> >>>-qemu=qemu-system-aarch64 >>>-command -v $qemu >/dev/null || qemu=qemu-kvm >>>+qemu_command() >>>+{ >>>+ qemu=qemu-system-aarch64 >>>+ command -v $qemu >/dev/null || qemu=qemu-kvm >>>+} >>> >>>-ipxe_script=ipxe_script >>>-curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script >>>-cat $ipxe_script >> ${log_file} >>>+write_logfile() >>>+{ >>>+ ipxe_script=ipxe_script >>>+ curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script >>>+ cat $ipxe_script >> ${log_file} >>>+} >>> >>>-append= >>>-initrds= >>>-while read a b c >>>-do >>>- case "$a" in >>>- '#') >>>- ;; >>>- initrd) >>>- file=$(basename "$b") >>>- wget --timestamping -a ${log_file} --progress=bar:force $b >>>- initrds+="$file " >>>- ;; >>>- kernel) >>>- kernel=$(basename "$b") >>>- wget --timestamping -a ${log_file} --progress=bar:force $b >>>- append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") >>>- ;; >>>- *) >>>- ;; >>>- esac >>>-done < $ipxe_script >>>+option_value() >>>+{ >>>+ append= >>>+ initrds= >>>+ while read a b c >>>+ do >>>+ case "$a" in >>>+ '#') >>>+ ;; >>>+ initrd) >>>+ file=$(basename "$b") >>>+ wget --timestamping -a ${log_file} --progress=bar:force $b >>>+ initrds+="$file " >>>+ ;; >>>+ kernel) >>>+ kernel=$(basename "$b") >>>+ wget --timestamping -a ${log_file} --progress=bar:force $b >>>+ append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") >>>+ ;; >>>+ *) >>>+ ;; >>>+ esac >>>+ done < $ipxe_script >>>+} >>> >>>-[ -s "$kernel" ] || { >>>- echo "can't get kernel or kernel size is 0" >>>- exit >>>+check_option_value() >>>+{ >>>+ [ -s "$kernel" ] || { >>>+ echo "can't get kernel or kernel size is 0" >>>+ exit >>>+ } >>>+ >>>+ [ -n "$initrds" ] || exit >>>+ >>>+ initrd=initrd >>>+ cat $initrds > $initrd >>> } >>> >>>-[ -n "$initrds" ] || { >>>- exit >>>+print_message() >>>+{ >>>+ echo $SCHED_PORT >>>+ echo kernel: $kernel >>>+ echo initrds: $initrds >>>+ echo append: $append >>>+ echo less $log_file >>>+ >>>+ sleep 5 >>> } >>> >>>-initrd=initrd >>>-cat $initrds > $initrd >>>+run_qemu() >>>+{ >>>+ kvm=( >>>+ $qemu >>>+ -machine virt-4.0,accel=kvm,gic-version=3 >>>+ -kernel $kernel >>>+ -initrd $initrd >>>+ -smp $nr_cpu >>>+ -m $memory >>>+ -cpu Kunpeng-920 >>>+ -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd >>>+ -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} >>>+ -k en-us >>>+ -no-reboot >>>+ -nographic >>>+ -serial file:${log_file} >>>+ -monitor null >>>+ ) >>>+ >>>+ "${kvm[@]}" --append "${append}" >>>+} >>> >>>-echo $SCHED_PORT >>>-echo kernel: $kernel >>>-echo initrds: $initrds >>>-echo append: $append >>>-echo less $log_file >>>+check_logfile >>>+write_logfile >>> >>>-sleep 5 >>>+qemu_command >>>+option_value >>>+check_option_value >>> >>>-kvm=( >>>- $qemu >>>- -machine virt-4.0,accel=kvm,gic-version=3 >>>- -kernel $kernel >>>- -initrd $initrd >>>- -smp $nr_cpu >>>- -m $memory >>>- -cpu Kunpeng-920 >>>- -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd >>>- -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} >>>- -k en-us >>>- -no-reboot >>>- -nographic >>>- -serial file:${log_file} >>>- -monitor null >>>-) >>>+print_message >>> >>>-"${kvm[@]}" --append "${append}" >>>+run_qemu >>>-- >>>2.23.0 >>>

2 1

[PATCH v3 compass-ci] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 30 Dec '20

30 Dec '20

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 71 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..29d50e9 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,50 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = {} + build_aggs_from_fields(fields, aggs_hash) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +190,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# aggs_hash = {} +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields, aggs_hash) + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] = { + "all_#{field}" => { + 'terms' => { field: field, size: 1000 } + } + } + + build_aggs_from_fields(fields, aggs_hash['aggs']["all_#{field}"]) +end -- 2.23.0

1 0

[PATCH v3 compass-ci] providers/qemu: refactor kvm script
by Liu Yinsi 30 Dec '20

30 Dec '20

encapsulating into functions. test: /srv/result/host-info/2020-12-30/vm-2p8g/openeuler-20.09-aarch64/z9.275242 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 156 +++++++++++++++++++++++++----------------- 1 file changed, 93 insertions(+), 63 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fd921dc..c48c0c7 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,78 +7,108 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + if [ ! -f "$log_file" ]; then + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + fi +} -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm +} -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} -append= -initrds= -while read a b c -do - case "$a" in - '#') - ;; - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - *) - ;; - esac -done < $ipxe_script +option_value() +{ + append= + initrds= + while read a b c + do + case "$a" in + '#') + ;; + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + *) + ;; + esac + done < $ipxe_script +} -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +check_option_value() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || exit + + initrd=initrd + cat $initrds > $initrd } -[ -n "$initrds" ] || { - exit +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 } -initrd=initrd -cat $initrds > $initrd +run_qemu() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) + + "${kvm[@]}" --append "${append}" +} -echo $SCHED_PORT -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file +check_logfile +write_logfile -sleep 5 +qemu_command +option_value +check_option_value -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) +print_message -"${kvm[@]}" --append "${append}" +run_qemu -- 2.23.0

3 4

[PATCH v2 compass-ci] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 30 Dec '20

30 Dec '20

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example1] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... [example2] input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... [example3] input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 52 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..a9b651e 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,45 @@ class ESQuery result end + + # select field1, field2, count(*) from index where query_items group by field1, field2 + # example: + # fields = ['suite', 'job_state'] + # query_items = { + # 'os' => 'openeuler' + # } + # input: + # fields, query_items + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = {} + build_aggs_from_fields(fields, aggs_hash) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +185,16 @@ def assign_desc_body(keyword) }] } end + +def build_aggs_from_fields(fields, aggs_hash) + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] = { + "all_#{field}" => { + 'terms' => { field: field, size: 1000 } + } + } + + build_aggs_from_fields(fields, aggs_hash['aggs']["all_#{field}"]) +end -- 2.23.0

2 1

[PATCH lkp-tests] distro/common: fix the scenario when the directory not exist
by Liu Shaofei 30 Dec '20

30 Dec '20

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- distro/common | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/distro/common b/distro/common index 166ff8550..1fd7df4e8 100755 --- a/distro/common +++ b/distro/common @@ -87,8 +87,9 @@ upload_to_target_dir() . $LKP_SRC/lib/upload.sh local dir="$1" - local file="$(ls -F ${dir} | grep -v "@")" + [ -d "${dir}" ] || return 0 + local file="$(ls -F ${dir} | grep -v "@")" [ -z "${file}" ] && return 0 upload_one_curl "${dir}/${file}" "${dir}" >/dev/null -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/monitor: print ssh info
by Wu Zhende 30 Dec '20

30 Dec '20

After the user logs out, can copy this printed information to log in again. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 283cd788d..12f5280f3 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -87,8 +87,11 @@ class Monitor def ssh_connect(ssh_host, ssh_port, web_socket) web_socket.close - exec "ssh-keygen -R #{ssh_host}; - ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" + cmd = "ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" + puts cmd + + cmd = "ssh-keygen -R #{ssh_host};" + cmd + exec cmd end def stop(data, web_socket, code = 1000, reason = 'normal') -- 2.23.0

1 0

[PATCH compass-ci 3/3] monitoring/filter.cr: fix send message error
by Wu Zhende 30 Dec '20

30 Dec '20

filter message error: Error writing to socket: Broken pipe Exception: Error writing to socket: Broken pipe (Errno) from /usr/lib/crystal/core/socket.cr:82:13 in 'unbuffered_write' from /usr/lib/crystal/core/io/buffered.cr:192:5 in 'flush' from /usr/lib/crystal/core/io/buffered.cr:200:5 in 'close' from /usr/lib/crystal/core/http/server/handlers/websocket_handler.cr:44:9 in 'call' from /usr/share/crystal/app/lib/kemal/src/kemal/websocket.cr:11:7 in 'call' from /usr/share/crystal/app/lib/kemal/src/kemal/websocket_handler.cr:14:7 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/static_file_handler.cr:64:9 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/log_handler.cr:8:37 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/init_handler.cr:12:7 in 'call' from /usr/lib/crystal/core/http/server/request_processor.cr:48:11 in 'process' from /usr/lib/crystal/core/http/server/request_processor.cr:22:3 in 'process' from /usr/lib/crystal/core/http/server.cr:497:5 in 'handle_client' from /usr/lib/crystal/core/http/server.cr:463:13 in '->' from /usr/lib/crystal/core/fiber.cr:255:3 in 'run' from /usr/lib/crystal/core/fiber.cr:48:34 in '->' from ??? Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 5d3bc8b..79af3ab 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -41,6 +41,9 @@ class Filter @hash[query].each do |socket| socket.send msg.to_json + rescue e + puts "send msg failed: #{e}" + remove_filter_rule(query, socket) end end -- 2.23.0

1 0

[PATCH compass-ci 2/3] monitoring/monitoring.cr: change rabbitmq queue
by Wu Zhende 30 Dec '20

30 Dec '20

Get logs from different rabbitmq queues. And use multi-process to improve service processing capabilities. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/monitoring.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/monitoring/monitoring.cr b/src/monitoring/monitoring.cr index d09c363..8d52d46 100644 --- a/src/monitoring/monitoring.cr +++ b/src/monitoring/monitoring.cr @@ -9,7 +9,8 @@ module Monitoring message_queue_client = MessageQueueClient.new - spawn message_queue_client.monitoring_message_queue(filter, "logging-test", "logging-test") + spawn message_queue_client.monitoring_message_queue(filter, "serial-logging", "serial-logging") + spawn message_queue_client.monitoring_message_queue(filter, "docker-logging", "docker-logging") ws "/filter" do |socket| query = JSON::Any.new("") -- 2.23.0

1 0

[PATCH compass-ci 1/3] master-fluentd/docker-fluentd.conf: send logs to different mq queue
by Wu Zhende 30 Dec '20

30 Dec '20

Send serial logs to serial-logging queue. Send docker logs to docker-logging queue. The two do not affect each other. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index 7bf2229..410f2eb 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -18,7 +18,7 @@ <match serial.**> @type rabbitmq host 172.17.0.1 - exchange logging-test + exchange serial-logging exchange_type fanout exchange_durable false heartbeat 10 @@ -57,7 +57,7 @@ <store> @type rabbitmq host 172.17.0.1 - exchange logging-test + exchange docker-logging exchange_type fanout exchange_durable false heartbeat 10 -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: print ssh info
by Wu Zhende 30 Dec '20

30 Dec '20

After the user logs out, can copy this printed information to log in again. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/monitor.rb b/lib/monitor.rb index 283cd788d..522df713f 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -86,6 +86,7 @@ class Monitor def ssh_connect(ssh_host, ssh_port, web_socket) web_socket.close + puts "ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" exec "ssh-keygen -R #{ssh_host}; ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" -- 2.23.0

2 1

[PATCH v2 compass-ci] providers/qemu: encapsulated into functions
by Liu Yinsi 30 Dec '20

30 Dec '20

use function better. test: /srv/result/host-info/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/z9.273758 ssssss-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 163 ++++++++++++++++++++++++++---------------- 1 file changed, 100 insertions(+), 63 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fd921dc..6c4ad00 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,78 +7,115 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + if [ ! -f "$log_file" ]; then + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + fi +} -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm +} -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} + +option_value() +{ + append= + initrds= + while read a b c + do + case "$a" in + '#') + ;; + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + *) + ;; + esac + done < $ipxe_script +} -append= -initrds= -while read a b c -do - case "$a" in - '#') - ;; - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - *) - ;; - esac -done < $ipxe_script +check_option_value() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || { + exit + } + + initrd=initrd + cat $initrds > $initrd +} + +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 +} -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +qemu_options() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) } -[ -n "$initrds" ] || { - exit +run_qemu() +{ + "${kvm[@]}" --append "${append}" } -initrd=initrd -cat $initrds > $initrd +check_logfile +write_logfile -echo $SCHED_PORT -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file +qemu_command +option_value +check_option_value -sleep 5 +print_message -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) +qemu_options -"${kvm[@]}" --append "${append}" +run_qemu -- 2.23.0

3 4

[PATCH v3 compass-ci] sbin/run_job: Automatic task execution
by Hu Xuejiao 30 Dec '20

30 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 0000000..c6cf9e8 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="${tbox_group}.${HOSTNAME}" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

2 1

[PATCH compass-ci] container/assist-result: fix failed to start
by Liu Yinsi 30 Dec '20

30 Dec '20

[error] docker: invalid publish opts format (should be name=value but got ':'). [why] $ASSIST_RESULT_PORT not set default value, it's null when deploy compass-ci. [how] set default value ASSIST_RESULT_PORT=8102 in 3-code/dev-env. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 750738f..ffaa6e1 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,6 +18,7 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${SEND_MAIL_PORT:=49000} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${ASSIST_RESULT_PORT:=8102} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -39,6 +40,7 @@ SEND_MAIL_PORT: $SEND_MAIL_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH compass-ci] mail-robot: fix bug
by Luan Shengde 30 Dec '20

30 Dec '20

[why] when received email_file's from like "Xing Ming", add has double quotes with it, the email message may end unexpectedly From: compass-ci-reply(a)qq.com To: shdluan(a)163.com Subject: [compass-ci] Account Ready Dear Luan ----email end [how] delete extra double quotes Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 8c55d8f..7024731 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -36,7 +36,7 @@ class ParseApplyAccountEmail @my_info = { 'my_email' => mail_content.from[0], - 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '').gsub(/"/, ''), 'my_ssh_pubkey' => [] } end -- 2.23.0

2 2

[PATCH v2 compass-ci] mail-robot: fix bug
by Luan Shengde 30 Dec '20

30 Dec '20

[why] parsed email address for received email file may like: "Xing Ming <email_address(a)163.com>" or "\"Xing Ming\" <eamil_address(a)163.com>" if the email address contains double quotes as part of string, the email message may end unexpectedly, like: Date: Wed, 23 Dec 2020 18:32:06 +0800^M From: compass-ci-reply(a)qq.com To: shdluan(a)163.com Subject: [compass-ci] Account Ready^M Dear Luan ====email message ends [how] delete extra double quotes Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 8c55d8f..7024731 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -36,7 +36,7 @@ class ParseApplyAccountEmail @my_info = { 'my_email' => mail_content.from[0], - 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '').gsub(/"/, ''), 'my_ssh_pubkey' => [] } end -- 2.23.0

2 2