April 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/2] container/srv-http: add /srv/initrd/osimage
by Liu Yinsi 14 Apr '21

14 Apr '21

for user locally deploy compass-ci, can download osimage to run qemu kvm when os_mount=initramfs, and add env SRV_HTTP_INITRD_PORT Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/srv-http/start-initrd | 1 + sparrow/3-code/dev-env | 2 ++ 2 files changed, 3 insertions(+) diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 93be548..457465b 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -14,6 +14,7 @@ cmd=( -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro + -v /srv/initrd/osimage:/srv/initrd/osimage:ro -v /etc/localtime:/etc/localtime:ro -d srv-http diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 3aeba90..bb576c7 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,6 +18,7 @@ server_name=$(hostname | cut -f1 -d.) : ${srv_http_git_host:=$server_ip} : ${srv_http_cci_host:=$server_ip} : ${SRV_HTTP_RESULT_PORT:=20007} +: ${SRV_HTTP_INITRD_PORT:=20008} : ${SRV_HTTP_OS_PORT:=20009} : ${SRV_HTTP_GIT_PORT:=20010} : ${SRV_HTTP_CCI_PORT:=20011} @@ -92,6 +93,7 @@ SRV_HTTP_OS_HOST: $srv_http_os_host SRV_HTTP_GIT_HOST: $srv_http_git_host SRV_HTTP_CCI_HOST: $srv_http_cci_host SRV_HTTP_RESULT_PORT: $SRV_HTTP_RESULT_PORT +SRV_HTTP_INITRD_PORT: $SRV_HTTP_INITRD_PORT SRV_HTTP_OS_PORT: $SRV_HTTP_OS_PORT SRV_HTTP_GIT_PORT: $SRV_HTTP_GIT_PORT SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT -- 2.23.0

2 1

[PATCH compass-ci 2/2] sbin: add download-osimage script
by Liu Yinsi 14 Apr '21

14 Apr '21

for user locally deploy compass-ci, if testbox=vm-*, default os_mount=initramfs, so need to use osimage instead of os rootfs, delete download-rootfs script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-osimage | 36 ++++++++++++++++++++++++++++++++++++ sbin/download-rootfs | 40 ---------------------------------------- sparrow/5-build/os-ready | 2 +- 3 files changed, 37 insertions(+), 41 deletions(-) create mode 100755 sbin/download-osimage delete mode 100755 sbin/download-rootfs diff --git a/sbin/download-osimage b/sbin/download-osimage new file mode 100755 index 0000000..f041846 --- /dev/null +++ b/sbin/download-osimage @@ -0,0 +1,36 @@ +#!/bin/bash +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download osimage. +# 1. download default osimage openeuler aarch64 20.03 +# example usage: +# ./download-osimage +# 2. download osimage you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-osimage debian aarch64 sid + +os=${1:-openeuler} +os_arch=${2:-$(arch)} +os_version=${3:-20.03} + +. ${CCI_SRC}/container/defconfig.sh + +load_cci_defaults + +rootfs_dir=/initrd/osimage/${os}/${os_arch}/${os_version} + +download_osimage() +{ + mkdir -p /srv/$rootfs_dir + cd $rootfs_dir + + wget http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/$rootfs_dir/cur… + wget http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/$rootfs_dir/run… +} + +is_virtual() +{ + dmidecode -s system-product-name | grep -iq "virtual" +} + +is_virtual || download_osimage diff --git a/sbin/download-rootfs b/sbin/download-rootfs deleted file mode 100755 index f8c245d..0000000 --- a/sbin/download-rootfs +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash -# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. - -# download os rootfs. -# 1. download default os rootfs openeuler aarch64 20.03 -# example usage: -# ./download-rootfs -# 2. download os rootfs you want by add params $os, $os_arch, $os_version. -# example usage: -# ./download-rootfs debian aarch64 sid - -os=${1:-openeuler} -os_arch=${2:-$(arch)} -os_version=${3:-20.03} - -. ${CCI_SRC}/container/defconfig.sh -. ${CCI_SRC}/lib/log.sh - -load_cci_defaults - -rootfs_dir=/srv/os/${os}/${os_arch}/${os_version} - -download_os_rootfs() -{ - mkdir -p $rootfs_dir - cd $rootfs_dir - - download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" - wget $download_url - - [ "$?" -ne 0 ] && { - log_error "failed to download os rootfs: $download_url" - exit 1 - } - - gzip -dc ${os_version}.cgz | cpio -id - rm -rf ${os_version}.cgz -} - -download_os_rootfs diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready index 5e1acc4..d4ed2b8 100755 --- a/sparrow/5-build/os-ready +++ b/sparrow/5-build/os-ready @@ -4,4 +4,4 @@ . $CCI_SRC/lib/check_public_network.sh -public_network_ok && $CCI_SRC/sbin/download-rootfs +public_network_ok && $CCI_SRC/sbin/download-osimage -- 2.23.0

2 2

[PATCH compass-ci] sbin: add config account yaml script
by Cao Xueliang 14 Apr '21

14 Apr '21

delimiter/delimiter.cr needs it to create delimiter account. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/config_account_yaml.rb | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 sbin/config_account_yaml.rb diff --git a/sbin/config_account_yaml.rb b/sbin/config_account_yaml.rb new file mode 100755 index 0000000..a555276 --- /dev/null +++ b/sbin/config_account_yaml.rb @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "../lib/config_account" + +config_yaml ARGV[0] -- 2.23.0

1 0

[PATCH compass-ci] container/etcd: use a scheduled task for compacting in etcd
by Cao Xueliang 14 Apr '21

14 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/etcd/Dockerfile | 6 ++++++ container/etcd/entry.sh | 8 ++++++++ container/etcd/etcd_compact.sh | 7 +++++++ container/etcd/start | 2 +- 4 files changed, 22 insertions(+), 1 deletion(-) create mode 100755 container/etcd/entry.sh create mode 100755 container/etcd/etcd_compact.sh diff --git a/container/etcd/Dockerfile b/container/etcd/Dockerfile index b3957e8..f4746ab 100644 --- a/container/etcd/Dockerfile +++ b/container/etcd/Dockerfile @@ -14,3 +14,9 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou rm -Rf etcd-${VERSION}-linux-arm64* /var/cache/apk/* VOLUME /data + +COPY etcd_compact.sh / +COPY entry.sh / +RUN echo '* 0 * * * /test.sh > /etcd_compact.log 2>&1' >> /var/spool/cron/crontabs/root + +ENTRYPOINT ["/entry.sh"] diff --git a/container/etcd/entry.sh b/container/etcd/entry.sh new file mode 100755 index 0000000..4bfea1e --- /dev/null +++ b/container/etcd/entry.sh @@ -0,0 +1,8 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +crond +/bin/etcd +exec "$@" + diff --git a/container/etcd/etcd_compact.sh b/container/etcd/etcd_compact.sh new file mode 100755 index 0000000..acc20d0 --- /dev/null +++ b/container/etcd/etcd_compact.sh @@ -0,0 +1,7 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +rev=$(etcdctl endpoint status --write-out="json"|egrep -o '"revision":[0-9]*'|egrep -o '[0-9].*') +etcdctl compact $rev && etcdctl defrag + diff --git a/container/etcd/start b/container/etcd/start index db94833..6d32cd2 100755 --- a/container/etcd/start +++ b/container/etcd/start @@ -14,10 +14,10 @@ cmd=( -p 4001:4001 -p 7001:7001 -v /srv/etcd/dir:/data + -v /etc/localtime:/etc/localtime:ro -e ETCD_UNSUPPORTED_ARCH=arm64 --name etcd-server etcd-v3.4.15:alpine - /bin/etcd --name node-1 --data-dir /data --initial-cluster-state new -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit: set default values
by Liu Yinsi 14 Apr '21

14 Apr '21

before: submit host-info.yaml os_arch=x86_64 arch=x86_64 after : submit host-info.yaml [why] when submit job in x86_64 machine, must execute like this: `submit host-info.yaml os_arch=x86_64 arch=x86_64`, then the job can be consumed by docker testbox, so add default values, it will be more convenient. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/submit | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index d2d9bec76..47b377001 100755 --- a/sbin/submit +++ b/sbin/submit @@ -10,7 +10,10 @@ require "#{LKP_SRC}/lib/scheduler_client" require 'optparse' require 'yaml' -opt_set_key_value = {} +opt_set_key_value = { + 'os_arch' => `arch`.chomp, + 'arch' => `arch`.chomp +} opt_cmdline_defaults = {} opt_output_dir = nil opt_auto_define_files = false -- 2.23.0

3 4

[PATCH compass-ci] container/dracut-initrd: set yes for choices when pvcreate and vgcreate
by Xu Xijian 13 Apr '21

13 Apr '21

[why] When create physical volume, some choices need to be made, we should set it as yes(-y) to keep going instead of skipping [errlog] [ 260.644037] dracut-pre-mount[4008]: /bin/dracut-pre-mount@50(sync_src_lv): lvm pvcreate /dev/sda [ 260.664078] dracut-pre-mount[4008]: WARNING: ext4 signature detected on /dev/sda at offset 1080. Wipe it? [y/n]: [n] [ 260.684036] dracut-pre-mount[4008]: Aborted wiping of ext4. [ 260.700034] dracut-pre-mount[4008]: 1 existing signature left on the device. [ 260.716076] dracut-pre-mount[4008]: /bin/dracut-pre-mount@50(sync_src_lv): reboot Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 2a07270..9e83742 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -29,13 +29,13 @@ sync_src_lv() { # ensure the physical disk has been initialized as physical volume real_pv_device="$(lvm pvs | grep -w $pv_device | awk '{print $1}')" [ "$real_pv_device" = "$pv_device" ] || { - lvm pvcreate "$pv_device" || reboot + lvm pvcreate -y "$pv_device" || reboot } # ensure the volume group $vg_name exists real_vg_name="$(lvm pvs | grep -w $vg_name | awk '{print $2}')" [ "$real_vg_name" = "$vg_name" ] || { - lvm vgcreate "$vg_name" "$pv_device" || reboot + lvm vgcreate -y "$vg_name" "$pv_device" || reboot } } -- 2.23.0

1 0

[PATCH compass-ci] es-jobs: reuse results of other programs
by Wu Zhende 13 Apr '21

13 Apr '21

[Why] es-jobs's result may need use in other programs. But it's hard to get it right now. [How] use: es_jobs.generate_result can get the hash result Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 6 +++++- sbin/es-jobs | 1 + 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index b0d0fb2..25e54b4 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -168,6 +168,10 @@ class ESJobs end def output + output_yaml('', @result) + end + + def generate_result if @jobs.empty? puts "No query result is found: #{@es_query}" return @@ -175,6 +179,6 @@ class ESJobs @result = query_jobs_state(@jobs) @result['kvcount'] = @result['kvcount'].sort.to_h @result['raw.id'] = @result['raw.id'].sort.to_h - output_yaml('', @result) + @result end end diff --git a/sbin/es-jobs b/sbin/es-jobs index ff0d540..6d17d6b 100755 --- a/sbin/es-jobs +++ b/sbin/es-jobs @@ -52,4 +52,5 @@ items = parse_argv raise 'Please enter a query' if items.empty? es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter) +es_jobs.generate_result es_jobs.output -- 2.23.0

1 0

[PATCH compass-ci] lib/sched.cr: change testbox's queues
by Wu Zhende 13 Apr '21

13 Apr '21

[Before] ["sched/dc-8g.aarch64/ready"] [After] ["dc-8g.aarch64"] [Why] "sched/.*/ready" is a fixed format. The length of each column on kibana is limited, and removing the fixed format can also highlinght the point. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f62f83e..35315ce 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -158,6 +158,16 @@ class Sched spawn mq_publish_confirm(JOB_MQ, mq_msg.to_json) end + # input: ["sched/dc-8g.aarch64/ready"] + # output: ["dc-8g.aarch64"] + def fetch_queues(queues) + new_queues = [] of String + queues.each do |queue| + new_queues << queue.split("/")[1] + end + new_queues + end + def set_lifecycle(job, testbox, queues) if job deadline = job.get_deadline @@ -172,7 +182,7 @@ class Sched end type = get_type(testbox) - queues = JSON.parse(queues.to_json) + queues = JSON.parse(fetch_queues(queues).to_json) hash = { "job_id" => job_id, "state" => state, -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow: call public_network_ok
by Liu Yinsi 13 Apr '21

13 Apr '21

[why] For user locally deploy Compass CI, if public network not ok, they will install pkg by themselves, in order to fix error message, so call function public_network_ok(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 8 ++++++-- sparrow/0-package/os/openEuler | 14 +++++++++----- sparrow/5-build/os-ready | 4 +++- 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 9fdb4be..eb86b24 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,8 +2,12 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && { + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ + gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +} grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 9cb7b04..e742fa1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. $CCI_SRC/lib/check_public_network.sh + pkgs=( vim git @@ -26,10 +28,12 @@ pkgs=( sudo ) -yum install -y --skip-broken "${pkgs[@]}" +public_network_ok && { + yum install -y --skip-broken "${pkgs[@]}" -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm -rm -rf *.rpm + rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm + rm -rf *.rpm +} diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready index 48a740b..5e1acc4 100755 --- a/sparrow/5-build/os-ready +++ b/sparrow/5-build/os-ready @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -$CCI_SRC/sbin/download-rootfs +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && $CCI_SRC/sbin/download-rootfs -- 2.23.0

2 4

[PATCH compass-ci] lib/git_mirror.rb: retry es query
by Li Yuanchao 13 Apr '21

13 Apr '21

[errmsg] git-mirror: /usr/lib/ruby/2.5.0/socket.rb:452:in `__read_nonblock': Connection reset by peer (Faraday::ConnectionFailed) Sometimes the error above occurs, and container git-mirror restart. It's not a stable error, and next time es query can be success. So retry here. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bcdb077..6100d6b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -432,7 +432,14 @@ class MirrorMain query = { query: { match: { _id: git_repo } } } return fork_stat unless @es_client.count(index: 'repo', body: query)['count'].positive? - result = @es_client.search(index: 'repo', body: query)['hits'] + begin + result = @es_client.search(index: 'repo', body: query)['hits'] + rescue StandardError + puts $ERROR_INFO + sleep 1 + retry + end + fork_stat.each_key do |key| fork_stat[key] = result['hits'][0]['_source'][key.to_s] || fork_stat[key] end -- 2.23.0

1 0