- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 4/4] sparrow/4-docker: optimise function log_info() calling
by Liu Yinsi 05 Feb '21

05 Feb '21

before: just execute $container/build script, echo "build $container" after: only execute $container/build -> `docker build -t`, echo "build $container" [why] when execute `sparrow/4-docker/buildall run-only`, will execute $container/build script, but if image already exist, actually will not build container, it will exit code "1", so only exit code "0", call log_info "finish build $container" Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 83ecc9f..b24eca5 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -29,13 +29,9 @@ do_one_build() mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" - - log_info "start build $container." - [ -x build ] && ./build + [ "$?" == 0 ] && log_info "finish build $container" [ -x install ] && ./install - - log_info "finish build $container." ) lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

1 0

[PATCH compass-ci 3/4] sparrow/4-docker: export env DOCKER_SKIP_REBUILD
by Liu Yinsi 05 Feb '21

05 Feb '21

if env DOCKER_SKIP_REBUILD exists, then check image exists in $container/build script by call docker_skip_rebuild(), then not rebuild container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index d4e9419..83ecc9f 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -70,11 +70,9 @@ do container_name=$(basename $dir) [ "$container_name" == 'ssh-r' ] && continue - docker images | grep -wq "$container_name" - if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then - build_depends $dir & - wait - fi + [ "$action" == "run-only" ] && export DOCKER_SKIP_REBUILD=DOCKER_SKIP_REBUILD + build_depends $dir & + wait start_depends $dir done -- 2.23.0

1 0

[PATCH compass-ci 2/4] container: call docker_skip_rebuild()
by Liu Yinsi 05 Feb '21

05 Feb '21

prevent rebuild container when execute `sparrow/4-docker/buildall run-only` Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/add-software/build | 4 ++++ container/alpine/build | 4 ++++ container/archlinux/build | 4 ++++ container/assign-account/build | 4 ++++ container/assist-result/build | 4 ++++ container/assistant/build | 4 ++++ container/auto-submit/build | 4 ++++ container/conserver/build | 4 ++++ container/crystal-base/build | 4 ++++ container/crystal-compiler/build | 4 ++++ container/crystal-shards/build | 3 +++ container/debian/build | 4 ++++ container/delimiter/build | 2 ++ container/dnsmasq/build | 4 ++++ container/dracut-initrd/build | 4 ++++ container/es/build | 2 ++ container/extract-stats/build | 4 ++++ container/fetch-mail/build | 4 ++++ container/fluentd-base/build | 4 ++++ container/git-daemon/build | 4 ++++ container/git-mirror/build | 4 ++++ container/gitcache/build | 4 ++++ container/initrd-cifs/build | 4 ++++ container/initrd-http/build | 4 ++++ container/kibana/build | 4 ++++ container/lifecycle/build | 2 ++ container/lkp-initrd/build | 4 ++++ container/logging-es/build | 2 ++ container/mail-robot/build | 4 ++++ container/manjaro/build | 4 ++++ container/master-fluentd/build | 4 ++++ container/minio/build | 4 ++++ container/monitoring/build | 2 ++ container/netdata-slave/build | 4 ++++ container/netdata/build | 4 ++++ container/ntp-server/build | 4 ++++ container/openresty-proxy-cache/build | 4 ++++ container/os-cifs/build | 4 ++++ container/os-http/build | 4 ++++ container/os-nfs/build | 4 ++++ container/qcow2rootfs/build | 4 ++++ container/qemu-efi/build | 4 ++++ container/rabbitmq/build | 4 ++++ container/redis/build | 4 ++++ container/register-accounts/build | 4 ++++ container/registry/build | 4 ++++ container/remote-git/build | 4 ++++ container/result-cifs/build | 4 ++++ container/result-webdav/build | 4 ++++ container/rsync-server/build | 4 ++++ container/scheduler-dev/build | 4 ++++ container/scheduler/build | 2 ++ container/send-internet-mail/build | 4 ++++ container/send-mail/build | 4 ++++ container/shellcheck/build | 4 ++++ container/srv-http/build | 4 ++++ container/ssh-r/build | 4 ++++ container/sub-fluentd/build | 4 ++++ container/submit/build | 4 ++++ container/taskqueue/build | 2 ++ container/upload-libvirt-xml/build | 4 ++++ container/web-backend-nginx/build | 4 ++++ container/web-backend/build | 4 ++++ container/webhook/build | 4 ++++ 64 files changed, 241 insertions(+) diff --git a/container/add-software/build b/container/add-software/build index 234c8b9..bc3c95a 100755 --- a/container/add-software/build +++ b/container/add-software/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "addsoftware:auto" + docker build -t addsoftware:auto . diff --git a/container/alpine/build b/container/alpine/build index bd277c3..3168799 100755 --- a/container/alpine/build +++ b/container/alpine/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:testbed" + docker build -t alpine:testbed . diff --git a/container/archlinux/build b/container/archlinux/build index 9749489..6db5a68 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "archlinux:testbed" + docker build --build-arg ARCH=$(arch) -t archlinux:testbed . diff --git a/container/assign-account/build b/container/assign-account/build index 3984341..480c410 100755 --- a/container/assign-account/build +++ b/container/assign-account/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "assign-account:latest" + docker build -t assign-account:latest . diff --git a/container/assist-result/build b/container/assist-result/build index 130127b..600b80a 100755 --- a/container/assist-result/build +++ b/container/assist-result/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "assist-result" + docker build -t assist-result . diff --git a/container/assistant/build b/container/assistant/build index 7778c9f..89a8a97 100755 --- a/container/assistant/build +++ b/container/assistant/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:assistant" + docker build -t debian:assistant . diff --git a/container/auto-submit/build b/container/auto-submit/build index 96383e3..48ff111 100755 --- a/container/auto-submit/build +++ b/container/auto-submit/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:auto-submit" + docker build -t alpine:auto-submit . diff --git a/container/conserver/build b/container/conserver/build index 0700c5e..a7e3867 100755 --- a/container/conserver/build +++ b/container/conserver/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "conserver_server" + ./generate_conserver.rb || exit 1 docker build -t conserver_server . diff --git a/container/crystal-base/build b/container/crystal-base/build index 4df65a5..642e190 100755 --- a/container/crystal-base/build +++ b/container/crystal-base/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:crystal-base" + if [ -n "$SQUID_HOST" ] && [ -n "$SQUID_PORT" ]; then proxy="http://$SQUID_HOST:$SQUID_PORT/" else diff --git a/container/crystal-compiler/build b/container/crystal-compiler/build index cee43eb..570fe98 100755 --- a/container/crystal-compiler/build +++ b/container/crystal-compiler/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:crystal-compiler" + docker build -t alpine:crystal-compiler . diff --git a/container/crystal-shards/build b/container/crystal-shards/build index 784f571..7c2a712 100755 --- a/container/crystal-shards/build +++ b/container/crystal-shards/build @@ -6,6 +6,9 @@ # If job.yaml does not have these two parameters, git will use the local proxy cache, # GITCACHE_HOST defaults to the local ip, GITCACHE_PORT default port is 5000 . ../defconfig.sh + +docker_skip_rebuild "alpine:crystal-shards" + load_cci_defaults docker build --build-arg GITCACHE_HOST=$GITCACHE_HOST --build-arg GITCACHE_PORT=$GITCACHE_PORT -t alpine:crystal-shards . diff --git a/container/debian/build b/container/debian/build index 6bff579..eac9af2 100755 --- a/container/debian/build +++ b/container/debian/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:testbed" + docker build -t debian:testbed . diff --git a/container/delimiter/build b/container/delimiter/build index f7f53e7..606149d 100755 --- a/container/delimiter/build +++ b/container/delimiter/build @@ -4,6 +4,8 @@ . ../defconfig.sh +docker_skip_rebuild "delimiter" + load_cci_defaults docker build -t delimiter . diff --git a/container/dnsmasq/build b/container/dnsmasq/build index 8a22575..b5738bc 100755 --- a/container/dnsmasq/build +++ b/container/dnsmasq/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "dnsmasq:alpine" + [[ -e /tftpboot/boot.ipxe ]] || { cp tftpboot/boot.ipxe /tftpboot/boot.ipxe } diff --git a/container/dracut-initrd/build b/container/dracut-initrd/build index 181b896..f02cea9 100755 --- a/container/dracut-initrd/build +++ b/container/dracut-initrd/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:dracut" + docker build -t debian:dracut . diff --git a/container/es/build b/container/es/build index 17a225c..c7d7115 100755 --- a/container/es/build +++ b/container/es/build @@ -5,6 +5,8 @@ require_relative '../defconfig.rb' +docker_skip_rebuild "es643b:alpine311" + available_memory = get_available_memory system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." diff --git a/container/extract-stats/build b/container/extract-stats/build index 1aa9982..c315f0c 100755 --- a/container/extract-stats/build +++ b/container/extract-stats/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "extract-stats" + CCI_SRC=$CCI_SRC bash $CCI_SRC/container/compile . diff --git a/container/fetch-mail/build b/container/fetch-mail/build index e72bed8..9b1a501 100755 --- a/container/fetch-mail/build +++ b/container/fetch-mail/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "fetch-mail:latest" + docker build -t fetch-mail:latest . diff --git a/container/fluentd-base/build b/container/fluentd-base/build index 32e0a38..72b85bf 100755 --- a/container/fluentd-base/build +++ b/container/fluentd-base/build @@ -3,4 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "fluentd-base:alpine" + system 'docker build -t fluentd-base:alpine .' diff --git a/container/git-daemon/build b/container/git-daemon/build index d81e4b9..5ba9a52 100755 --- a/container/git-daemon/build +++ b/container/git-daemon/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "apline311:git-daemon" + docker build -t apline311:git-daemon . diff --git a/container/git-mirror/build b/container/git-mirror/build index 66967e4..c302ac7 100755 --- a/container/git-mirror/build +++ b/container/git-mirror/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine38:git-mirror" + docker build -t alpine38:git-mirror . diff --git a/container/gitcache/build b/container/gitcache/build index 5135f7c..d7f3523 100755 --- a/container/gitcache/build +++ b/container/gitcache/build @@ -3,4 +3,8 @@ # Copyright (C) 2019-2020 Eric D.little51 # SPDX-License-Identifier: GPL-3.0 +. ../defconfig.sh + +docker_skip_rebuild "gitcache" + docker build -t gitcache . diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build index 4b93647..19d5125 100755 --- a/container/initrd-cifs/build +++ b/container/initrd-cifs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "initrd-cifs" + docker build -t initrd-cifs . diff --git a/container/initrd-http/build b/container/initrd-http/build index 1b039dc..22babcd 100755 --- a/container/initrd-http/build +++ b/container/initrd-http/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "initrd-http" + docker build -t initrd-http . diff --git a/container/kibana/build b/container/kibana/build index 52d5a2a..9a1fb6d 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -3,6 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "kibana:7.6.2" + BASE_IMAGE_DICT = { 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', 'x86_64' => 'kibana:7.6.2' }.freeze diff --git a/container/lifecycle/build b/container/lifecycle/build index bfb7490..00a9c84 100755 --- a/container/lifecycle/build +++ b/container/lifecycle/build @@ -11,6 +11,8 @@ defaults = relevant_defaults(names) LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11312' +docker_skip_rebuild "lifecycle-#{LIFECYCLE_PORT}" + CCI_SRC = ENV['CCI_SRC'] system "bash #{CCI_SRC}/container/compile ." system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." diff --git a/container/lkp-initrd/build b/container/lkp-initrd/build index 2383323..cfba617 100755 --- a/container/lkp-initrd/build +++ b/container/lkp-initrd/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:lkp" + docker build -t alpine:lkp . diff --git a/container/logging-es/build b/container/logging-es/build index 6465ace..3be841a 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -5,6 +5,8 @@ require_relative '../defconfig.rb' +docker_skip_rebuild "logging-es:7.6.2" + BASE_IMAGE_DICT = { 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', 'x86_64' => 'elasticsearch:7.6.2' diff --git a/container/mail-robot/build b/container/mail-robot/build index c564add..4483e89 100755 --- a/container/mail-robot/build +++ b/container/mail-robot/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "mail-robot:latest" + docker build -t mail-robot:latest . diff --git a/container/manjaro/build b/container/manjaro/build index 114e550..6aeb3c6 100755 --- a/container/manjaro/build +++ b/container/manjaro/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "manjaro:testbed" + declare -A BASE_IMAGE_DICT BASE_IMAGE_DICT=( diff --git a/container/master-fluentd/build b/container/master-fluentd/build index 7d0291d..987cfcd 100755 --- a/container/master-fluentd/build +++ b/container/master-fluentd/build @@ -3,4 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "master-fluentd:alpine" + system 'docker build -t master-fluentd:alpine .' diff --git a/container/minio/build b/container/minio/build index d1ccd06..b0909e9 100755 --- a/container/minio/build +++ b/container/minio/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "jessestuart/minio" + docker pull jessestuart/minio diff --git a/container/monitoring/build b/container/monitoring/build index bcb0c37..121ede0 100755 --- a/container/monitoring/build +++ b/container/monitoring/build @@ -11,6 +11,8 @@ defaults = relevant_defaults(names) MONITOR_PORT = (defaults['MONITOR_PORT'] || '11310') +docker_skip_rebuild "monitoring-#{MONITOR_PORT}" + CCI_SRC = ENV['CCI_SRC'] system "bash #{CCI_SRC}/container/compile ." system "docker build -t monitoring-#{MONITOR_PORT} ." diff --git a/container/netdata-slave/build b/container/netdata-slave/build index 1a84550..cb53364 100755 --- a/container/netdata-slave/build +++ b/container/netdata-slave/build @@ -2,4 +2,8 @@ # Origin: https://hub.docker.com/r/netdata/netdata # SPDX-License-Identifier: GPL-3.0+ +. ../defconfig.sh + +docker_skip_rebuild "netdata/netdata-slave" + docker build -t netdata/netdata-slave . diff --git a/container/netdata/build b/container/netdata/build index 809b445..6aa9a17 100755 --- a/container/netdata/build +++ b/container/netdata/build @@ -2,4 +2,8 @@ # Origin: https://hub.docker.com/r/netdata/netdata # SPDX-License-Identifier: GPL-3.0+ +. ../defconfig.sh + +docker_skip_rebuild "netdata/netdatalkp" + docker build -t netdata/netdatalkp . diff --git a/container/ntp-server/build b/container/ntp-server/build index 7826fb2..d52f9dd 100755 --- a/container/ntp-server/build +++ b/container/ntp-server/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "ntp-server" + docker build -t ntp-server . diff --git a/container/openresty-proxy-cache/build b/container/openresty-proxy-cache/build index 507c4fe..7665209 100755 --- a/container/openresty-proxy-cache/build +++ b/container/openresty-proxy-cache/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "openresty:proxy_cache" + [ -d "openresty" ] && rm -rf openresty git clone https://gitee.com/cuiyili/openresty.git || exit 1 diff --git a/container/os-cifs/build b/container/os-cifs/build index 77be8e9..de44c44 100755 --- a/container/os-cifs/build +++ b/container/os-cifs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "os-cifs" + docker build -t os-cifs . diff --git a/container/os-http/build b/container/os-http/build index c1cb008..367748e 100755 --- a/container/os-http/build +++ b/container/os-http/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "os-http" + docker build -t os-http . diff --git a/container/os-nfs/build b/container/os-nfs/build index eea5a7d..e0fd476 100755 --- a/container/os-nfs/build +++ b/container/os-nfs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "os-nfs" + docker build -t os-nfs . diff --git a/container/qcow2rootfs/build b/container/qcow2rootfs/build index 565de24..7f31bb4 100755 --- a/container/qcow2rootfs/build +++ b/container/qcow2rootfs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "qcow2rootfs:auto" + docker build -t qcow2rootfs:auto . diff --git a/container/qemu-efi/build b/container/qemu-efi/build index cac9426..4f1d488 100755 --- a/container/qemu-efi/build +++ b/container/qemu-efi/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:qemu-efi" + [ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 [ $(arch) == "x86_64" ] && uefi=ovmf diff --git a/container/rabbitmq/build b/container/rabbitmq/build index c0bc179..930c112 100755 --- a/container/rabbitmq/build +++ b/container/rabbitmq/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "rabbitmq:3-management" + docker pull rabbitmq:3-management diff --git a/container/redis/build b/container/redis/build index e5230b0..490f99d 100755 --- a/container/redis/build +++ b/container/redis/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "redis507n:alpine311" + docker build -t redis507n:alpine311 . diff --git a/container/register-accounts/build b/container/register-accounts/build index e1dcfa4..2364b34 100755 --- a/container/register-accounts/build +++ b/container/register-accounts/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "register-accounts" + docker build -t register-accounts . diff --git a/container/registry/build b/container/registry/build index ca8680a..2edc2c1 100755 --- a/container/registry/build +++ b/container/registry/build @@ -2,4 +2,8 @@ # Origin: https://docs.docker.com/registry # SPDX-License-Identifier: Apache license +. ../defconfig.sh + +docker_skip_rebuild "registry" + docker pull registry diff --git a/container/remote-git/build b/container/remote-git/build index dad1a9b..727c7f9 100755 --- a/container/remote-git/build +++ b/container/remote-git/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:remote-git" + docker build -t debian:remote-git . diff --git a/container/result-cifs/build b/container/result-cifs/build index d97b6dd..fd9d2df 100755 --- a/container/result-cifs/build +++ b/container/result-cifs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "result-cifs" + docker build -t result-cifs . diff --git a/container/result-webdav/build b/container/result-webdav/build index 7d70ae6..b5be9e9 100755 --- a/container/result-webdav/build +++ b/container/result-webdav/build @@ -4,4 +4,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "result-webdav" + docker build -t result-webdav . diff --git a/container/rsync-server/build b/container/rsync-server/build index 72bfc47..5d36265 100755 --- a/container/rsync-server/build +++ b/container/rsync-server/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "rsync_server" + docker build -t rsync_server . diff --git a/container/scheduler-dev/build b/container/scheduler-dev/build index 1e4c57e..56d27c0 100755 --- a/container/scheduler-dev/build +++ b/container/scheduler-dev/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:scheduler-dev" + docker build -t alpine:scheduler-dev . diff --git a/container/scheduler/build b/container/scheduler/build index 367aa4e..412988a 100755 --- a/container/scheduler/build +++ b/container/scheduler/build @@ -11,6 +11,8 @@ defaults = relevant_defaults(names) SCHED_PORT = (defaults['SCHED_PORT'] || '3000') +docker_skip_rebuild "sch-ruby-a:v0.00d-#{SCHED_PORT}" + CCI_SRC = ENV['CCI_SRC'] system "bash #{CCI_SRC}/container/compile ." system "cp #{CCI_SRC}/sbin/create-job-cpio.sh ." diff --git a/container/send-internet-mail/build b/container/send-internet-mail/build index 078adaf..07ca143 100755 --- a/container/send-internet-mail/build +++ b/container/send-internet-mail/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "send-internet-mail:latest" + docker build -t send-internet-mail:latest . diff --git a/container/send-mail/build b/container/send-mail/build index 9159641..8ae9788 100755 --- a/container/send-mail/build +++ b/container/send-mail/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "send-mail:latest" + docker build -t send-mail:latest . diff --git a/container/shellcheck/build b/container/shellcheck/build index 854bae6..f295e51 100755 --- a/container/shellcheck/build +++ b/container/shellcheck/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "shellcheck" + docker build -t shellcheck . diff --git a/container/srv-http/build b/container/srv-http/build index 054e826..c8c9aa4 100755 --- a/container/srv-http/build +++ b/container/srv-http/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "srv-http" + : ${H5AI_SRC:=/c/h5ai} cp -r $H5AI_SRC ./ diff --git a/container/ssh-r/build b/container/ssh-r/build index 4bec543..2ad1bd3 100755 --- a/container/ssh-r/build +++ b/container/ssh-r/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "ssh-r:0.001" + if [ ! -f "known_auth_keys" ]; then echo "-----> ERROR: no testbox rsa public key file 'known_auth_keys' at currrent DIR" exit 1 diff --git a/container/sub-fluentd/build b/container/sub-fluentd/build index 797cd0d..1ec2246 100755 --- a/container/sub-fluentd/build +++ b/container/sub-fluentd/build @@ -3,4 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "sub-fluentd:alpine" + system 'docker build -t sub-fluentd:alpine .' diff --git a/container/submit/build b/container/submit/build index fba4281..dff884a 100755 --- a/container/submit/build +++ b/container/submit/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "submit:latest" + docker build -t submit:latest . diff --git a/container/taskqueue/build b/container/taskqueue/build index 3a5d4c9..4f69f85 100755 --- a/container/taskqueue/build +++ b/container/taskqueue/build @@ -10,6 +10,8 @@ load_cci_defaults service_port=${TASKQUEUE_PORT:-3060} +docker_skip_rebuild "taskqueue-alpine:v0.001-${service_port}" + if [[ ! -f "taskqueue" ]]; then $CCI_SRC/container/compile . else diff --git a/container/upload-libvirt-xml/build b/container/upload-libvirt-xml/build index d39838b..0a22173 100755 --- a/container/upload-libvirt-xml/build +++ b/container/upload-libvirt-xml/build @@ -4,6 +4,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "upload-libvirt-xml" + # The libvirt xml upload has nothing to do with the job. It renames the uploaded file and does not currently have permission to upload. # it is a transitional service. Two different functions use the same port service, diff --git a/container/web-backend-nginx/build b/container/web-backend-nginx/build index 4c9d3fe..d282570 100755 --- a/container/web-backend-nginx/build +++ b/container/web-backend-nginx/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "web-backend-nginx" + docker build -t web-backend-nginx . diff --git a/container/web-backend/build b/container/web-backend/build index 671fc97..c7f7d8a 100755 --- a/container/web-backend/build +++ b/container/web-backend/build @@ -1,4 +1,8 @@ #!/bin/sh # SPDX-License-Identifier: MulanPSL-2.0+ +. ../defconfig.sh + +docker_skip_rebuild "debian:web-backend" + docker build -t debian:web-backend . diff --git a/container/webhook/build b/container/webhook/build index bb5c0a7..a3475c4 100755 --- a/container/webhook/build +++ b/container/webhook/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "node:webhook" + docker build -t node:webhook . -- 2.23.0

1 0

[PATCH compass-ci] providers: add my-libvirt script for user
by Xiao Shenwei 05 Feb '21

05 Feb '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/my-libvirt | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100755 providers/my-libvirt diff --git a/providers/my-libvirt b/providers/my-libvirt new file mode 100755 index 0000000..f1ba374 --- /dev/null +++ b/providers/my-libvirt @@ -0,0 +1,13 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './libvirt/run' + +tbox_group = 'vt-2p16g' +hostname = "#{tbox_group}.#{ENV['USER']}-#{Process.pid}" +# specify which queues will be request, use "," to separate more than 2 values +queues = "#{tbox_group}~#{ENV['USER']}" + +main(hostname, queues) -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: replace gem source
by Liu Yinsi 05 Feb '21

05 Feb '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..fca591c 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,6 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] doc: add job/rootfs_disk.md
by Yu Chuan 05 Feb '21

05 Feb '21

Add description of rootfs_disk field in job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/rootfs_disk.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 doc/job/rootfs_disk.md diff --git a/doc/job/rootfs_disk.md b/doc/job/rootfs_disk.md new file mode 100644 index 000000000000..72762cf72f2f --- /dev/null +++ b/doc/job/rootfs_disk.md @@ -0,0 +1,27 @@ +# Summary +========= + +`rootfs_disk` defines a disk which storing multiple rootfs, used for jobs which need boot from disk. + +Each of these rootfs corresponds to an logical volume, such as follow: + /dev/mapper/ + . + ├── os-openeuler_aarch64_20.03 + ├── os-openeuler_aarch64_20.03_{timestamp} + ├── os-openeuler_aarch64_20.03_sp1 + ├── os-openeuler_aarch64_20.03_sp1_{timestamp} + └── ... + +Addition: +1. `/dev/mapper/os-{os}_{os_arch}_{os_version}_{timestamp}` + - is a readonly logical volume + - used for backup and rollback + +2. `/dev/mapper/os-{os}_{os_arch}_{os_version}` + - is a read write logical volume + - use for boot and run job + - defaultly, this logical volume will be wiped before every job run + +3. every logical volume size: 10G + +4. more info can refer os_mount.md -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] doc: add job/os_mount.md
by Yu Chuan 05 Feb '21

05 Feb '21

Add description of os_mount field in job.yaml. This patch is the description when os_mount=local, and other values of os_mount will be added in the patch later. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 89 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 doc/job/os_mount.md diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md new file mode 100644 index 000000000000..257bc12619fe --- /dev/null +++ b/doc/job/os_mount.md @@ -0,0 +1,89 @@ +# Summary +========= + +`os_mount` defines the mount method of testbox's root partition. + +It has the following optional values: + - nfs + - cifs + - initramfs + - container + - local + +Usage example: + - submit iperf.yaml testbox=vm-2p8g os_mount=nfs + - submit iperf.yaml testbox=vm-2p8g os_mount=cifs + - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs + - submit iperf.yaml testbox=dc-8g os_mount=container + - submit iperf.yaml testbox=vm-2p8g os_mount=local + +# Optional Values +================= + +## Work flow when os_mount=local + +1. user submit job with os_mount: local + - optional kernel cmdline params: # can be added in {kernel_append_root} field of job.yaml + - save_root: give an iconic string, then the root partition data used by this job will be retained. + and you can use the root partition data of this job in the futher job by {use_root}. + - use_root : give an iconic string, then the root partition data generated by + the pervious job which has the {save_root} field will be used. + - tips: + - value of {save_root} and {use_root} need to be iconic enough, uuid is recommended. + - value of {save_root} and {use_root} will be the suffix of logical volume. + +2. scheduler return the ipxe_str to testbox + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz + imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + boot + ``` + +3. dracut step of boot + ``` + if have {use_root}; then + base_lv_name=openeuler_aarch64_20.03_{value_use_root} + lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 + + if have {save_root}; then + boot_lv_name=openeuler_aarch64_20.03_{value_save_root} + + if {save_root} == {use_root}; then + boot from /dev/mappaer/os-{boot_lv_name} + fi + + lvremove -f /dev/mappaer/os-{boot_lv_name} + create /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} + boot from /dev/mappaer/os-{boot_lv_name} + else + boot_lv_name=openeuler_aarch64_20.03_{value_use_root} + boot from /dev/mappaer/os-{boot_lv_name} + fi + + else: + base_lv_name=openeuler_aarch64_20.03_{timestamp} + lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { + # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} + mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt + mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly + lvchange -p r /dev/mapper/os-{base_lv_name} + } + + boot_lv_name=openeuler_aarch64_20.03 + if have {save_root}; then + boot_lv_name=openeuler_aarch64_20.03_{value_save_root} + fi + + lvremove -f /dev/mappaer/os-{boot_lv_name} + create /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} + boot from /dev/mappaer/os-{boot_lv_name} + fi + ``` + +4. execute the job -- 2.23.0

1 0

[PATCH v6 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 05 Feb '21

05 Feb '21

add limit for borrowing machine: runtime <= 10 days only jobs that runtime value is less than or equal to 10 days are allowed for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 89c92c1..61f18d6 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -401,6 +402,20 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + + # case runtime/sleep value count is more than 10 days, job will fail. + raise error_msg if @hash["runtime"].to_i > max_run_time + raise error_msg if @hash["sleep"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

2 1

[PATCH v2 compass-ci] sbin/compare remove change of failure metric
by Lu Weitao 05 Feb '21

05 Feb '21

[Why] We only display the failure metrics that belong the latest job, aimed to express the latest job exists error. so the change rate of failure metric is useless like: centos openeuler metric -------------------- ------------------------------ ------------------------------ fails:runs change fails:runs | | | 1:12 -8.3% 0:24 stderr./lkp/lkp/src/lib/env.sh:line#:/proc/sys/kernel/nmi_watchdog:Permission_denied [How] centos openeuler metric -------------------- ------------------------------ ------------------------------ fails:runs fails:runs | | 1:12 0:24 stderr./lkp/lkp/src/lib/env.sh:line#:/proc/sys/kernel/nmi_watchdog:Permission_denied centos openeuler metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2.638681e+10 ± 23% -0.1% 2.636343e+10 ± 22% iperf.tcp.receiver.bps 2.638853e+10 ± 23% +0.1% 2.641691e+10 ± 22% iperf.tcp.sender.bps 6.80 ± 141% -50.4% 3.37 ± 223% iperf.time.elapsed_time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 58 ++++++++--------------------------------- 1 file changed, 11 insertions(+), 47 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 6c88384..ffc6500 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -83,15 +83,12 @@ def get_values(value_list, success) end def get_compare_value(base_value_average, value_average, success) - # get compare value(change or reproduction) + # get compare value(change) # - if success - return if base_value_average.zero? + return unless success + return if base_value_average.zero? - (100 * value_average / base_value_average - 100).round(1) - else - (100 * (value_average - base_value_average)).round(1) - end + (100 * value_average / base_value_average - 100).round(1) end def field_changed?(base_values, values, success, field, options) @@ -265,7 +262,7 @@ def get_html_header(matrixes_titles, success) compare_title = CHANGE_STR else common_title = FAILS_RUNS_STR - compare_title = REPRODUCTION_STR + compare_title = '' end header = get_html_index(matrixes_titles) @@ -551,35 +548,6 @@ def format_fails_runs(fails, runs) fails_str + ':' + runs_str end -def get_reproduction_index_str(reproduction, compare_index) - reproduction_str = format('%+.1f%%', reproduction) - reproduction_index = compare_index - reproduction_str.index('.') - if reproduction_index.negative? - reproduction_str = format('%+.1e%%', reproduction).sub('e+0', 'e+').sub('e-0', 'e-') - reproduction_index = compare_index - reproduction_str.index('e') - 1 - end - return reproduction_index, reproduction_str -end - -def get_reprodcution(reproduction, compare_index) - if reproduction - reproduction_index, reproduction_str = get_reproduction_index_str(reproduction, compare_index) - space_str = ' ' * (SUB_SHORT_COLUMN_WIDTH - reproduction_str.length) - reproduction_str = space_str.insert(reproduction_index, reproduction_str) - else - reproduction_str = format("%-#{SUB_SHORT_COLUMN_WIDTH}s", '0') - end - reproduction_str -end - -def format_reproduction(reproduction, theme, compare_index) - color = get_compare_value_color(reproduction, theme) - colorize( - color, - get_reprodcution(reproduction, compare_index) - ) -end - def get_change_index_str(change, compare_index) change_str = format('%+.1f%%', change) change_index = compare_index - change_str.index('.') @@ -696,7 +664,7 @@ def get_title_name(success) compare_title = CHANGE_STR else common_title = FAILS_RUNS_STR - compare_title = REPRODUCTION_STR + compare_title = '' end return common_title, compare_title end @@ -721,14 +689,14 @@ def get_base_matrix_title_symbol(common_index, success) title_symbol end -def get_other_matrixes_title_symbol(_compare_title, matrixes_number, common_index, success) +def get_other_matrixes_title_symbol(compare_title, matrixes_number, common_index, success) title_symbol = ' ' * ( (INTERVAL_WIDTH + COLUMN_WIDTH) * matrixes_number ) start_point = 0 common_symbol = success ? '\\' : '|' - compare_symbol = '|' + compare_symbol = compare_title.empty? ? '' : '|' compare_index = SUB_SHORT_COLUMN_WIDTH / 2 (matrixes_number - 1).times do |_| @@ -789,12 +757,8 @@ def get_success_str(values, index, theme, compare_index) (change_str || '') + stddev_str end -def get_failure_str(values, index, theme, compare_index) - unless index.zero? - reproduction_str = format_reproduction( - values[:reproduction], theme, compare_index - ) - end +def get_failure_str(values, index) + reproduction_str = format("%-#{SUB_SHORT_COLUMN_WIDTH}s", '') unless index.zero? fails_runs_str = format_fails_runs( values[:fails], @@ -813,7 +777,7 @@ def get_values_str(matrixes, success, theme) ) + INTERVAL_BLANK else get_failure_str( - values, index, theme, compare_index + values, index ) + INTERVAL_BLANK end end -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/libvirt: consume job by libvirt
by Xiao Shenwei 05 Feb '21

05 Feb '21

function: 1. check host is the capability of executing tasks? - check libvirt - check qemu-kvm 2. request boot.libvirt and load files based on response 3. generate a domain.xml and start a vm used ruby-libvirt Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 68 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 64 insertions(+), 4 deletions(-) diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb index a500c31..2883197 100755 --- a/providers/libvirt/run.rb +++ b/providers/libvirt/run.rb @@ -9,6 +9,8 @@ require_relative "#{ENV['CCI_SRC']}/lib/log" require_relative "#{ENV['CCI_SRC']}/lib/sched_client" require_relative "#{ENV['CCI_SRC']}/providers/lib/context" require_relative "#{ENV['CCI_SRC']}/providers/lib/resource" +require_relative "#{ENV['CCI_SRC']}/providers/lib/domain" +require_relative "#{ENV['CCI_SRC']}/providers/lib/libvirt" def create_logger(hostname) filename = "#{hostname}.log" @@ -48,17 +50,75 @@ def request_job(context, sched_client, logger) return response end +def parse_response(response, context, resource) + context.merge!(response) + resource.parse_response(response) + context.merge!(resource.info) +end + +def generate_domain(context, logger) + domain = Domain.new(context, logger) + xml_path = '' + begin + domain.generate + xml_path = domain.save_to('domain.xml') + rescue StandardError => e + logger.error(e) + logger.error('please check your xml file format') + ensure + upload_log(context) + end + return xml_path +end + +def upload_log(context, upload_qemu_log: false) + id = context.info['job_id'] + host = context.info['LKP_SERVER'] + port = context.info['RESULT_WEBDAV_PORT'] + result_root = context.info['result_root'] + result_url = "http://#{host}:#{port}#{result_root}" + log_name = "#{context.info['hostname']}.log" + system "curl -sSf -T #{log_name} #{result_url}/ --cookie 'JOBID= #{id}'" + return unless upload_qemu_log + + qemu_log_name = "/var/log/libvirt/qemu/#{id}.log" + return unless FileTest.exist?(qemu_log_name) + + system "curl -sSf -T #{qemu_log_name} #{result_url}/ --cookie 'JOBID= #{id}'" +end + +def start(context, logger, libvirt) + xml_path = generate_domain(context, logger) + return if xml_path.empty? + + begin + libvirt.define(xml_path) + libvirt.create + libvirt.wait + rescue StandardError => e + logger.error(e) + ensure + upload_log(context, upload_qemu_log: true) + end +end + +def clean(context, sched_client, libvirt) + sched_client.delete_mac2host(context.info['mac']) + sched_client.delete_host2queues(context.info['hostname']) + libvirt.close +end + def main(hostname, queues) logger = create_logger(hostname) mac = compute_mac hostname context = Context.new(mac, hostname, queues) + libvirt = LibvirtConnect.new resource = Resource.new(hostname, logger) sched_client = SchedClient.new response = request_job(context, sched_client, logger) return if response.nil? - resource.parse_response(response) - context.merge!(resource.info) - sched_client.delete_mac2host(mac) - sched_client.delete_host2queues(hostname) + parse_response(response, context, resource) + start(context, logger, libvirt) + clean(context, sched_client, libvirt) end -- 2.23.0

1 0