- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests 2/2] installer/kali: add kali installer symlink
by Wang Yong 05 Feb '21

05 Feb '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/installer/kali | 1 + 1 file changed, 1 insertion(+) create mode 120000 distro/installer/kali diff --git a/distro/installer/kali b/distro/installer/kali new file mode 120000 index 000000000..b2f7fd3e9 --- /dev/null +++ b/distro/installer/kali @@ -0,0 +1 @@ +debian \ No newline at end of file -- 2.23.0

1 0

[PATCH lkp-tests 1/2] distro/kali: add kali distro file
by Wang Yong 05 Feb '21

05 Feb '21

add kali pack method Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/kali | 1 + 1 file changed, 1 insertion(+) create mode 120000 distro/kali diff --git a/distro/kali b/distro/kali new file mode 120000 index 000000000..b2f7fd3e9 --- /dev/null +++ b/distro/kali @@ -0,0 +1 @@ +debian \ No newline at end of file -- 2.23.0

1 0

[PATCH lkp-tests] stats/ansible_stats_openeuler: stats result for ansible migrating
by Wang Chenglong 05 Feb '21

05 Feb '21

[How] ansible_stats_openeuler ${job_output} ./ansible_stats_openeuler /srv/result/ansible_test/2021-02-04/vm-2p16g--wcl/openeuler-20.03-aarch64/linux-system-roles-storage/crystal.974233/output Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_stats_openeuler | 48 +++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100755 stats/ansible_stats_openeuler diff --git a/stats/ansible_stats_openeuler b/stats/ansible_stats_openeuler new file mode 100755 index 000000000..5a6870eef --- /dev/null +++ b/stats/ansible_stats_openeuler @@ -0,0 +1,48 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +job_error_msg = `grep -E 'FAILED! | failed: .*}$' #{ARGV[0]} | awk -F'=>' '{$1="";print $0}'` +error_msg_json = JSON.parse(job_error_msg) + +unless error_msg_json['msg'].nil? + puts "config.error.no-file-find-by.#{$1}: 1" if error_msg_json['msg'].match(/No file was found when using (.+)\./) + + puts "yaml.error.syntax-problem.[#{$1}]: 1" if error_msg_json['msg'].match(/The error was: (.+)/) + + if error_msg_json['msg'].match(/No such file or directory/) + puts "command.error.no-such-command.#{error_msg_json['cmd']}: 1" + end + + if error_msg_json['msg'].match(/Failed to download metadata for repo '(.+)': /) + puts "download.error.failed-to-download.#{$1}: 1" + end + + puts "script.error [#{error_msg_json['cmd']}]: 1" if error_msg_json['msg'].match(/non-zero return code/) + + puts "download.error.http-error-404 [#{$1}]: 1" if error_msg_json['msg'].match(/Failure downloading (.+), /) + + if error_msg_json['msg'].match(/Could not find the requested service firewall: (.+)/) + puts "firewall.error.could-not-find-server.#{$1}: 1" + end + + puts "service.error.unable-to-star-service.#{$1}: 1" if error_msg_json['msg'].match(/Unable to start service (.+):/) + + puts 'system.error.not-supporte: 1' if error_msg_json['msg'].match(/is not supporte/) + + puts "depsolve.error.depend-package-error.#{$1}: 1" if error_msg_json['msg'].match(/needed by (.+)/) + + puts 'read.error.read-timed-out: 1' if error_msg_json['msg'].match(/failed to create temporary content file/) +end + +error_msg_json['failures']&.each do |items| + items.match(/No package (.+) available/) + puts "yum.error.no-package-match.#{$1}: 1" +end + +unless error_msg_json['message'].nil? + puts "yaml.error.not-find-yml.#{$1}: 1" if a['message'].to_s.match(/Could not find or access (.+)/) +end -- 2.23.0

1 0

[PATCH compass-ci] providers/lib: fix wget in multi-process mode
by Xiao Shenwei 05 Feb '21

05 Feb '21

[why] Redirecting output to ‘wget-log’. Redirecting output to ‘wget-log.1’. Redirecting output to ‘wget-log.2’. Redirecting output to ‘wget-log.3’. [how] append info to log file Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/resource.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/lib/resource.rb b/providers/lib/resource.rb index bcce612..4279766 100644 --- a/providers/lib/resource.rb +++ b/providers/lib/resource.rb @@ -56,7 +56,7 @@ class Resource end def load_file(url) - system "wget --timestamping --progress=bar:force #{url}" + system "wget --timestamping -a #{(a)hostname}.log --progress=bar:force #{url}" basename = File.basename(url) file_size = %x(ls -s --block-size=M "#{basename}").chomp @logger.info("Load file size: #{file_size}") -- 2.23.0

1 0

[PATCH lab-z9] add physical machine information
by Bai Jing 05 Feb '21

05 Feb '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- hosts/taishan200-2280-2s64p-256g--a123 | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a123 diff --git a/hosts/taishan200-2280-2s64p-256g--a123 b/hosts/taishan200-2280-2s64p-256g--a123 new file mode 100644 index 0000000..f765092 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a123 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-3500003998881eb19 + - /dev/disk/by-id/scsi-3500003998881e9a1 +mac_addr: + - 44:a1:91:9d:f5:4a + - 44:a1:91:9d:f5:4b + - 44:a1:91:9d:f5:4c + - 44:a1:91:9d:f5:4d + - 44:a1:91:e5:a8:e1 + - 44:a1:91:e5:a8:e2 + - 44:a1:91:e5:a8:e3 + - 44:a1:91:e5:a8:e4 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.12.4 -- 2.23.0

1 0

[PATCH compass-ci 4/4] sparrow/4-docker: optimise function log_info() calling
by Liu Yinsi 05 Feb '21

05 Feb '21

before: just execute $container/build script, echo "build $container" after: only execute $container/build -> `docker build -t`, echo "build $container" [why] when execute `sparrow/4-docker/buildall run-only`, will execute $container/build script, but if image already exist, actually will not build container, it will exit code "1", so only exit code "0", call log_info "finish build $container" Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 83ecc9f..b24eca5 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -29,13 +29,9 @@ do_one_build() mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" - - log_info "start build $container." - [ -x build ] && ./build + [ "$?" == 0 ] && log_info "finish build $container" [ -x install ] && ./install - - log_info "finish build $container." ) lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

1 0

[PATCH compass-ci 3/4] sparrow/4-docker: export env DOCKER_SKIP_REBUILD
by Liu Yinsi 05 Feb '21

05 Feb '21

if env DOCKER_SKIP_REBUILD exists, then check image exists in $container/build script by call docker_skip_rebuild(), then not rebuild container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index d4e9419..83ecc9f 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -70,11 +70,9 @@ do container_name=$(basename $dir) [ "$container_name" == 'ssh-r' ] && continue - docker images | grep -wq "$container_name" - if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then - build_depends $dir & - wait - fi + [ "$action" == "run-only" ] && export DOCKER_SKIP_REBUILD=DOCKER_SKIP_REBUILD + build_depends $dir & + wait start_depends $dir done -- 2.23.0

1 0

[PATCH compass-ci 2/4] container: call docker_skip_rebuild()
by Liu Yinsi 05 Feb '21

05 Feb '21

prevent rebuild container when execute `sparrow/4-docker/buildall run-only` Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/add-software/build | 4 ++++ container/alpine/build | 4 ++++ container/archlinux/build | 4 ++++ container/assign-account/build | 4 ++++ container/assist-result/build | 4 ++++ container/assistant/build | 4 ++++ container/auto-submit/build | 4 ++++ container/conserver/build | 4 ++++ container/crystal-base/build | 4 ++++ container/crystal-compiler/build | 4 ++++ container/crystal-shards/build | 3 +++ container/debian/build | 4 ++++ container/delimiter/build | 2 ++ container/dnsmasq/build | 4 ++++ container/dracut-initrd/build | 4 ++++ container/es/build | 2 ++ container/extract-stats/build | 4 ++++ container/fetch-mail/build | 4 ++++ container/fluentd-base/build | 4 ++++ container/git-daemon/build | 4 ++++ container/git-mirror/build | 4 ++++ container/gitcache/build | 4 ++++ container/initrd-cifs/build | 4 ++++ container/initrd-http/build | 4 ++++ container/kibana/build | 4 ++++ container/lifecycle/build | 2 ++ container/lkp-initrd/build | 4 ++++ container/logging-es/build | 2 ++ container/mail-robot/build | 4 ++++ container/manjaro/build | 4 ++++ container/master-fluentd/build | 4 ++++ container/minio/build | 4 ++++ container/monitoring/build | 2 ++ container/netdata-slave/build | 4 ++++ container/netdata/build | 4 ++++ container/ntp-server/build | 4 ++++ container/openresty-proxy-cache/build | 4 ++++ container/os-cifs/build | 4 ++++ container/os-http/build | 4 ++++ container/os-nfs/build | 4 ++++ container/qcow2rootfs/build | 4 ++++ container/qemu-efi/build | 4 ++++ container/rabbitmq/build | 4 ++++ container/redis/build | 4 ++++ container/register-accounts/build | 4 ++++ container/registry/build | 4 ++++ container/remote-git/build | 4 ++++ container/result-cifs/build | 4 ++++ container/result-webdav/build | 4 ++++ container/rsync-server/build | 4 ++++ container/scheduler-dev/build | 4 ++++ container/scheduler/build | 2 ++ container/send-internet-mail/build | 4 ++++ container/send-mail/build | 4 ++++ container/shellcheck/build | 4 ++++ container/srv-http/build | 4 ++++ container/ssh-r/build | 4 ++++ container/sub-fluentd/build | 4 ++++ container/submit/build | 4 ++++ container/taskqueue/build | 2 ++ container/upload-libvirt-xml/build | 4 ++++ container/web-backend-nginx/build | 4 ++++ container/web-backend/build | 4 ++++ container/webhook/build | 4 ++++ 64 files changed, 241 insertions(+) diff --git a/container/add-software/build b/container/add-software/build index 234c8b9..bc3c95a 100755 --- a/container/add-software/build +++ b/container/add-software/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "addsoftware:auto" + docker build -t addsoftware:auto . diff --git a/container/alpine/build b/container/alpine/build index bd277c3..3168799 100755 --- a/container/alpine/build +++ b/container/alpine/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:testbed" + docker build -t alpine:testbed . diff --git a/container/archlinux/build b/container/archlinux/build index 9749489..6db5a68 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "archlinux:testbed" + docker build --build-arg ARCH=$(arch) -t archlinux:testbed . diff --git a/container/assign-account/build b/container/assign-account/build index 3984341..480c410 100755 --- a/container/assign-account/build +++ b/container/assign-account/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "assign-account:latest" + docker build -t assign-account:latest . diff --git a/container/assist-result/build b/container/assist-result/build index 130127b..600b80a 100755 --- a/container/assist-result/build +++ b/container/assist-result/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "assist-result" + docker build -t assist-result . diff --git a/container/assistant/build b/container/assistant/build index 7778c9f..89a8a97 100755 --- a/container/assistant/build +++ b/container/assistant/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:assistant" + docker build -t debian:assistant . diff --git a/container/auto-submit/build b/container/auto-submit/build index 96383e3..48ff111 100755 --- a/container/auto-submit/build +++ b/container/auto-submit/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:auto-submit" + docker build -t alpine:auto-submit . diff --git a/container/conserver/build b/container/conserver/build index 0700c5e..a7e3867 100755 --- a/container/conserver/build +++ b/container/conserver/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "conserver_server" + ./generate_conserver.rb || exit 1 docker build -t conserver_server . diff --git a/container/crystal-base/build b/container/crystal-base/build index 4df65a5..642e190 100755 --- a/container/crystal-base/build +++ b/container/crystal-base/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:crystal-base" + if [ -n "$SQUID_HOST" ] && [ -n "$SQUID_PORT" ]; then proxy="http://$SQUID_HOST:$SQUID_PORT/" else diff --git a/container/crystal-compiler/build b/container/crystal-compiler/build index cee43eb..570fe98 100755 --- a/container/crystal-compiler/build +++ b/container/crystal-compiler/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:crystal-compiler" + docker build -t alpine:crystal-compiler . diff --git a/container/crystal-shards/build b/container/crystal-shards/build index 784f571..7c2a712 100755 --- a/container/crystal-shards/build +++ b/container/crystal-shards/build @@ -6,6 +6,9 @@ # If job.yaml does not have these two parameters, git will use the local proxy cache, # GITCACHE_HOST defaults to the local ip, GITCACHE_PORT default port is 5000 . ../defconfig.sh + +docker_skip_rebuild "alpine:crystal-shards" + load_cci_defaults docker build --build-arg GITCACHE_HOST=$GITCACHE_HOST --build-arg GITCACHE_PORT=$GITCACHE_PORT -t alpine:crystal-shards . diff --git a/container/debian/build b/container/debian/build index 6bff579..eac9af2 100755 --- a/container/debian/build +++ b/container/debian/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:testbed" + docker build -t debian:testbed . diff --git a/container/delimiter/build b/container/delimiter/build index f7f53e7..606149d 100755 --- a/container/delimiter/build +++ b/container/delimiter/build @@ -4,6 +4,8 @@ . ../defconfig.sh +docker_skip_rebuild "delimiter" + load_cci_defaults docker build -t delimiter . diff --git a/container/dnsmasq/build b/container/dnsmasq/build index 8a22575..b5738bc 100755 --- a/container/dnsmasq/build +++ b/container/dnsmasq/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "dnsmasq:alpine" + [[ -e /tftpboot/boot.ipxe ]] || { cp tftpboot/boot.ipxe /tftpboot/boot.ipxe } diff --git a/container/dracut-initrd/build b/container/dracut-initrd/build index 181b896..f02cea9 100755 --- a/container/dracut-initrd/build +++ b/container/dracut-initrd/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:dracut" + docker build -t debian:dracut . diff --git a/container/es/build b/container/es/build index 17a225c..c7d7115 100755 --- a/container/es/build +++ b/container/es/build @@ -5,6 +5,8 @@ require_relative '../defconfig.rb' +docker_skip_rebuild "es643b:alpine311" + available_memory = get_available_memory system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." diff --git a/container/extract-stats/build b/container/extract-stats/build index 1aa9982..c315f0c 100755 --- a/container/extract-stats/build +++ b/container/extract-stats/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "extract-stats" + CCI_SRC=$CCI_SRC bash $CCI_SRC/container/compile . diff --git a/container/fetch-mail/build b/container/fetch-mail/build index e72bed8..9b1a501 100755 --- a/container/fetch-mail/build +++ b/container/fetch-mail/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "fetch-mail:latest" + docker build -t fetch-mail:latest . diff --git a/container/fluentd-base/build b/container/fluentd-base/build index 32e0a38..72b85bf 100755 --- a/container/fluentd-base/build +++ b/container/fluentd-base/build @@ -3,4 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "fluentd-base:alpine" + system 'docker build -t fluentd-base:alpine .' diff --git a/container/git-daemon/build b/container/git-daemon/build index d81e4b9..5ba9a52 100755 --- a/container/git-daemon/build +++ b/container/git-daemon/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "apline311:git-daemon" + docker build -t apline311:git-daemon . diff --git a/container/git-mirror/build b/container/git-mirror/build index 66967e4..c302ac7 100755 --- a/container/git-mirror/build +++ b/container/git-mirror/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine38:git-mirror" + docker build -t alpine38:git-mirror . diff --git a/container/gitcache/build b/container/gitcache/build index 5135f7c..d7f3523 100755 --- a/container/gitcache/build +++ b/container/gitcache/build @@ -3,4 +3,8 @@ # Copyright (C) 2019-2020 Eric D.little51 # SPDX-License-Identifier: GPL-3.0 +. ../defconfig.sh + +docker_skip_rebuild "gitcache" + docker build -t gitcache . diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build index 4b93647..19d5125 100755 --- a/container/initrd-cifs/build +++ b/container/initrd-cifs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "initrd-cifs" + docker build -t initrd-cifs . diff --git a/container/initrd-http/build b/container/initrd-http/build index 1b039dc..22babcd 100755 --- a/container/initrd-http/build +++ b/container/initrd-http/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "initrd-http" + docker build -t initrd-http . diff --git a/container/kibana/build b/container/kibana/build index 52d5a2a..9a1fb6d 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -3,6 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "kibana:7.6.2" + BASE_IMAGE_DICT = { 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', 'x86_64' => 'kibana:7.6.2' }.freeze diff --git a/container/lifecycle/build b/container/lifecycle/build index bfb7490..00a9c84 100755 --- a/container/lifecycle/build +++ b/container/lifecycle/build @@ -11,6 +11,8 @@ defaults = relevant_defaults(names) LIFECYCLE_PORT = defaults['LIFECYCLE_PORT'] || '11312' +docker_skip_rebuild "lifecycle-#{LIFECYCLE_PORT}" + CCI_SRC = ENV['CCI_SRC'] system "bash #{CCI_SRC}/container/compile ." system "docker build -t lifecycle-#{LIFECYCLE_PORT} ." diff --git a/container/lkp-initrd/build b/container/lkp-initrd/build index 2383323..cfba617 100755 --- a/container/lkp-initrd/build +++ b/container/lkp-initrd/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:lkp" + docker build -t alpine:lkp . diff --git a/container/logging-es/build b/container/logging-es/build index 6465ace..3be841a 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -5,6 +5,8 @@ require_relative '../defconfig.rb' +docker_skip_rebuild "logging-es:7.6.2" + BASE_IMAGE_DICT = { 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', 'x86_64' => 'elasticsearch:7.6.2' diff --git a/container/mail-robot/build b/container/mail-robot/build index c564add..4483e89 100755 --- a/container/mail-robot/build +++ b/container/mail-robot/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "mail-robot:latest" + docker build -t mail-robot:latest . diff --git a/container/manjaro/build b/container/manjaro/build index 114e550..6aeb3c6 100755 --- a/container/manjaro/build +++ b/container/manjaro/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "manjaro:testbed" + declare -A BASE_IMAGE_DICT BASE_IMAGE_DICT=( diff --git a/container/master-fluentd/build b/container/master-fluentd/build index 7d0291d..987cfcd 100755 --- a/container/master-fluentd/build +++ b/container/master-fluentd/build @@ -3,4 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "master-fluentd:alpine" + system 'docker build -t master-fluentd:alpine .' diff --git a/container/minio/build b/container/minio/build index d1ccd06..b0909e9 100755 --- a/container/minio/build +++ b/container/minio/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "jessestuart/minio" + docker pull jessestuart/minio diff --git a/container/monitoring/build b/container/monitoring/build index bcb0c37..121ede0 100755 --- a/container/monitoring/build +++ b/container/monitoring/build @@ -11,6 +11,8 @@ defaults = relevant_defaults(names) MONITOR_PORT = (defaults['MONITOR_PORT'] || '11310') +docker_skip_rebuild "monitoring-#{MONITOR_PORT}" + CCI_SRC = ENV['CCI_SRC'] system "bash #{CCI_SRC}/container/compile ." system "docker build -t monitoring-#{MONITOR_PORT} ." diff --git a/container/netdata-slave/build b/container/netdata-slave/build index 1a84550..cb53364 100755 --- a/container/netdata-slave/build +++ b/container/netdata-slave/build @@ -2,4 +2,8 @@ # Origin: https://hub.docker.com/r/netdata/netdata # SPDX-License-Identifier: GPL-3.0+ +. ../defconfig.sh + +docker_skip_rebuild "netdata/netdata-slave" + docker build -t netdata/netdata-slave . diff --git a/container/netdata/build b/container/netdata/build index 809b445..6aa9a17 100755 --- a/container/netdata/build +++ b/container/netdata/build @@ -2,4 +2,8 @@ # Origin: https://hub.docker.com/r/netdata/netdata # SPDX-License-Identifier: GPL-3.0+ +. ../defconfig.sh + +docker_skip_rebuild "netdata/netdatalkp" + docker build -t netdata/netdatalkp . diff --git a/container/ntp-server/build b/container/ntp-server/build index 7826fb2..d52f9dd 100755 --- a/container/ntp-server/build +++ b/container/ntp-server/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "ntp-server" + docker build -t ntp-server . diff --git a/container/openresty-proxy-cache/build b/container/openresty-proxy-cache/build index 507c4fe..7665209 100755 --- a/container/openresty-proxy-cache/build +++ b/container/openresty-proxy-cache/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "openresty:proxy_cache" + [ -d "openresty" ] && rm -rf openresty git clone https://gitee.com/cuiyili/openresty.git || exit 1 diff --git a/container/os-cifs/build b/container/os-cifs/build index 77be8e9..de44c44 100755 --- a/container/os-cifs/build +++ b/container/os-cifs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "os-cifs" + docker build -t os-cifs . diff --git a/container/os-http/build b/container/os-http/build index c1cb008..367748e 100755 --- a/container/os-http/build +++ b/container/os-http/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "os-http" + docker build -t os-http . diff --git a/container/os-nfs/build b/container/os-nfs/build index eea5a7d..e0fd476 100755 --- a/container/os-nfs/build +++ b/container/os-nfs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "os-nfs" + docker build -t os-nfs . diff --git a/container/qcow2rootfs/build b/container/qcow2rootfs/build index 565de24..7f31bb4 100755 --- a/container/qcow2rootfs/build +++ b/container/qcow2rootfs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "qcow2rootfs:auto" + docker build -t qcow2rootfs:auto . diff --git a/container/qemu-efi/build b/container/qemu-efi/build index cac9426..4f1d488 100755 --- a/container/qemu-efi/build +++ b/container/qemu-efi/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:qemu-efi" + [ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 [ $(arch) == "x86_64" ] && uefi=ovmf diff --git a/container/rabbitmq/build b/container/rabbitmq/build index c0bc179..930c112 100755 --- a/container/rabbitmq/build +++ b/container/rabbitmq/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "rabbitmq:3-management" + docker pull rabbitmq:3-management diff --git a/container/redis/build b/container/redis/build index e5230b0..490f99d 100755 --- a/container/redis/build +++ b/container/redis/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "redis507n:alpine311" + docker build -t redis507n:alpine311 . diff --git a/container/register-accounts/build b/container/register-accounts/build index e1dcfa4..2364b34 100755 --- a/container/register-accounts/build +++ b/container/register-accounts/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "register-accounts" + docker build -t register-accounts . diff --git a/container/registry/build b/container/registry/build index ca8680a..2edc2c1 100755 --- a/container/registry/build +++ b/container/registry/build @@ -2,4 +2,8 @@ # Origin: https://docs.docker.com/registry # SPDX-License-Identifier: Apache license +. ../defconfig.sh + +docker_skip_rebuild "registry" + docker pull registry diff --git a/container/remote-git/build b/container/remote-git/build index dad1a9b..727c7f9 100755 --- a/container/remote-git/build +++ b/container/remote-git/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "debian:remote-git" + docker build -t debian:remote-git . diff --git a/container/result-cifs/build b/container/result-cifs/build index d97b6dd..fd9d2df 100755 --- a/container/result-cifs/build +++ b/container/result-cifs/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "result-cifs" + docker build -t result-cifs . diff --git a/container/result-webdav/build b/container/result-webdav/build index 7d70ae6..b5be9e9 100755 --- a/container/result-webdav/build +++ b/container/result-webdav/build @@ -4,4 +4,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "result-webdav" + docker build -t result-webdav . diff --git a/container/rsync-server/build b/container/rsync-server/build index 72bfc47..5d36265 100755 --- a/container/rsync-server/build +++ b/container/rsync-server/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "rsync_server" + docker build -t rsync_server . diff --git a/container/scheduler-dev/build b/container/scheduler-dev/build index 1e4c57e..56d27c0 100755 --- a/container/scheduler-dev/build +++ b/container/scheduler-dev/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "alpine:scheduler-dev" + docker build -t alpine:scheduler-dev . diff --git a/container/scheduler/build b/container/scheduler/build index 367aa4e..412988a 100755 --- a/container/scheduler/build +++ b/container/scheduler/build @@ -11,6 +11,8 @@ defaults = relevant_defaults(names) SCHED_PORT = (defaults['SCHED_PORT'] || '3000') +docker_skip_rebuild "sch-ruby-a:v0.00d-#{SCHED_PORT}" + CCI_SRC = ENV['CCI_SRC'] system "bash #{CCI_SRC}/container/compile ." system "cp #{CCI_SRC}/sbin/create-job-cpio.sh ." diff --git a/container/send-internet-mail/build b/container/send-internet-mail/build index 078adaf..07ca143 100755 --- a/container/send-internet-mail/build +++ b/container/send-internet-mail/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "send-internet-mail:latest" + docker build -t send-internet-mail:latest . diff --git a/container/send-mail/build b/container/send-mail/build index 9159641..8ae9788 100755 --- a/container/send-mail/build +++ b/container/send-mail/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "send-mail:latest" + docker build -t send-mail:latest . diff --git a/container/shellcheck/build b/container/shellcheck/build index 854bae6..f295e51 100755 --- a/container/shellcheck/build +++ b/container/shellcheck/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "shellcheck" + docker build -t shellcheck . diff --git a/container/srv-http/build b/container/srv-http/build index 054e826..c8c9aa4 100755 --- a/container/srv-http/build +++ b/container/srv-http/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "srv-http" + : ${H5AI_SRC:=/c/h5ai} cp -r $H5AI_SRC ./ diff --git a/container/ssh-r/build b/container/ssh-r/build index 4bec543..2ad1bd3 100755 --- a/container/ssh-r/build +++ b/container/ssh-r/build @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "ssh-r:0.001" + if [ ! -f "known_auth_keys" ]; then echo "-----> ERROR: no testbox rsa public key file 'known_auth_keys' at currrent DIR" exit 1 diff --git a/container/sub-fluentd/build b/container/sub-fluentd/build index 797cd0d..1ec2246 100755 --- a/container/sub-fluentd/build +++ b/container/sub-fluentd/build @@ -3,4 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../defconfig' + +docker_skip_rebuild "sub-fluentd:alpine" + system 'docker build -t sub-fluentd:alpine .' diff --git a/container/submit/build b/container/submit/build index fba4281..dff884a 100755 --- a/container/submit/build +++ b/container/submit/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "submit:latest" + docker build -t submit:latest . diff --git a/container/taskqueue/build b/container/taskqueue/build index 3a5d4c9..4f69f85 100755 --- a/container/taskqueue/build +++ b/container/taskqueue/build @@ -10,6 +10,8 @@ load_cci_defaults service_port=${TASKQUEUE_PORT:-3060} +docker_skip_rebuild "taskqueue-alpine:v0.001-${service_port}" + if [[ ! -f "taskqueue" ]]; then $CCI_SRC/container/compile . else diff --git a/container/upload-libvirt-xml/build b/container/upload-libvirt-xml/build index d39838b..0a22173 100755 --- a/container/upload-libvirt-xml/build +++ b/container/upload-libvirt-xml/build @@ -4,6 +4,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "upload-libvirt-xml" + # The libvirt xml upload has nothing to do with the job. It renames the uploaded file and does not currently have permission to upload. # it is a transitional service. Two different functions use the same port service, diff --git a/container/web-backend-nginx/build b/container/web-backend-nginx/build index 4c9d3fe..d282570 100755 --- a/container/web-backend-nginx/build +++ b/container/web-backend-nginx/build @@ -2,4 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "web-backend-nginx" + docker build -t web-backend-nginx . diff --git a/container/web-backend/build b/container/web-backend/build index 671fc97..c7f7d8a 100755 --- a/container/web-backend/build +++ b/container/web-backend/build @@ -1,4 +1,8 @@ #!/bin/sh # SPDX-License-Identifier: MulanPSL-2.0+ +. ../defconfig.sh + +docker_skip_rebuild "debian:web-backend" + docker build -t debian:web-backend . diff --git a/container/webhook/build b/container/webhook/build index bb5c0a7..a3475c4 100755 --- a/container/webhook/build +++ b/container/webhook/build @@ -2,5 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +docker_skip_rebuild "node:webhook" + docker build -t node:webhook . -- 2.23.0

1 0

[PATCH compass-ci] providers: add my-libvirt script for user
by Xiao Shenwei 05 Feb '21

05 Feb '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/my-libvirt | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100755 providers/my-libvirt diff --git a/providers/my-libvirt b/providers/my-libvirt new file mode 100755 index 0000000..f1ba374 --- /dev/null +++ b/providers/my-libvirt @@ -0,0 +1,13 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './libvirt/run' + +tbox_group = 'vt-2p16g' +hostname = "#{tbox_group}.#{ENV['USER']}-#{Process.pid}" +# specify which queues will be request, use "," to separate more than 2 values +queues = "#{tbox_group}~#{ENV['USER']}" + +main(hostname, queues) -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: replace gem source
by Liu Yinsi 05 Feb '21

05 Feb '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..fca591c 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,6 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] doc: add job/rootfs_disk.md
by Yu Chuan 05 Feb '21

05 Feb '21

Add description of rootfs_disk field in job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/rootfs_disk.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 doc/job/rootfs_disk.md diff --git a/doc/job/rootfs_disk.md b/doc/job/rootfs_disk.md new file mode 100644 index 000000000000..72762cf72f2f --- /dev/null +++ b/doc/job/rootfs_disk.md @@ -0,0 +1,27 @@ +# Summary +========= + +`rootfs_disk` defines a disk which storing multiple rootfs, used for jobs which need boot from disk. + +Each of these rootfs corresponds to an logical volume, such as follow: + /dev/mapper/ + . + ├── os-openeuler_aarch64_20.03 + ├── os-openeuler_aarch64_20.03_{timestamp} + ├── os-openeuler_aarch64_20.03_sp1 + ├── os-openeuler_aarch64_20.03_sp1_{timestamp} + └── ... + +Addition: +1. `/dev/mapper/os-{os}_{os_arch}_{os_version}_{timestamp}` + - is a readonly logical volume + - used for backup and rollback + +2. `/dev/mapper/os-{os}_{os_arch}_{os_version}` + - is a read write logical volume + - use for boot and run job + - defaultly, this logical volume will be wiped before every job run + +3. every logical volume size: 10G + +4. more info can refer os_mount.md -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] doc: add job/os_mount.md
by Yu Chuan 05 Feb '21

05 Feb '21

Add description of os_mount field in job.yaml. This patch is the description when os_mount=local, and other values of os_mount will be added in the patch later. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 89 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 doc/job/os_mount.md diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md new file mode 100644 index 000000000000..257bc12619fe --- /dev/null +++ b/doc/job/os_mount.md @@ -0,0 +1,89 @@ +# Summary +========= + +`os_mount` defines the mount method of testbox's root partition. + +It has the following optional values: + - nfs + - cifs + - initramfs + - container + - local + +Usage example: + - submit iperf.yaml testbox=vm-2p8g os_mount=nfs + - submit iperf.yaml testbox=vm-2p8g os_mount=cifs + - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs + - submit iperf.yaml testbox=dc-8g os_mount=container + - submit iperf.yaml testbox=vm-2p8g os_mount=local + +# Optional Values +================= + +## Work flow when os_mount=local + +1. user submit job with os_mount: local + - optional kernel cmdline params: # can be added in {kernel_append_root} field of job.yaml + - save_root: give an iconic string, then the root partition data used by this job will be retained. + and you can use the root partition data of this job in the futher job by {use_root}. + - use_root : give an iconic string, then the root partition data generated by + the pervious job which has the {save_root} field will be used. + - tips: + - value of {save_root} and {use_root} need to be iconic enough, uuid is recommended. + - value of {save_root} and {use_root} will be the suffix of logical volume. + +2. scheduler return the ipxe_str to testbox + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz + imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + boot + ``` + +3. dracut step of boot + ``` + if have {use_root}; then + base_lv_name=openeuler_aarch64_20.03_{value_use_root} + lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 + + if have {save_root}; then + boot_lv_name=openeuler_aarch64_20.03_{value_save_root} + + if {save_root} == {use_root}; then + boot from /dev/mappaer/os-{boot_lv_name} + fi + + lvremove -f /dev/mappaer/os-{boot_lv_name} + create /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} + boot from /dev/mappaer/os-{boot_lv_name} + else + boot_lv_name=openeuler_aarch64_20.03_{value_use_root} + boot from /dev/mappaer/os-{boot_lv_name} + fi + + else: + base_lv_name=openeuler_aarch64_20.03_{timestamp} + lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { + # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} + mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt + mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly + lvchange -p r /dev/mapper/os-{base_lv_name} + } + + boot_lv_name=openeuler_aarch64_20.03 + if have {save_root}; then + boot_lv_name=openeuler_aarch64_20.03_{value_save_root} + fi + + lvremove -f /dev/mappaer/os-{boot_lv_name} + create /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} + boot from /dev/mappaer/os-{boot_lv_name} + fi + ``` + +4. execute the job -- 2.23.0

1 0

[PATCH v6 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 05 Feb '21

05 Feb '21

add limit for borrowing machine: runtime <= 10 days only jobs that runtime value is less than or equal to 10 days are allowed for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 89c92c1..61f18d6 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -401,6 +402,20 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + + # case runtime/sleep value count is more than 10 days, job will fail. + raise error_msg if @hash["runtime"].to_i > max_run_time + raise error_msg if @hash["sleep"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

2 1

[PATCH v2 compass-ci] sbin/compare remove change of failure metric
by Lu Weitao 05 Feb '21

05 Feb '21

[Why] We only display the failure metrics that belong the latest job, aimed to express the latest job exists error. so the change rate of failure metric is useless like: centos openeuler metric -------------------- ------------------------------ ------------------------------ fails:runs change fails:runs | | | 1:12 -8.3% 0:24 stderr./lkp/lkp/src/lib/env.sh:line#:/proc/sys/kernel/nmi_watchdog:Permission_denied [How] centos openeuler metric -------------------- ------------------------------ ------------------------------ fails:runs fails:runs | | 1:12 0:24 stderr./lkp/lkp/src/lib/env.sh:line#:/proc/sys/kernel/nmi_watchdog:Permission_denied centos openeuler metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2.638681e+10 ± 23% -0.1% 2.636343e+10 ± 22% iperf.tcp.receiver.bps 2.638853e+10 ± 23% +0.1% 2.641691e+10 ± 22% iperf.tcp.sender.bps 6.80 ± 141% -50.4% 3.37 ± 223% iperf.time.elapsed_time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 58 ++++++++--------------------------------- 1 file changed, 11 insertions(+), 47 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 6c88384..ffc6500 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -83,15 +83,12 @@ def get_values(value_list, success) end def get_compare_value(base_value_average, value_average, success) - # get compare value(change or reproduction) + # get compare value(change) # - if success - return if base_value_average.zero? + return unless success + return if base_value_average.zero? - (100 * value_average / base_value_average - 100).round(1) - else - (100 * (value_average - base_value_average)).round(1) - end + (100 * value_average / base_value_average - 100).round(1) end def field_changed?(base_values, values, success, field, options) @@ -265,7 +262,7 @@ def get_html_header(matrixes_titles, success) compare_title = CHANGE_STR else common_title = FAILS_RUNS_STR - compare_title = REPRODUCTION_STR + compare_title = '' end header = get_html_index(matrixes_titles) @@ -551,35 +548,6 @@ def format_fails_runs(fails, runs) fails_str + ':' + runs_str end -def get_reproduction_index_str(reproduction, compare_index) - reproduction_str = format('%+.1f%%', reproduction) - reproduction_index = compare_index - reproduction_str.index('.') - if reproduction_index.negative? - reproduction_str = format('%+.1e%%', reproduction).sub('e+0', 'e+').sub('e-0', 'e-') - reproduction_index = compare_index - reproduction_str.index('e') - 1 - end - return reproduction_index, reproduction_str -end - -def get_reprodcution(reproduction, compare_index) - if reproduction - reproduction_index, reproduction_str = get_reproduction_index_str(reproduction, compare_index) - space_str = ' ' * (SUB_SHORT_COLUMN_WIDTH - reproduction_str.length) - reproduction_str = space_str.insert(reproduction_index, reproduction_str) - else - reproduction_str = format("%-#{SUB_SHORT_COLUMN_WIDTH}s", '0') - end - reproduction_str -end - -def format_reproduction(reproduction, theme, compare_index) - color = get_compare_value_color(reproduction, theme) - colorize( - color, - get_reprodcution(reproduction, compare_index) - ) -end - def get_change_index_str(change, compare_index) change_str = format('%+.1f%%', change) change_index = compare_index - change_str.index('.') @@ -696,7 +664,7 @@ def get_title_name(success) compare_title = CHANGE_STR else common_title = FAILS_RUNS_STR - compare_title = REPRODUCTION_STR + compare_title = '' end return common_title, compare_title end @@ -721,14 +689,14 @@ def get_base_matrix_title_symbol(common_index, success) title_symbol end -def get_other_matrixes_title_symbol(_compare_title, matrixes_number, common_index, success) +def get_other_matrixes_title_symbol(compare_title, matrixes_number, common_index, success) title_symbol = ' ' * ( (INTERVAL_WIDTH + COLUMN_WIDTH) * matrixes_number ) start_point = 0 common_symbol = success ? '\\' : '|' - compare_symbol = '|' + compare_symbol = compare_title.empty? ? '' : '|' compare_index = SUB_SHORT_COLUMN_WIDTH / 2 (matrixes_number - 1).times do |_| @@ -789,12 +757,8 @@ def get_success_str(values, index, theme, compare_index) (change_str || '') + stddev_str end -def get_failure_str(values, index, theme, compare_index) - unless index.zero? - reproduction_str = format_reproduction( - values[:reproduction], theme, compare_index - ) - end +def get_failure_str(values, index) + reproduction_str = format("%-#{SUB_SHORT_COLUMN_WIDTH}s", '') unless index.zero? fails_runs_str = format_fails_runs( values[:fails], @@ -813,7 +777,7 @@ def get_values_str(matrixes, success, theme) ) + INTERVAL_BLANK else get_failure_str( - values, index, theme, compare_index + values, index ) + INTERVAL_BLANK end end -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/libvirt: consume job by libvirt
by Xiao Shenwei 05 Feb '21

05 Feb '21

function: 1. check host is the capability of executing tasks? - check libvirt - check qemu-kvm 2. request boot.libvirt and load files based on response 3. generate a domain.xml and start a vm used ruby-libvirt Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 68 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 64 insertions(+), 4 deletions(-) diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb index a500c31..2883197 100755 --- a/providers/libvirt/run.rb +++ b/providers/libvirt/run.rb @@ -9,6 +9,8 @@ require_relative "#{ENV['CCI_SRC']}/lib/log" require_relative "#{ENV['CCI_SRC']}/lib/sched_client" require_relative "#{ENV['CCI_SRC']}/providers/lib/context" require_relative "#{ENV['CCI_SRC']}/providers/lib/resource" +require_relative "#{ENV['CCI_SRC']}/providers/lib/domain" +require_relative "#{ENV['CCI_SRC']}/providers/lib/libvirt" def create_logger(hostname) filename = "#{hostname}.log" @@ -48,17 +50,75 @@ def request_job(context, sched_client, logger) return response end +def parse_response(response, context, resource) + context.merge!(response) + resource.parse_response(response) + context.merge!(resource.info) +end + +def generate_domain(context, logger) + domain = Domain.new(context, logger) + xml_path = '' + begin + domain.generate + xml_path = domain.save_to('domain.xml') + rescue StandardError => e + logger.error(e) + logger.error('please check your xml file format') + ensure + upload_log(context) + end + return xml_path +end + +def upload_log(context, upload_qemu_log: false) + id = context.info['job_id'] + host = context.info['LKP_SERVER'] + port = context.info['RESULT_WEBDAV_PORT'] + result_root = context.info['result_root'] + result_url = "http://#{host}:#{port}#{result_root}" + log_name = "#{context.info['hostname']}.log" + system "curl -sSf -T #{log_name} #{result_url}/ --cookie 'JOBID= #{id}'" + return unless upload_qemu_log + + qemu_log_name = "/var/log/libvirt/qemu/#{id}.log" + return unless FileTest.exist?(qemu_log_name) + + system "curl -sSf -T #{qemu_log_name} #{result_url}/ --cookie 'JOBID= #{id}'" +end + +def start(context, logger, libvirt) + xml_path = generate_domain(context, logger) + return if xml_path.empty? + + begin + libvirt.define(xml_path) + libvirt.create + libvirt.wait + rescue StandardError => e + logger.error(e) + ensure + upload_log(context, upload_qemu_log: true) + end +end + +def clean(context, sched_client, libvirt) + sched_client.delete_mac2host(context.info['mac']) + sched_client.delete_host2queues(context.info['hostname']) + libvirt.close +end + def main(hostname, queues) logger = create_logger(hostname) mac = compute_mac hostname context = Context.new(mac, hostname, queues) + libvirt = LibvirtConnect.new resource = Resource.new(hostname, logger) sched_client = SchedClient.new response = request_job(context, sched_client, logger) return if response.nil? - resource.parse_response(response) - context.merge!(resource.info) - sched_client.delete_mac2host(mac) - sched_client.delete_host2queues(hostname) + parse_response(response, context, resource) + start(context, logger, libvirt) + clean(context, sched_client, libvirt) end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow/3-code: set server_ip
by Liu Yinsi 05 Feb '21

05 Feb '21

[how] before: server_ip is the local host ip after: server_ip from config yaml /etc/compass-ci/setup.yaml [why] if deploy compass-ci client and server not in a same machine, need to config server ip in config yaml, or client can't access server services. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 46f450d..f1e946c 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -2,8 +2,13 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +mkdir -p /etc/compass-ci +cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml +server_ip=$(awk '/^server_ip:\s/ {print $2; exit}' /etc/compass-ci/setup.yaml) + +[ -z $server_ip ] && server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') + server_name=$(hostname | cut -f1 -d.) -server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${sched_host:=$server_ip} : ${sched_port:=3000} : ${os_http_host:=$server_ip} -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sparrow: add initial config setup.yaml
by Liu Yinsi 05 Feb '21

05 Feb '21

[why] before locally deploy compass-ci, fill config setup.yaml firstly, will make the whole process cleaner. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/setup.yaml | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 sparrow/setup.yaml diff --git a/sparrow/setup.yaml b/sparrow/setup.yaml new file mode 100644 index 0000000..e0197b9 --- /dev/null +++ b/sparrow/setup.yaml @@ -0,0 +1,12 @@ +# please copy this template with the below commands, +# then fill the configuration according to comments. + +# account name +my_name: xxx + +# email address +my_email: xxx(a)xxx.com + +# when you deploy compass-ci client, fill the compass-ci server's ip, +# then client can access server service. +server_ip: -- 2.23.0

1 0

[PATCH v2 compass-ci] create-image: fix error path
by Wang Chenglong 05 Feb '21

05 Feb '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 7f3d982..ed0bc14 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -37,7 +37,7 @@ setup_login() { pack_cgz() { echo "Packing package. Please wait." - find / ! -path "/tmp/${IMAGE_NAME}" | grep -vf /tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > /tmp/${IMAGE_NAME} + find ./ ! -path "./tmp/${IMAGE_NAME}" | grep -vf ./tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > ./tmp/${IMAGE_NAME} chmod 644 /tmp/${IMAGE_NAME} } -- 2.23.0

1 0

[PATCH compass-ci] container/dracut-initrd: use -qq for apt-get install
by Xu Xijian 04 Feb '21

04 Feb '21

Just remove >/dev/null 2>&1, use -qq for apt-get install will not output except for errors. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 75148a1..7394575 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -5,9 +5,9 @@ export DEBIAN_FRONTEND=noninteractive apt-get update -apt-get install -y --fix-missing --no-install-recommends apt-utils >/dev/null 2>&1 -apt-get install -y --fix-missing nfs-common netbase cifs-utils kmod rsync -apt-get install -y --fix-missing dracut dracut-network dracut-config-generic +apt-get install -y --no-install-recommends --fix-missing -qq apt-utils +apt-get install -y nfs-common netbase cifs-utils kmod rsync util-linux --fix-missing +apt-get install -y dracut dracut-network dracut-config-generic --fix-missing apt-get clean rm -rf /var/lib/apt/lists/* -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/dracut-initrd: add --fix-missing when apt-get install
by Xu Xijian 04 Feb '21

04 Feb '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 7e2bf51..75148a1 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -5,9 +5,9 @@ export DEBIAN_FRONTEND=noninteractive apt-get update -apt-get install -y --no-install-recommends apt-utils >/dev/null 2>&1 -apt-get install -y nfs-common netbase cifs-utils kmod rsync -apt-get install -y dracut dracut-network dracut-config-generic +apt-get install -y --fix-missing --no-install-recommends apt-utils >/dev/null 2>&1 +apt-get install -y --fix-missing nfs-common netbase cifs-utils kmod rsync +apt-get install -y --fix-missing dracut dracut-network dracut-config-generic apt-get clean rm -rf /var/lib/apt/lists/* -- 2.23.0

2 2

[PATCH compass-ci] git_mirror.rb: fix that git clone failure isn't known
by Li Yuanchao 04 Feb '21

04 Feb '21

Some repo git clone failed just puts "Cloning into '/srv/git/t/twitch/twitch.git'..." and nothing more. So can not take "stderr.include?('fatal')" as the condition to judge fail or not. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 8898a44..940d5e2 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -39,14 +39,12 @@ class GitMirror end def git_clone(url, mirror_dir) - ret = false url = get_url(Array(url)[0]) 10.times do - stderr = %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) - ret = !stderr.include?('fatal') - break if ret + %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) + return true if File.directory?(mirror_dir) && File.exist?("#{mirror_dir}/config") end - return ret + return false end def git_fetch(mirror_dir) -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: puts the info about the define_file
by Li Ping 04 Feb '21

04 Feb '21

[why] test the function "submit -a" always need to check define_files in job.sh puts the info about the define_files when submit job.yaml is better. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 9174cadc7..f19d1e6dc 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -680,6 +680,7 @@ class Job Dir.glob("#{LKP_SRC}/{*,*/*}/{#{program},#{program}-dev,#{program}.*}{,/*}").each do |file| file = File.realpath(file) if git_files_list.include?(file) + puts "define_file: #{file}" relative_path = file.sub("#{LKP_SRC}/", '') @job["define_files"][relative_path] = File.read(file) end -- 2.23.0

1 0

[PATCH v2 compass-ci] container/redis: delete rebundant code
by Liu Yinsi 04 Feb '21

04 Feb '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/redis/start | 5 ----- 1 file changed, 5 deletions(-) diff --git a/container/redis/start b/container/redis/start index 34266a0..5e2dc44 100755 --- a/container/redis/start +++ b/container/redis/start @@ -22,8 +22,3 @@ cmd=( ) "${cmd[@]}" - -# test -echo you can use cmd: redis-cli to start a redis client -echo and then use cmd: keys * - -- 2.23.0

1 0

[PATCH v2 compass-ci] git_mirror.rb: optimize the use of hash.merge
by Li Yuanchao 04 Feb '21

04 Feb '21

1. hash_c = hash_a.merge(hash_b) will return a new hash which contains key-values of both hash_a and hash_b, and if the two hashes have some same keys, the key-values of hash_b will be kept in hash_c. So that if a repo file has the same key with DEFAULTS file, its own will be kept. 2. fix that if there's no DEFAULTS file, the merge_defaults method will return nil. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index fa2e0cc..8898a44 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -453,10 +453,12 @@ class MirrorMain def merge_defaults(git_repo) defaults_key = File.dirname(git_repo) while defaults_key != '.' - return @git_info[git_repo].merge!(@defaults[defaults_key]) if @defaults[defaults_key] + return @defaults[defaults_key].merge(@git_info[git_repo]) if @defaults[defaults_key] defaults_key = File.dirname(defaults_key) end - return @git_info[git_repo].merge!(@defaults['default']) if @defaults['default'] + return @defaults['default'].merge(@git_info[git_repo]) if @defaults['default'] + + return @git_info[git_repo] end end -- 2.23.0

2 1

[PATCH compass-ci] redis: fix PoolTimeoutError
by Wu Zhende 04 Feb '21

04 Feb '21

[Error] Exception: No free connection (used 64 of 64) after timeout of 0.01s (Redis::PoolTimeoutError) [How] I looked at the number of concurrent Redis visits on our cluster, and it peaked at over 900. The scheduler and TaskQueue are about 50/50. So increase the size of the Redis connection pool. Adapt to our concurrency Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 1 + src/scheduler/redis_client.cr | 9 +++++++-- src/taskqueue/constants.cr | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index b90759f..dfec365 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -32,6 +32,7 @@ class Sched def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new + Redis::Client.set_pool_size(1000) @redis = Redis::Client.instance @task_queue = TaskQueueAPI.new @rgc = RemoteGitClient.new diff --git a/src/scheduler/redis_client.cr b/src/scheduler/redis_client.cr index a8d9dcb..030d1b7 100644 --- a/src/scheduler/redis_client.cr +++ b/src/scheduler/redis_client.cr @@ -12,13 +12,18 @@ class Redis::Client class_property :client HOST = (ENV.has_key?("REDIS_HOST") ? ENV["REDIS_HOST"] : JOB_REDIS_HOST) PORT = (ENV.has_key?("REDIS_PORT") ? ENV["REDIS_PORT"] : JOB_REDIS_PORT).to_i32 + @@size = 25 def self.instance Singleton::Of(self).instance end - def initialize(host = HOST, port = PORT) - @client = Redis::PooledClient.new(host: host, port: port, pool_size: 25, pool_timeout: 0.01) + def initialize(host = HOST, port = PORT, pool_size = @@size) + @client = Redis::PooledClient.new(host: host, port: port, pool_size: pool_size, pool_timeout: 0.01) + end + + def self.set_pool_size(pool_size) + @@size = pool_size end def keys(pattern) diff --git a/src/taskqueue/constants.cr b/src/taskqueue/constants.cr index c702382..6240c54 100644 --- a/src/taskqueue/constants.cr +++ b/src/taskqueue/constants.cr @@ -11,7 +11,7 @@ REDIS_PORT = 6379 # delimiter and exttract-ststs will loop consume job # when use 32 (scheduler use 25), meet Exception: # Exception: No free connection (used 32 of 32) -REDIS_POOL_NUM = 64 +REDIS_POOL_NUM = 1000 REDIS_POOL_TIMEOUT = 10 # ms -- 2.23.0

1 0