- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 2/2] sparrow: call download-rootfs script
by Liu Yinsi 29 Jan '21

29 Jan '21

For locall deploy compass-ci, call $CCI_SRC/sbin/download-rootfs to download default os rootfs openeuler aarch64 20.03. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/os-ready | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/5-build/os-ready diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready new file mode 100755 index 0000000..48a740b --- /dev/null +++ b/sparrow/5-build/os-ready @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +$CCI_SRC/sbin/download-rootfs diff --git a/sparrow/install-tiny b/sparrow/install-tiny index a291549..5f026e5 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -19,4 +19,5 @@ cd $CCI_SRC/sparrow || exit . /etc/profile.d/compass.sh 4-docker/buildall 5-build/ipxe +5-build/os-ready 7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu: add name parameter for qemu-kvm
by Xiao Shenwei 29 Jan '21

29 Jan '21

[why] when we need reboot a vm, we need get it's pid from hostname, then check job_id is the same as required. [how] -name guest=$hostname,process=$job_id if you need reboot the vm: pkill $job_id Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..dd657cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -39,6 +39,9 @@ parse_ipxe_script() ;; initrd) file=$(basename "$b") + [ $file == "job.cgz" ] && { + job_id=$(basename $(dirname "$b")) + } wget --timestamping -a ${log_file} --progress=bar:force $b initrds+="$file " ;; @@ -144,6 +147,7 @@ public_option() { kvm=( $qemu + -name guest=$hostname,process=$job_id -kernel $kernel -initrd $initrd -smp $nr_cpu -- 2.23.0

2 2

[PATCH compass-ci] lib/compare_matrixes.rb: treatment of stddev < 1%
by Lu Weitao 29 Jan '21

29 Jan '21

[Why] if stddev < 1% | == 0 We display nothing for stddev column like bellow, it may confuse user. 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 276589.17 -5.2% 262249.03 redis.get_avg_throughput_rps 61.47 +5.5% 64.83 redis.get_avg_time_sec 58.33 ± 53% -100.0% 0.00 redis.get_latency_0ms% [How] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 276589.17 < 1% -5.2% 262249.03 < 1% redis.get_avg_throughput_rps 61.47 < 1% +5.5% 64.83 < 1% redis.get_avg_time_sec 58.33 ± 53% -100.0% 0.00 redis.get_latency_0ms% Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 7760c93..6c88384 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -72,7 +72,7 @@ def get_values(value_list, success) if length > 1 && average != 0 stddev_percent = ( standard_deviation(value_list, average, length) * 100 / average - ).to_i + ) end { average: average, stddev_percent: stddev_percent, min: sorted[0], max: sorted[-1], sorted: sorted } @@ -204,7 +204,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, latest_jobs = nil, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs=nil, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -615,6 +615,8 @@ def format_stddev_percent(stddev_percent, average_width) percent_width = SUB_LONG_COLUMN_WIDTH - average_width if stddev_percent if stddev_percent != 0 + return " < 1% " if stddev_percent < 1 + percent_str = get_suitable_number_str( stddev_percent.abs, percent_width - 4, -- 2.23.0

1 0

[PATCH compass-ci 1/2] srv-http/start: add os images data download directory
by Cui Yili 29 Jan '21

29 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index b56fe4d..8c8ca0b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -14,6 +14,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/os:/srv/os:ro -d srv-http ) -- 2.23.0

3 4

[PATCH v4 compass-ci 3/3] replace log.sh
by Liu Yinsi 29 Jan '21

29 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 2 +- container/os-cifs/start | 2 +- container/os-nfs/start | 2 +- container/result-cifs/start | 2 +- providers/my-qemu.sh | 4 +--- providers/qemu/kvm.sh | 3 +-- providers/qemu/pxe.sh | 4 +--- sparrow/4-docker/buildall | 6 +++--- 8 files changed, 10 insertions(+), 15 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 0b9601a..6a91b37 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { log_error "kernel module cifs is missing (try: sudo modprobe cifs)" diff --git a/container/os-cifs/start b/container/os-cifs/start index 09ae3d2..85017cc 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { log_error "kernel module cifs is missing (try: sudo modprobe cifs)" diff --git a/container/os-nfs/start b/container/os-nfs/start index 9539561..1f4ed42 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { log_error "kernel module nfs is missing (try: sudo modprobe nfs)" diff --git a/container/result-cifs/start b/container/result-cifs/start index de38ca2..7ce066a 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { log_error "kernel module cifs is missing (try: sudo modprobe cifs)" diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 113b3ef..3e3b2f4 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,9 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh help() { diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..d394681 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,9 +6,8 @@ : ${nr_cpu:=1} : ${memory:=1G} -: ${LKP_SRC:="/c/lkp-tests"} -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh check_logfile() { diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 0ba052c..3ab3c2b 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -4,9 +4,7 @@ # - nr_cpu # - memory -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh : ${nr_cpu:=1} : ${memory:=1G} diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 014036b..8d6a85a 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -5,7 +5,7 @@ CONTAINER_PATH="$CCI_SRC/container" export action=$1 -[ -n "$LKP_SRC" ] && source "${LKP_SRC}/lib/log.sh" +source ${CCI_SRC}/lib/log.sh build_depends() { @@ -27,7 +27,7 @@ do_one() lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock echo "$started_containers" | grep -w -q "$container_name" && return 0 - [ -n "$LKP_SRC" ] && log_info "start build $container." + log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" @@ -41,7 +41,7 @@ do_one() [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run ) - [ -n "$LKP_SRC" ] && log_info "finish build $container." + log_info "finish build $container." started_containers+="$container_name" lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container: delete redundant log error info
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: log_error "[ERROR] kernel module ..." after: log_error "kernel module ..." [why] log_error(): echo "$(date +'%F %T') [ERROR] -- $*" >&2 when call log_error(), already has complete log format, so delete redundant "[ERROR]" in parmas. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 2 +- container/os-cifs/start | 2 +- container/os-nfs/start | 4 ++-- container/result-cifs/start | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 119d1b1..0b9601a 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -6,7 +6,7 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-cifs/start b/container/os-cifs/start index 47da260..09ae3d2 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -6,7 +6,7 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-nfs/start b/container/os-nfs/start index 8c08db8..9539561 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -6,11 +6,11 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { - log_error "[ERROR] kernel module nfs is missing (try: sudo modprobe nfs)" + log_error "kernel module nfs is missing (try: sudo modprobe nfs)" exit 1 } lsmod | grep -q "^nfsd\s" || { - log_error "[ERROR] kernel module nfsd is missing (try: sudo modprobe nfsd)" + log_error "kernel module nfsd is missing (try: sudo modprobe nfsd)" exit 1 } diff --git a/container/result-cifs/start b/container/result-cifs/start index 4f8c2b9..de38ca2 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -6,7 +6,7 @@ . $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] lib: add log.sh
by Liu Yinsi 29 Jan '21

29 Jan '21

Records the run logs of the shell program Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 0000000..626a6e4 --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +log_info() +{ + date +"[INFO] %F %T ${0##*/}: $*" +} + +log_debug() +{ + date +"[DEBUG] %F %T ${0##*/}: $*" +} + +log_warn() +{ + date +"[WARN] %F %T ${0##*/}: $*" >&2 +} + +log_error() +{ + date +"[ERROR] %F %T ${0##*/}: $*" >&2 +} -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] replace log.sh
by Liu Yinsi 29 Jan '21

29 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 4 ++-- container/os-cifs/start | 4 ++-- container/os-nfs/start | 4 ++-- container/result-cifs/start | 4 ++-- providers/my-qemu.sh | 4 +--- providers/qemu/kvm.sh | 3 +-- providers/qemu/pxe.sh | 4 +--- sparrow/4-docker/buildall | 8 ++++---- 8 files changed, 15 insertions(+), 20 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 119d1b1..6a91b37 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-cifs/start b/container/os-cifs/start index 47da260..85017cc 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-nfs/start b/container/os-nfs/start index 8c08db8..7577647 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { - log_error "[ERROR] kernel module nfs is missing (try: sudo modprobe nfs)" + log_error "kernel module nfs is missing (try: sudo modprobe nfs)" exit 1 } lsmod | grep -q "^nfsd\s" || { diff --git a/container/result-cifs/start b/container/result-cifs/start index 4f8c2b9..7ce066a 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 113b3ef..3e3b2f4 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,9 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh help() { diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..d394681 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,9 +6,8 @@ : ${nr_cpu:=1} : ${memory:=1G} -: ${LKP_SRC:="/c/lkp-tests"} -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh check_logfile() { diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 0ba052c..3ab3c2b 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -4,9 +4,7 @@ # - nr_cpu # - memory -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh : ${nr_cpu:=1} : ${memory:=1G} diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a60067f..9d6c977 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -2,11 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +source ${CCI_SRC}/lib/log.sh + CONTAINER_PATH="$CCI_SRC/container" export action=$1 -[ -n "$LKP_SRC" ] && source "${LKP_SRC}/lib/log.sh" - build_depends() { local container=$1 @@ -27,7 +27,7 @@ do_one() lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock echo "$started_containers" | grep -w -q "$container_name" && return 0 - [ -n "$LKP_SRC" ] && log_info "start build $container." + log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" @@ -40,7 +40,7 @@ do_one() [ -x first-run ] && ./first-run [ -x start ] && ./start ) - [ -n "$LKP_SRC" ] && log_info "finish build $container." + log_info "finish build $container." started_containers+="$container_name" lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

2 4

[PATCH compass-ci] sparrow/4-docker: set param value
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: action=reboot after: action=run-only [why] new param value more easy to understand. usage 1: ./buildall -> execute $container/build, install first-run and start script usage 2: ./buildall run-only -> just execute $continer/first-run, start script, no build, install script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- sparrow/7-systemd/cci-network | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 014036b..3c0b7b0 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" != 0 ] || [ "$action" != "reboot" ]; then + if [ "$?" != 0 ] || [ "$action" != "run-only" ]; then [ -x build ] && ./build [ -x install ] && ./install fi diff --git a/sparrow/7-systemd/cci-network b/sparrow/7-systemd/cci-network index 6e59dbb..0438935 100755 --- a/sparrow/7-systemd/cci-network +++ b/sparrow/7-systemd/cci-network @@ -14,4 +14,4 @@ $CCI_SRC/sparrow/2-network/cifs # --restart=always option is not absolutely reliable # container need to start its dependences firstly. -$CCI_SRC/sparrow/4-docker/buildall reboot +$CCI_SRC/sparrow/4-docker/buildall run-only -- 2.23.0

1 0

[PATCH v2 compass-ci] git_mirror.rb: change the new_refs_count hash
by Li Yuanchao 29 Jan '21

29 Jan '21

Flatten the hash new_refs_count{}, to make it easier to query. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 2090f6b..13cb7c0 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -378,7 +378,7 @@ class MirrorMain offset_new_refs: 0, new_refs_count: {} } - query = { query: { match: { git_repo: git_repo } } } + query = { query: { match: { _id: git_repo } } } result = @es_client.search(index: 'repo', body: query)['hits'] return fork_stat unless result['total'].positive? @@ -389,37 +389,45 @@ class MirrorMain end def create_year_hash(new_refs_count, year, month, day) - new_refs_count[year] = { 'year_count' => 1, month => { 'month_count' => 1, day => 1 } } + new_refs_count[year] = 1 + new_refs_count[month] = 1 + new_refs_count[day] = 1 return new_refs_count end def update_year_hash(new_refs_count, year, month, day) - new_refs_count[year]['year_count'] += 1 - return create_month_hash(new_refs_count, year, month, day) if new_refs_count[year][month].nil? + new_refs_count[year] += 1 + return create_month_hash(new_refs_count, month, day) if new_refs_count[month].nil? - return update_month_hash(new_refs_count, year, month, day) + return update_month_hash(new_refs_count, month, day) end - def create_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month] = { 'month_count' => 1, day => 1 } + def create_month_hash(new_refs_count, month, day) + new_refs_count[month] = 1 + new_refs_count[day] = 1 + return new_refs_count end - def update_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month]['month_count'] += 1 - if new_refs_count[year][month][day].nil? - new_refs_count[year][month][day] = 1 + def update_month_hash(new_refs_count, month, day) + new_refs_count[month] += 1 + if new_refs_count[day].nil? + new_refs_count[day] = 1 else - new_refs_count[year][month][day] += 1 + new_refs_count[day] += 1 end return new_refs_count end def update_new_refs_count(new_refs_count) t = Time.now - year = t.year.to_s - month = t.month.to_s - day = t.day.to_s + + # example: 2021-01-28 + day = t.strftime('%Y-%m-%d') + # example: 2021-01 + month = t.strftime('%Y-%m') + # example: 2021 + year = t.strftime('%Y') return create_year_hash(new_refs_count, year, month, day) if new_refs_count[year].nil? return update_year_hash(new_refs_count, year, month, day) -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] container/defconfig: load config yaml
by Liu Yinsi 29 Jan '21

29 Jan '21

Load "/etc/compass-ci/register/*.yaml", get register account info from cci defaults config. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index fd13312..f9b0824 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -9,6 +9,7 @@ def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', '/etc/compass-ci/accounts/*.yaml', + '/etc/compass-ci/register/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| hash.update YAML.load_file(file) || {} end -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container/auto-submit: bind mount volume
by Liu Yinsi 29 Jan '21

29 Jan '21

mount /etc/compass-ci/register, to get auto-submit account info from config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/auto-submit/start b/container/auto-submit/start index 4632ab0..79463f6 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,6 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro + -v /etc/compass-ci/register:/etc/compass-ci/register:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] sbin/auto-submit: config account yaml
by Liu Yinsi 29 Jan '21

29 Jan '21

[how] before: define my_name, my_email, lab, then store into ES and config yaml after: query my_name, my_email, lab, my_token from ES, then config yaml [why] store account info into ES, this part already done by container service register-accounts, so when start auto-submit service, no need to store data, only query ES and config yaml. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 5648d59..d8c4b5a 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -7,6 +7,8 @@ require 'bunny' require 'json' require 'yaml' require_relative '../lib/build_my_info_client' +require 'set' +require "#{ENV['CCI_SRC']}/container/defconfig" # receive message and auto submit job class AutoSubmit @@ -86,15 +88,30 @@ class AutoSubmit end end -option = { - my_name: 'Auto Submit Robot', - my_email: 'autosubmit@localhost', - lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def my_email + names = Set.new %w[auto-submit] + + defaults = relevant_defaults(names) + defaults['auto-submit']['my_email'] +end + +def query_account_info + option = ESQuery.new(index: 'accounts').query_by_id(my_email) + raise "query auto-submit account info failed: #{my_email}" unless option + + option +end + +def config_yaml + option = query_account_info + + build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) + build_my_info.config_default_yaml + build_my_info.config_lab_yaml +end # fix verification problems when submit jobs by auto-submit service -build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) -build_my_info.config_my_info +config_yaml auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 0

[PATCH compass-ci] lib/scheduler_api.cr: provide scheduler api
by Wu Zhende 29 Jan '21

29 Jan '21

Enable other crystal services to invoke the scheduler api. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 src/lib/scheduler_api.cr diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr new file mode 100644 index 0000000..b8dd514 --- /dev/null +++ b/src/lib/scheduler_api.cr @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "http/client" + +# scheduler API class +class SchedulerAPI + def initialize + @port = ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"].to_i32 : 3000 + @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" + end + + def close_job(job_id) + client = HTTP::Client.new(@host, port: @port) + response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + client.close() + return response + end +end -- 2.23.0

1 0

[PATCH compass-ci] doc: add support-os-os_arch-os_version.md
by Xu Xijian 29 Jan '21

29 Jan '21

This md document contains the currently supporting os in our server. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/support-os-os_arch-os_version.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 doc/support-os-os_arch-os_version.md diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md new file mode 100644 index 0000000..6532f92 --- /dev/null +++ b/doc/support-os-os_arch-os_version.md @@ -0,0 +1,22 @@ +``` +os os_arch os_version supported-os_mount +openeuler aarch64 1.0 cifs/nfs, + 20.03 cifs/nfs, + 20.09 cifs/nfs, + mainline-dailybuild cifs/nfs, + 20.03-dailybuild cifs/nfs, + 20.09-dailybuild cifs/nfs, + 20.03-sp1-dailybuild cifs/nfs, + x86 20.03 cifs/nfs, + 20.09 cifs/nfs, +centos aarch64 7 container, + 7.6.1810 cifs/nfs, + 7.8.2003 cifs/nfs, + 8.1.1911 cifs/nfs, + 8 container, + x86 7.6.1810 cifs/nfs, + 7.8.2003 cifs/nfs, +debian aarch64 sid cifs/nfs, + x86 sid cifs/nfs, +archlinux aarch64 2020-11-12 cifs/nfs, +``` -- 2.23.0

3 4

[PATCH v3 compass-ci] lib/es_jobs.rb: more accurate matching summary info
by Wu Zhende 29 Jan '21

29 Jan '21

1. The fail/error/warning cases should not take input from stderr.* stderr is rather chaotic -- let it stay in any_stderr. 2. The warning match should be more exact. Avoid matching nr_warning So does error Don't use stderr.* as examples for them. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index e367a91..2d2f3bf 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -33,29 +33,30 @@ class ESJobs def set_job_summary(stats, job) summary_result = '' stats.each_key do |stat| - # "stderr.warning:rubygems_will_be_installed_before_its_ruby_dependency": 1, - # "stderr._#warning_FORTIFY_SOURCE_requires_compiling_with_optimization(-O)": 1, - # "stderr.disk_src.c:#:#:warning:incompatible_implicit_declaration_of_built-in_function'strcpy'": 1, - if stat.match(/warning/i) - job['summary.any_warning'] = 1 - summary_result = 'warning' - end # "stderr.linux-perf": 1, # "stderr.error:target_not_found:ruby-dev": 1, # "stderr.error:could_not_open_file/var/lib/pacman/local/ldb-#:#-#/desc:Not_a_directory": 1, if stat.match(/stderr\./i) job['summary.any_stderr'] = 1 summary_result = 'stderr' + next end - # "stderr.::Proceed_with_installation?[Y/n]error:target_not_found:liblzma-dev": 1, - # "stderr.==>ERROR:Failure_while_downloading_apache-cassandra": 1, - # "stderr.ftq.h:#:#:error:unknown_type_name'ticks'": 1, + + # sum.stats.build-pkg.mb_cache.c:warning:‘read_cache’defined-but-not-used[-Wunused-function]: 1 + # sum.stats.build-pkg.mb_cache.c:warning:control-reaches-end-of-non-void-function[-Wreturn-type]: 1 + if stat.match(/:warning:|\.warning$/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + # "last_state.test.iperf.exit_code.127": 1, # "last_state.test.cci-makepkg.exit_code.1": 1, - if stat.match(/error|\.exit_code\./i) + # sum.stats.build-pkg.cc1plus:error:unrecognized-command-line-option‘-Wno-unknown-warning-option’[-Werror]: 2 + if stat.match(/:error:|\.error$|\.exit_code\./i) job['summary.any_error'] = 1 summary_result = 'error' end + if stat.match(/\.fail$/i) job['summary.any_fail'] = 1 summary_result = 'fail' -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: change the new_refs_count hash
by Li Yuanchao 29 Jan '21

29 Jan '21

Flatten the hash new_refs_count{}, to make it easier to query. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 40 ++++++++++++++++++++++++---------------- 1 file changed, 24 insertions(+), 16 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 2090f6b..95503ed 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -378,7 +378,7 @@ class MirrorMain offset_new_refs: 0, new_refs_count: {} } - query = { query: { match: { git_repo: git_repo } } } + query = { query: { match: { _id: git_repo } } } result = @es_client.search(index: 'repo', body: query)['hits'] return fork_stat unless result['total'].positive? @@ -389,37 +389,45 @@ class MirrorMain end def create_year_hash(new_refs_count, year, month, day) - new_refs_count[year] = { 'year_count' => 1, month => { 'month_count' => 1, day => 1 } } + new_refs_count[year] = 1 + new_refs_count[month] = 1 + new_refs_count[day] = 1 return new_refs_count end def update_year_hash(new_refs_count, year, month, day) - new_refs_count[year]['year_count'] += 1 - return create_month_hash(new_refs_count, year, month, day) if new_refs_count[year][month].nil? + new_refs_count[year] += 1 + return create_month_hash(new_refs_count, month, day) if new_refs_count[month].nil? - return update_month_hash(new_refs_count, year, month, day) + return update_month_hash(new_refs_count, month, day) end - def create_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month] = { 'month_count' => 1, day => 1 } + def create_month_hash(new_refs_count, month, day) + new_refs_count[month] = 1 + new_refs_count[day] = 1 + return new_refs_count end - def update_month_hash(new_refs_count, year, month, day) - new_refs_count[year][month]['month_count'] += 1 - if new_refs_count[year][month][day].nil? - new_refs_count[year][month][day] = 1 + def update_month_hash(new_refs_count, month, day) + new_refs_count[month] += 1 + if new_refs_count[day].nil? + new_refs_count[day] = 1 else - new_refs_count[year][month][day] += 1 + new_refs_count[day] += 1 end return new_refs_count end def update_new_refs_count(new_refs_count) - t = Time.now - year = t.year.to_s - month = t.month.to_s - day = t.day.to_s + t = Time.now.to_s + + # example: 2021-01-28 + day = t[0..9] + # example: 2021-01 + month = t[0..6] + # example: 2021 + year = t[0..3] return create_year_hash(new_refs_count, year, month, day) if new_refs_count[year].nil? return update_year_hash(new_refs_count, year, month, day) -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/4-docker: pack lkp-$arch.cgz
by Liu Yinsi 29 Jan '21

29 Jan '21

Pack lkp-$arch.cgz by deploy script, it is convenient for users to submit jobs, it doesn't need to be executed by the user. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a60067f..014036b 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -39,6 +39,7 @@ do_one() fi [ -x first-run ] && ./first-run [ -x start ] && ./start + [ "$container_name" == 'initrd-lkp' ] && ./run ) [ -n "$LKP_SRC" ] && log_info "finish build $container." started_containers+="$container_name" -- 2.23.0

2 2

[PATCH compass-ci 1/2] sparrow: move gem pkg into install script
by Liu Yinsi 29 Jan '21

29 Jan '21

Easy to call individually common script without internet Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 2 -- sparrow/0-package/install | 2 +- sparrow/install-tiny | 1 + 3 files changed, 2 insertions(+), 3 deletions(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..ccdc3f3 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,8 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch - grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer grep -q '^team:' /etc/group || groupadd team diff --git a/sparrow/0-package/install b/sparrow/0-package/install index 2b9a8f5..e01fc45 100755 --- a/sparrow/0-package/install +++ b/sparrow/0-package/install @@ -6,4 +6,4 @@ source /etc/os-release . $(dirname ${BASH_SOURCE[0]})/os/${ID} -. $(dirname ${BASH_SOURCE[0]})/common +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch diff --git a/sparrow/install-tiny b/sparrow/install-tiny index a291549..cb47cf5 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -8,6 +8,7 @@ cd $CCI_SRC/sparrow || exit 0-package/install +0-package/common 1-storage/tiny 1-storage/permission 2-network/br0 -- 2.23.0

2 2

[PATCH v3 compass-ci 1/2] lib: add log.sh
by Liu Yinsi 29 Jan '21

29 Jan '21

Records the run logs of the shell program Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 0000000..626a6e4 --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +log_info() +{ + date +"[INFO] %F %T ${0##*/}: $*" +} + +log_debug() +{ + date +"[DEBUG] %F %T ${0##*/}: $*" +} + +log_warn() +{ + date +"[WARN] %F %T ${0##*/}: $*" >&2 +} + +log_error() +{ + date +"[ERROR] %F %T ${0##*/}: $*" >&2 +} -- 2.23.0

3 4

[PATCH v2 compass-ci] sparrow/0-package: set umask 002
by Liu Yinsi 29 Jan '21

29 Jan '21

Insure programs have necessary permission to work normally. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..6b4a834 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -27,6 +27,12 @@ cat >> /etc/modules-load.d/cifs <<EOF cifs EOF +cat >> /etc/bashrc <<EOF +umask 002 +EOF + +source /etc/bashrc + server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${DOCKER_REGISTRY_HOST:=$server_ip} -- 2.23.0

2 2

[PATCH v2 compass-ci] lib/es_jobs.rb: more accurate matching summary info
by Wu Zhende 29 Jan '21

29 Jan '21

1. The fail/error/warning cases should not take input from stderr.* stderr is rather chaotic -- let it stay in any_stderr. 2. The warning match should be more exact. Avoid matching nr_warning So does error Don't use stderr.* as examples for them. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index e367a91..9e0c671 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -33,29 +33,30 @@ class ESJobs def set_job_summary(stats, job) summary_result = '' stats.each_key do |stat| - # "stderr.warning:rubygems_will_be_installed_before_its_ruby_dependency": 1, - # "stderr._#warning_FORTIFY_SOURCE_requires_compiling_with_optimization(-O)": 1, - # "stderr.disk_src.c:#:#:warning:incompatible_implicit_declaration_of_built-in_function'strcpy'": 1, - if stat.match(/warning/i) - job['summary.any_warning'] = 1 - summary_result = 'warning' - end # "stderr.linux-perf": 1, # "stderr.error:target_not_found:ruby-dev": 1, # "stderr.error:could_not_open_file/var/lib/pacman/local/ldb-#:#-#/desc:Not_a_directory": 1, if stat.match(/stderr\./i) job['summary.any_stderr'] = 1 summary_result = 'stderr' + next end - # "stderr.::Proceed_with_installation?[Y/n]error:target_not_found:liblzma-dev": 1, - # "stderr.==>ERROR:Failure_while_downloading_apache-cassandra": 1, - # "stderr.ftq.h:#:#:error:unknown_type_name'ticks'": 1, + + # sum.stats.build-pkg.mb_cache.c:warning:‘read_cache’defined-but-not-used[-Wunused-function]: 1 + # sum.stats.build-pkg.mb_cache.c:warning:control-reaches-end-of-non-void-function[-Wreturn-type]: 1 + if stat.match(/warning/i) && !stat.match(/nr_warning/i) + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + # "last_state.test.iperf.exit_code.127": 1, # "last_state.test.cci-makepkg.exit_code.1": 1, - if stat.match(/error|\.exit_code\./i) + # sum.stats.build-pkg.cc1plus:error:unrecognized-command-line-option‘-Wno-unknown-warning-option’[-Werror]: 2 + if stat.match(/error|\.exit_code\./i) && !stat.match(/nr_error/i) job['summary.any_error'] = 1 summary_result = 'error' end + if stat.match(/\.fail$/i) job['summary.any_fail'] = 1 summary_result = 'fail' -- 2.23.0

2 1

[PATCH v3 lab-z9] modify the lab-z9/allot/idle
by Zhang Dewan 28 Jan '21

28 Jan '21

dewan@crystal ~/128/lab-z9/allot% tree . . └── idle ├── taishan200-2280-2s48p-256g--a11 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a12 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a13 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a14 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a15 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a16 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a17 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a18 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a19 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a20 -> taishan200-2280-2s48p-256g--multi-docker ├── taishan200-2280-2s48p-256g--a60 -> taishan200-2280-2s48p-256g--multi-qemu ├── taishan200-2280-2s48p-256g--a61 -> taishan200-2280-2s48p-256g--multi-qemu ├── taishan200-2280-2s48p-256g--a62 -> taishan200-2280-2s48p-256g--multi-qemu ├── taishan200-2280-2s48p-256g--multi-docker │ └── multi-docker.yaml └── taishan200-2280-2s48p-256g--multi-qemu └── multi-qemu.yaml Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- allot/idle/taishan200-2280-2s48p-256g--a11 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a12 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a13 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a14 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a15 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a16 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a17 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a18 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a19 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a20 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a21 | 1 - allot/idle/taishan200-2280-2s48p-256g--a22 | 1 - allot/idle/taishan200-2280-2s48p-256g--a24 | 1 - allot/idle/taishan200-2280-2s48p-256g--a25 | 1 - allot/idle/taishan200-2280-2s48p-256g--a26 | 1 - allot/idle/taishan200-2280-2s48p-256g--a27 | 1 - allot/idle/taishan200-2280-2s48p-256g--a28 | 1 - allot/idle/taishan200-2280-2s48p-256g--a29 | 1 - allot/idle/taishan200-2280-2s48p-256g--a30 | 1 - allot/idle/taishan200-2280-2s48p-256g--a31 | 1 - allot/idle/taishan200-2280-2s48p-256g--a32 | 1 - allot/idle/taishan200-2280-2s48p-256g--a33 | 1 - allot/idle/taishan200-2280-2s48p-256g--a34 | 1 - allot/idle/taishan200-2280-2s48p-256g--a35 | 1 - allot/idle/taishan200-2280-2s48p-256g--a60 | 1 + allot/idle/taishan200-2280-2s48p-256g--a61 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a62 | 2 +- allot/idle/taishan200-2280-2s48p-256g--a63 | 1 - allot/idle/taishan200-2280-2s48p-256g--a64 | 1 - allot/idle/taishan200-2280-2s48p-256g--a65 | 1 - allot/idle/taishan200-2280-2s48p-256g--a66 | 1 - allot/idle/taishan200-2280-2s48p-256g--a67 | 1 - allot/idle/taishan200-2280-2s48p-256g--a68 | 1 - allot/idle/taishan200-2280-2s48p-256g--a69 | 1 - allot/idle/taishan200-2280-2s48p-256g--a70 | 1 - allot/idle/taishan200-2280-2s48p-256g--a71 | 1 - .../multi-docker.yaml | 17 +++++++++++++++++ .../multi-qemu.yaml | 0 38 files changed, 30 insertions(+), 35 deletions(-) delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a21 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a22 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a24 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a25 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a26 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a27 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a28 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a29 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a30 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a31 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a32 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a33 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a34 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a35 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a60 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a63 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a64 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a65 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a66 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a67 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a68 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a69 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a70 delete mode 120000 allot/idle/taishan200-2280-2s48p-256g--a71 create mode 100644 allot/idle/taishan200-2280-2s48p-256g--multi-docker/multi-docker.yaml rename allot/idle/{taishan200-2280-2s48p-256g--a60 => taishan200-2280-2s48p-256g--multi-qemu}/multi-qemu.yaml (100%) diff --git a/allot/idle/taishan200-2280-2s48p-256g--a11 b/allot/idle/taishan200-2280-2s48p-256g--a11 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a11 +++ b/allot/idle/taishan200-2280-2s48p-256g--a11 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a12 b/allot/idle/taishan200-2280-2s48p-256g--a12 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a12 +++ b/allot/idle/taishan200-2280-2s48p-256g--a12 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a13 b/allot/idle/taishan200-2280-2s48p-256g--a13 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a13 +++ b/allot/idle/taishan200-2280-2s48p-256g--a13 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a14 b/allot/idle/taishan200-2280-2s48p-256g--a14 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a14 +++ b/allot/idle/taishan200-2280-2s48p-256g--a14 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a15 b/allot/idle/taishan200-2280-2s48p-256g--a15 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a15 +++ b/allot/idle/taishan200-2280-2s48p-256g--a15 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a16 b/allot/idle/taishan200-2280-2s48p-256g--a16 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a16 +++ b/allot/idle/taishan200-2280-2s48p-256g--a16 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a17 b/allot/idle/taishan200-2280-2s48p-256g--a17 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a17 +++ b/allot/idle/taishan200-2280-2s48p-256g--a17 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a18 b/allot/idle/taishan200-2280-2s48p-256g--a18 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a18 +++ b/allot/idle/taishan200-2280-2s48p-256g--a18 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a19 b/allot/idle/taishan200-2280-2s48p-256g--a19 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a19 +++ b/allot/idle/taishan200-2280-2s48p-256g--a19 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a20 b/allot/idle/taishan200-2280-2s48p-256g--a20 index 479a75a..d9306d4 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a20 +++ b/allot/idle/taishan200-2280-2s48p-256g--a20 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-docker \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a21 b/allot/idle/taishan200-2280-2s48p-256g--a21 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a21 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a22 b/allot/idle/taishan200-2280-2s48p-256g--a22 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a22 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a24 b/allot/idle/taishan200-2280-2s48p-256g--a24 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a24 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a25 b/allot/idle/taishan200-2280-2s48p-256g--a25 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a25 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a26 b/allot/idle/taishan200-2280-2s48p-256g--a26 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a26 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a27 b/allot/idle/taishan200-2280-2s48p-256g--a27 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a27 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a28 b/allot/idle/taishan200-2280-2s48p-256g--a28 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a28 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a29 b/allot/idle/taishan200-2280-2s48p-256g--a29 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a29 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a30 b/allot/idle/taishan200-2280-2s48p-256g--a30 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a30 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a31 b/allot/idle/taishan200-2280-2s48p-256g--a31 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a31 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a32 b/allot/idle/taishan200-2280-2s48p-256g--a32 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a32 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a33 b/allot/idle/taishan200-2280-2s48p-256g--a33 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a33 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a34 b/allot/idle/taishan200-2280-2s48p-256g--a34 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a34 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a35 b/allot/idle/taishan200-2280-2s48p-256g--a35 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a35 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a60 b/allot/idle/taishan200-2280-2s48p-256g--a60 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a60 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a61 b/allot/idle/taishan200-2280-2s48p-256g--a61 index 479a75a..3c3d0c3 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a61 +++ b/allot/idle/taishan200-2280-2s48p-256g--a61 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a62 b/allot/idle/taishan200-2280-2s48p-256g--a62 index 479a75a..3c3d0c3 120000 --- a/allot/idle/taishan200-2280-2s48p-256g--a62 +++ b/allot/idle/taishan200-2280-2s48p-256g--a62 @@ -1 +1 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a63 b/allot/idle/taishan200-2280-2s48p-256g--a63 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a63 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a64 b/allot/idle/taishan200-2280-2s48p-256g--a64 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a64 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a65 b/allot/idle/taishan200-2280-2s48p-256g--a65 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a65 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a66 b/allot/idle/taishan200-2280-2s48p-256g--a66 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a66 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a67 b/allot/idle/taishan200-2280-2s48p-256g--a67 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a67 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a68 b/allot/idle/taishan200-2280-2s48p-256g--a68 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a68 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a69 b/allot/idle/taishan200-2280-2s48p-256g--a69 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a69 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a70 b/allot/idle/taishan200-2280-2s48p-256g--a70 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a70 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a71 b/allot/idle/taishan200-2280-2s48p-256g--a71 deleted file mode 120000 index 479a75a..0000000 --- a/allot/idle/taishan200-2280-2s48p-256g--a71 +++ /dev/null @@ -1 +0,0 @@ -taishan200-2280-2s48p-256g--a60 \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-docker/multi-docker.yaml b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/multi-docker.yaml new file mode 100644 index 0000000..8e42208 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-docker/multi-docker.yaml @@ -0,0 +1,17 @@ +suite: multi-docker +category: functional +simplify-ci: +runtime: 100d +multi-docker-0: + nr_container: 20 + tbox_group: dc-1g +multi-docker-1: + nr_container: 10 + tbox_group: dc-2g +multi-docker-2: + nr_container: 5 + tbox_group: dc-4g +multi-docker-3: + nr_container: 20 + tbox_group: dc-8g +sleep: diff --git a/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml similarity index 100% rename from allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml rename to allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml -- 2.23.0

1 0

[PATCH compass-ci 2/3] scheduler/redis_client.cr: add singleton pattern
by Wu Zhende 28 Jan '21

28 Jan '21

Singleton pattern will only generate an instance. The Redis pool needs to be initialized only once and the connection is obtained from the pool. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/redis_client.cr | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/scheduler/redis_client.cr b/src/scheduler/redis_client.cr index 1253108..a8d9dcb 100644 --- a/src/scheduler/redis_client.cr +++ b/src/scheduler/redis_client.cr @@ -6,12 +6,17 @@ require "redis" require "./constants" require "../lib/job" +require "singleton" class Redis::Client class_property :client HOST = (ENV.has_key?("REDIS_HOST") ? ENV["REDIS_HOST"] : JOB_REDIS_HOST) PORT = (ENV.has_key?("REDIS_PORT") ? ENV["REDIS_PORT"] : JOB_REDIS_PORT).to_i32 + def self.instance + Singleton::Of(self).instance + end + def initialize(host = HOST, port = PORT) @client = Redis::PooledClient.new(host: host, port: port, pool_size: 25, pool_timeout: 0.01) end -- 2.23.0

2 1

[PATCH compass-ci 3/3] lib/sched.cr: redis use singleton pattern
by Wu Zhende 28 Jan '21

28 Jan '21

[Why] A redis pool is created for each HTTP request. This is wrong. This can cause too many connections to be established. All HTTP requests should share a redis pool. [How] Use singleton pattern Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f46d97c..b90759f 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -32,7 +32,7 @@ class Sched def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new - @redis = Redis::Client.new + @redis = Redis::Client.instance @task_queue = TaskQueueAPI.new @rgc = RemoteGitClient.new @env = env -- 2.23.0

1 0