- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] sparrow/4-docker/buildall: build and start all in parallel to speedup
by Liu Yinsi 29 Mar '21

29 Mar '21

before: function build_depends() are parallel, but do_one_build(), start_depends(), do_one_run() are not parallel. after: function build_depends(), do_one_build(), start_depends(), do_one_run() all are parallel. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 3cd5719..3611158 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -24,7 +24,7 @@ do_one_build() { local container=$1 local container_name=$(basename $container) - lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock + lockfile-create -q --use-pid --retry 100 --lock-name "build_${container_name}".lock mkdir $tmpdir/$container_name 2>/dev/null && ( @@ -33,7 +33,7 @@ do_one_build() [ "$?" == 0 ] && log_info "finish build $container" [ -x install ] && ./install ) - lockfile-remove --lock-name "$container_name".lock + lockfile-remove --lock-name "build_${container_name}".lock } start_depends() @@ -42,8 +42,9 @@ start_depends() for dep in $(cat $container/start-depends 2> /dev/null) do - start_depends $CONTAINER_PATH/$dep + start_depends $CONTAINER_PATH/$dep & done + wait do_one_run $container } @@ -52,6 +53,7 @@ do_one_run() { local container=$1 local container_name=$(basename $container) + lockfile-create -q --use-pid --retry 100 --lock-name "start_${container_name}".lock mkdir $tmpdir/start_$container_name 2>/dev/null && ( @@ -64,14 +66,31 @@ do_one_run() [ -x start ] && ./start [ "$container_name" == 'lkp-initrd' ] && ./run ) + lockfile-remove --lock-name "start_${container_name}".lock +} + +build_all() +{ + for dir in $CONTAINER_PATH/*/ + do + build_depends $dir & + done + wait } +start_all() +{ + for dir in $CONTAINER_PATH/*/ + do + start_depends $dir & + done + wait +} + + tmpdir=$(mktemp -d) -for dir in $CONTAINER_PATH/*/ -do - [ "$action" != "start-only" ] && build_depends $dir - start_depends $dir -done +[ "$action" == "start-only" ] || build_all +start_all rm -fr $tmpdir -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: build and start all in parallel to speedup
by Liu Yinsi 29 Mar '21

29 Mar '21

before: function build_depends() are parallel, but do_one_build(), start_depends(), do_one_run() are not parallel. after: function build_depends(), do_one_build(), start_depends(), do_one_run() all are parallel. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 3cd5719..f6b9b65 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -24,7 +24,7 @@ do_one_build() { local container=$1 local container_name=$(basename $container) - lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock + lockfile-create -q --use-pid --retry 100 --lock-name "build_${container_name}".lock mkdir $tmpdir/$container_name 2>/dev/null && ( @@ -33,7 +33,7 @@ do_one_build() [ "$?" == 0 ] && log_info "finish build $container" [ -x install ] && ./install ) - lockfile-remove --lock-name "$container_name".lock + lockfile-remove --lock-name "build_${container_name}".lock } start_depends() @@ -42,8 +42,9 @@ start_depends() for dep in $(cat $container/start-depends 2> /dev/null) do - start_depends $CONTAINER_PATH/$dep + start_depends $CONTAINER_PATH/$dep & done + wait do_one_run $container } @@ -52,6 +53,7 @@ do_one_run() { local container=$1 local container_name=$(basename $container) + lockfile-create -q --use-pid --retry 100 --lock-name "start_${container_name}".lock mkdir $tmpdir/start_$container_name 2>/dev/null && ( @@ -64,14 +66,31 @@ do_one_run() [ -x start ] && ./start [ "$container_name" == 'lkp-initrd' ] && ./run ) + lockfile-remove --lock-name "start_${container_name}".lock +} + +build_all() +{ + for dir in $CONTAINER_PATH/*/ + do + build_depends $dir & + done + wait } +start_all() +{ + for dir in $CONTAINER_PATH/*/ + do + start_depends $dir & + done + wait +} + + tmpdir=$(mktemp -d) -for dir in $CONTAINER_PATH/*/ -do - [ "$action" != "start-only" ] && build_depends $dir - start_depends $dir -done +[ "$action" != "start-only" ] && build_all +start_all rm -fr $tmpdir -- 2.23.0

3 3

[PATCH compass-ci] remove gitchche env
by Liu Yinsi 29 Mar '21

29 Mar '21

refer patch: container/gitcache: delete gitcache docker that is no longer in use Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/crystal-shards/build-depends | 1 - sparrow/3-code/dev-env | 4 ---- 2 files changed, 5 deletions(-) diff --git a/container/crystal-shards/build-depends b/container/crystal-shards/build-depends index 6eed7ec..6627b6f 100755 --- a/container/crystal-shards/build-depends +++ b/container/crystal-shards/build-depends @@ -1,2 +1 @@ crystal-base -gitcache diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f96972e..c42fba5 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -78,8 +78,6 @@ SCHED_PORT: $sched_port LKP_SERVER: $server_ip GIT_SERVER: $GIT_SERVER OS_HTTP_HOST: $OS_HTTP_HOST -GITCACHE_HOST: $gitcache_host -GITCACHE_PORT: $gitcache_port SRV_HTTP_RESULT_HOST: $SRV_HTTP_RESULT_HOST SRV_HTTP_OS_HOST: $SRV_HTTP_OS_HOST SRV_HTTP_GIT_HOST: $SRV_HTTP_GIT_HOST @@ -125,8 +123,6 @@ mkdir -p /etc/compass-ci/scheduler cat > /etc/compass-ci/scheduler/local-testbox-env.yaml <<-EOF LKP_SERVER: GIT_SERVER: -GITCACHE_HOST: -GITCACHE_PORT: SRV_HTTP_RESULT_HOST: SRV_HTTP_RESULT_PORT: SRV_HTTP_OS_HOST: -- 2.23.0

1 0

[PATCH v1 compass-ci] container: fix library libgc.so.1: No such file or directory
by Lu Weitao 29 Mar '21

29 Mar '21

use the compile env as the runtime env in extract-stats and taskqueue container Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/extract-stats/Dockerfile | 2 +- container/taskqueue/Dockerfile | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/extract-stats/Dockerfile b/container/extract-stats/Dockerfile index 3a8c8f3..08abc8c 100644 --- a/container/extract-stats/Dockerfile +++ b/container/extract-stats/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:scheduler-dev RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ diff --git a/container/taskqueue/Dockerfile b/container/taskqueue/Dockerfile index 28ac925..2d5b10a 100644 --- a/container/taskqueue/Dockerfile +++ b/container/taskqueue/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:scheduler-dev RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/mail-post/email_limit_queue: use ENV instead of static value
by Luan Shengde 29 Mar '21

29 Mar '21

use ENV variable instead of static value. enable user-defined new value without edit the code. just restart the service. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_limit_queue.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/mail-post/email_limit_queue.rb b/lib/mail-post/email_limit_queue.rb index c7e3954..62e9634 100755 --- a/lib/mail-post/email_limit_queue.rb +++ b/lib/mail-post/email_limit_queue.rb @@ -54,9 +54,9 @@ class EmailRateLimit end def change_queue(email) - return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + return unless (@redis.hget 'email_in_limit', email).to_i >= ENV['EMAIL_LIMIT_COUNT'].to_i @redis.hdel 'email_in_limit', email - @redis.hset 'email_out_limit', email, 10 + @redis.hset 'email_out_limit', email, ENV['EMAIL_LIMIT_COUNT'] end end -- 2.23.0

1 0

[PATCH compass-ci] container/gitcache: delete gitcache docker that is no longer in use
by Cui Yili 28 Mar '21

28 Mar '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/gitcache/Dockerfile | 15 --------------- container/gitcache/build | 10 ---------- container/gitcache/start | 20 -------------------- 3 files changed, 45 deletions(-) delete mode 100644 container/gitcache/Dockerfile delete mode 100755 container/gitcache/build delete mode 100755 container/gitcache/start diff --git a/container/gitcache/Dockerfile b/container/gitcache/Dockerfile deleted file mode 100644 index d92a141..0000000 --- a/container/gitcache/Dockerfile +++ /dev/null @@ -1,15 +0,0 @@ -# Origin: https://github.com/git-cloner/gitcache -# Copyright (C) 2019-2020 Eric D.little51 -# SPDX-License-Identifier: GPL-3.0 - -FROM golang:alpine - -ENV GO111MODULE on -ENV GOPROXY https://goproxy.cn - - -RUN apk add --no-cache git && \ - git clone https://github.com/git-cloner/gitcache /gitcache && \ - cd /gitcache && go build - -ENTRYPOINT ["/gitcache/gitcache"] diff --git a/container/gitcache/build b/container/gitcache/build deleted file mode 100755 index d7f3523..0000000 --- a/container/gitcache/build +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/bash -# Origin: https://github.com/git-cloner/gitcache -# Copyright (C) 2019-2020 Eric D.little51 -# SPDX-License-Identifier: GPL-3.0 - -. ../defconfig.sh - -docker_skip_rebuild "gitcache" - -docker build -t gitcache . diff --git a/container/gitcache/start b/container/gitcache/start deleted file mode 100755 index 0e1de1b..0000000 --- a/container/gitcache/start +++ /dev/null @@ -1,20 +0,0 @@ -#!/bin/bash -# Origin: https://github.com/git-cloner/gitcache -# Copyright (C) 2019-2020 Eric D.little51 -# SPDX-License-Identifier: GPL-3.0 - -. $CCI_SRC/container/defconfig.sh - -docker_rm gitcache - -cmd=( - docker run -dt - --name gitcache - --restart=always - -p 5000:5000 - -v /srv/cache/gitcache/:/var/gitcache/ - -v /etc/localtime:/etc/localtime:ro - gitcache -) - -"${cmd[@]}" -- 2.23.0

2 1

[PATCH compass-ci] container: add etcd container
by Cao Xueliang 28 Mar '21

28 Mar '21

add etcd container Dockerfile, build and start script. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/etcd/Dockerfile | 16 ++++++++++++++++ container/etcd/build | 12 ++++++++++++ container/etcd/start | 28 ++++++++++++++++++++++++++++ 3 files changed, 56 insertions(+) create mode 100644 container/etcd/Dockerfile create mode 100755 container/etcd/build create mode 100755 container/etcd/start diff --git a/container/etcd/Dockerfile b/container/etcd/Dockerfile new file mode 100644 index 0000000..3654fc4 --- /dev/null +++ b/container/etcd/Dockerfile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.13 + +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> + +ARG VERSION +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && apk add ca-certificates && \ + wget https://github.com/etcd-io/etcd/releases/download/${VERSION}/etcd-${VERSION… && \ + tar zxvf etcd-${VERSION}-linux-arm64.tar.gz && \ + mv etcd-${VERSION}-linux-arm64/etcd* /bin/ && \ + rm -Rf etcd-${VERSION}-linux-arm64* /var/cache/apk/* + +VOLUME /data diff --git a/container/etcd/build b/container/etcd/build new file mode 100755 index 0000000..6ae25dd --- /dev/null +++ b/container/etcd/build @@ -0,0 +1,12 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ../defconfig.sh + +version=v3.4.15 +docker_skip_rebuild "etcd-${version}:alpine" + +docker build -t etcd-${version}:alpine --build-arg VERSION=${version} . + + diff --git a/container/etcd/start b/container/etcd/start new file mode 100755 index 0000000..7657b86 --- /dev/null +++ b/container/etcd/start @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +docker_rm etcd-server + +cmd=( + docker run + -it + -p 2379:2379 + -p 2380:2380 + -p 4001:4001 + -p 7001:7001 + -v /srv/etcd/dir:/data + -e ETCD_UNSUPPORTED_ARCH=arm64 + --name etcd-server + etcd-v3.4.15:alpine + /bin/etcd + --name node-1 + --data-dir /data + --initial-cluster-state new + --listen-client-urls http://0.0.0.0:2379 + --advertise-client-urls http://0.0.0.0:2379 +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] 0-package/os/openEuler: update lockfile rpm download URL
by Liu Yinsi 27 Mar '21

27 Mar '21

The old URL is invalid. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/openEuler | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 61a37c4..9cb7b04 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -28,8 +28,8 @@ pkgs=( yum install -y --skip-broken "${pkgs[@]}" -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… +wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… +wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm rm -rf *.rpm -- 2.23.0

1 0

[PATCH compass-ci] container/web-backend: fix SyntaxError)
by Lu Weitao 27 Mar '21

27 Mar '21

[Why] Traceback (most recent call last): 2: from ./web-backend:9:in `<main>' 1: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require' /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require': /home/lwt/compass-ci/src/lib/web_backend.rb:681: syntax error, unexpected end-of-input, expecting keyword_end (SyntaxError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 1 + src/lib/web_backend.rb | 1 + 2 files changed, 2 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index c7d1ec6..1172160 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -170,6 +170,7 @@ end # } get '/get_jobs_summary' do group_jobs_stats(params) +end # GET /get_job_error?suite=virttest&tbox_group=vm-2p8g # Response diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 7f26b9c..932d362 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -614,6 +614,7 @@ def group_jobs_stats(params) return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end # ------------------------------------------------------------------------------------------- # job error table like: -- 2.23.0

1 0

[PATCH v2 compass-ci] netdata: modify the service port
by Yu Chuan 27 Mar '21

27 Mar '21

[Why] According our latest port policy, modify the service port: - netdata: 20012 => 20013 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/netdata/start | 4 +++- sparrow/3-code/dev-env | 1 + 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/container/netdata/start b/container/netdata/start index 18d225b8fcae..93eabb45dba7 100755 --- a/container/netdata/start +++ b/container/netdata/start @@ -4,13 +4,15 @@ . $CCI_SRC/container/defconfig.sh +load_cci_defaults + docker_rm netdata cmd=( docker run -d --restart=always --name netdata - -p 20012:19999 + -p ${NETDATA_PORT:=20013}:19999 -v /proc:/host/proc:ro -v /sys:/host/sys:ro -v /etc/localtime:/etc/localtime:ro diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f96972eb102d..209c6663238f 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -50,6 +50,7 @@ server_name=$(hostname | cut -f1 -d.) : ${REDIS_PORT:=6379} : ${MAILDIR:=/srv/cci/Maildir/.compass-ci} : ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} +: ${NETDATA_PORT:=20013} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF -- 2.23.0

1 0