March 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 2/2] container/es/start: call check_es_ready()
by Liu Yinsi 01 Mar '21

01 Mar '21

call container/defconfig.sh:check_service_ready() to accelerate es/logging-es startup time. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/start | 7 +++---- container/logging-es/start | 5 ++--- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/container/es/start b/container/es/start index d84ad33..67d6531 100755 --- a/container/es/start +++ b/container/es/start @@ -20,7 +20,6 @@ cmd=( "${cmd[@]}" -echo "Please wait 60s for setting index" -sleep 60 - -set_es_indices +# set index +check_service_ready 9200 +find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; diff --git a/container/logging-es/start b/container/logging-es/start index 66464e6..efd89de 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -20,9 +20,8 @@ cmd=( "${cmd[@]}" -echo "Please wait 60s for setting index" -sleep 60 - +# set index +check_service_ready 9202 curl -sSH "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' { "index": { -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/defconfig.sh: check service ready
by Liu Yinsi 01 Mar '21

01 Mar '21

[why] check service ready or not every 2s, if ok, continue next step, rather than sleep 60s, it can save time. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.sh | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/container/defconfig.sh b/container/defconfig.sh index c33a790..50446d8 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -28,9 +28,15 @@ docker_rm() docker rm -f $container } -set_es_indices() +check_service_ready() { - find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; + local port=$1 + local i + for i in {1..30} + do + curl -s localhost:$port > /dev/null && return + sleep 2 + done } push_image() -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: fix that unexpected file would cause a load error
by Li Yuanchao 01 Mar '21

01 Mar '21

Sometimes there would be some not yaml like file in upstream-repos, such as README.md or user's temporary file. Now we'll do a judge after load and before any other actions, if it's not repo file, just return. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 76cdd1b..22c4e07 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -223,7 +223,10 @@ end class MirrorMain def load_repo_file(repodir, project, fork_name) git_repo = "#{project}/#{fork_name}" - @git_info[git_repo] = YAML.safe_load(File.open(repodir)) + git_info = YAML.safe_load(File.open(repodir)) + return if git_info.nil? || git_info['url'].nil? + + @git_info[git_repo] = git_info @git_info[git_repo]['git_repo'] = git_repo @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo]) fork_stat_init(git_repo) -- 2.23.0

2 2

[PATCH v2 compass-ci 1/2] sparrow/4-docker/buildall: check whether skip ssh-r
by Liu Yinsi 01 Mar '21

01 Mar '21

[why] for user locally deploy compass-ci, ssh-r is not exists, so check when ssh-r already exists, skip ssh-r. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index bb77a1f..f53bf4c 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,7 +56,7 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && exit + [ -n "$(docker ps -aqf name='ssh_r')" ] && exit [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] sparrow/4-docker/buildall: add $action value "start-only"
by Liu Yinsi 01 Mar '21

01 Mar '21

[usage] ./buildall start-only [how] before: ./buildall run-only: call build_depends(), check images exist or not, then call start_depends() after: ./buildall start-only: only call start_depends() [why] for some users locally deploy compass-ci, they will not build images, and some users can't install lockfile-create command, only run containers, no need to check images exist or not, so only call start_depends(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index f53bf4c..a933feb 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -67,7 +67,7 @@ tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir + [ "$action" != "start-only" ] && build_depends $dir start_depends $dir done -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/defconfig.sh: accelerate es/logging-es startup time
by Liu Yinsi 01 Mar '21

01 Mar '21

before: es/logging-es startup time is 60s after: es/logging-es startup time is 10s [how] check es ready or not every 2s, if ok, just set index, rather than sleep 60s. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.sh | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/container/defconfig.sh b/container/defconfig.sh index c33a790..ee66ea5 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -28,9 +28,18 @@ docker_rm() docker rm -f $container } -set_es_indices() +check_es_ready() { - find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; + port=$1 + for num in {1..30} + do + curl -s localhost:$port > /dev/null + if [ $? == "0" ]; then + break + else + sleep 2 + fi + done } push_image() -- 2.23.0

4 7

[PATCH compass-ci 2/2] sparrow/4-docker/buildall: add $action value "start-only"
by Liu Yinsi 01 Mar '21

01 Mar '21

[usage] ./buildall start-only [how] before: ./buildall run-only: call build_depends(), check images exist or not, then call start_depends() after: ./buildall start-only: only call start_depends() [why] for some users locally deploy compass-ci, they will not to build images, and some users can't install lockfile-create command, only run containers, no need to check images exist or not, so only call start_depends(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index c9edbe9..6619600 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -68,7 +68,7 @@ tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir + [ "$action" != "start-only" ] && build_depends $dir start_depends $dir done -- 2.23.0

2 2

[PATCH compass-ci] src/lib/job.cr: delete os_mount in common_dir for upload rpm
by Li Ping 01 Mar '21

01 Mar '21

[why] rpm has no much to do with os_mount. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 33456ca..91fab1d 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -279,7 +279,8 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" + rpm_dir = common_dir.split("/")[1..-1].join("/") + package_dir = ",/initrd/rpmbuild-pkg/#{rpm_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: split up service
by Lu Weitao 01 Mar '21

01 Mar '21

[Why] This service ties up different dirs to the same service. However we'll need the flexibility to hold the dirs in different file servers. [How] one service per dir, and access-like: result: http://$host:11300/ pub: http://$host:11302/ initrd: http://$host:11303/ os: http://$host:11304/ git: http://$host:11305/ cci: http://$host:11306/ Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/start | 27 ++++++--------------------- container/srv-http/start-cci | 19 +++++++++++++++++++ container/srv-http/start-git | 19 +++++++++++++++++++ container/srv-http/start-initrd | 20 ++++++++++++++++++++ container/srv-http/start-os | 19 +++++++++++++++++++ container/srv-http/start-pub | 19 +++++++++++++++++++ container/srv-http/start-result | 19 +++++++++++++++++++ 7 files changed, 121 insertions(+), 21 deletions(-) create mode 100755 container/srv-http/start-cci create mode 100755 container/srv-http/start-git create mode 100755 container/srv-http/start-initrd create mode 100755 container/srv-http/start-os create mode 100755 container/srv-http/start-pub create mode 100755 container/srv-http/start-result diff --git a/container/srv-http/start b/container/srv-http/start index fa56fca..f262d81 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -2,24 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -. $CCI_SRC/container/defconfig.sh -docker_rm srv-http - -cmd=( - docker run - --restart=always - --name srv-http - -p 11300:11300 - -v /srv/pub:/srv/pub:ro - -v /etc/localtime:/etc/localtime:ro - -v /srv/result:/srv/result:ro - -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro - -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro - -v /srv/os:/srv/os:ro - -v /srv/git/archlinux:/srv/git/archlinux:ro - -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro - -d - srv-http -) - -"${cmd[@]}" +./start-pub +./start-result +./start-initrd +./start-os +./start-git +./start-cci diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci new file mode 100755 index 0000000..78ab149 --- /dev/null +++ b/container/srv-http/start-cci @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-cci + +cmd=( + docker run + --restart=always + --name srv-http-cci + -p 11306:11300 + -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-git b/container/srv-http/start-git new file mode 100755 index 0000000..f2292bf --- /dev/null +++ b/container/srv-http/start-git @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-git + +cmd=( + docker run + --restart=always + --name srv-http-git + -p 11305:11300 + -v /srv/git/archlinux:/srv/git/archlinux:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd new file mode 100755 index 0000000..7c73ea1 --- /dev/null +++ b/container/srv-http/start-initrd @@ -0,0 +1,20 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-initrd + +cmd=( + docker run + --restart=always + --name srv-http-initrd + -p 11303:11300 + -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-os b/container/srv-http/start-os new file mode 100755 index 0000000..bb344cf --- /dev/null +++ b/container/srv-http/start-os @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-os + +cmd=( + docker run + --restart=always + --name srv-http-os + -p 11304:11300 + -v /srv/os:/srv/os:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-pub b/container/srv-http/start-pub new file mode 100755 index 0000000..dc1864c --- /dev/null +++ b/container/srv-http/start-pub @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-pub + +cmd=( + docker run + --restart=always + --name srv-http-pub + -p 11302:11300 + -v /srv/pub:/srv/pub:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-result b/container/srv-http/start-result new file mode 100755 index 0000000..f9ecc97 --- /dev/null +++ b/container/srv-http/start-result @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-result + +cmd=( + docker run + --restart=always + --name srv-http-result + -p 11300:11300 + -v /srv/result:/srv/result:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] lib/es_jobs: remove unnecessary null in raw.stats
by Wang Chenglong 01 Mar '21

01 Mar '21

before: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: [null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,"{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found\"}", null,null,null,null,null,null,null,null,null,null,null] after: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: ["{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped ifno files are found\"}"] Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/es_jobs.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index 2d2f3bf..c5c0f0d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -104,7 +104,7 @@ class ESJobs def set_default_value(result, stats, metrics) left_metrics = metrics - stats.keys - left_metrics.each { |metric| result['raw.stats'][metric] << nil } + left_metrics.each { |metric| result['raw.stats'][metric] } stats.each do |key, value| result['raw.stats'][key] << value -- 2.23.0

1 0