- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/2] sparrow/4-docker/buildall: add $action value "start-only"
by Liu Yinsi 01 Mar '21

01 Mar '21

[usage] ./buildall start-only [how] before: ./buildall run-only: call build_depends(), check images exist or not, then call start_depends() after: ./buildall start-only: only call start_depends() [why] for some users locally deploy compass-ci, they will not to build images, and some users can't install lockfile-create command, only run containers, no need to check images exist or not, so only call start_depends(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index c9edbe9..6619600 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -68,7 +68,7 @@ tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir + [ "$action" != "start-only" ] && build_depends $dir start_depends $dir done -- 2.23.0

2 2

[PATCH compass-ci] src/lib/job.cr: delete os_mount in common_dir for upload rpm
by Li Ping 01 Mar '21

01 Mar '21

[why] rpm has no much to do with os_mount. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 33456ca..91fab1d 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -279,7 +279,8 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" + rpm_dir = common_dir.split("/")[1..-1].join("/") + package_dir = ",/initrd/rpmbuild-pkg/#{rpm_dir}/#{@hash["rpm_repo"].to_s.split("/")[-1]}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: split up service
by Lu Weitao 01 Mar '21

01 Mar '21

[Why] This service ties up different dirs to the same service. However we'll need the flexibility to hold the dirs in different file servers. [How] one service per dir, and access-like: result: http://$host:11300/ pub: http://$host:11302/ initrd: http://$host:11303/ os: http://$host:11304/ git: http://$host:11305/ cci: http://$host:11306/ Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/start | 27 ++++++--------------------- container/srv-http/start-cci | 19 +++++++++++++++++++ container/srv-http/start-git | 19 +++++++++++++++++++ container/srv-http/start-initrd | 20 ++++++++++++++++++++ container/srv-http/start-os | 19 +++++++++++++++++++ container/srv-http/start-pub | 19 +++++++++++++++++++ container/srv-http/start-result | 19 +++++++++++++++++++ 7 files changed, 121 insertions(+), 21 deletions(-) create mode 100755 container/srv-http/start-cci create mode 100755 container/srv-http/start-git create mode 100755 container/srv-http/start-initrd create mode 100755 container/srv-http/start-os create mode 100755 container/srv-http/start-pub create mode 100755 container/srv-http/start-result diff --git a/container/srv-http/start b/container/srv-http/start index fa56fca..f262d81 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -2,24 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -. $CCI_SRC/container/defconfig.sh -docker_rm srv-http - -cmd=( - docker run - --restart=always - --name srv-http - -p 11300:11300 - -v /srv/pub:/srv/pub:ro - -v /etc/localtime:/etc/localtime:ro - -v /srv/result:/srv/result:ro - -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro - -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro - -v /srv/os:/srv/os:ro - -v /srv/git/archlinux:/srv/git/archlinux:ro - -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro - -d - srv-http -) - -"${cmd[@]}" +./start-pub +./start-result +./start-initrd +./start-os +./start-git +./start-cci diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci new file mode 100755 index 0000000..78ab149 --- /dev/null +++ b/container/srv-http/start-cci @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-cci + +cmd=( + docker run + --restart=always + --name srv-http-cci + -p 11306:11300 + -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-git b/container/srv-http/start-git new file mode 100755 index 0000000..f2292bf --- /dev/null +++ b/container/srv-http/start-git @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-git + +cmd=( + docker run + --restart=always + --name srv-http-git + -p 11305:11300 + -v /srv/git/archlinux:/srv/git/archlinux:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd new file mode 100755 index 0000000..7c73ea1 --- /dev/null +++ b/container/srv-http/start-initrd @@ -0,0 +1,20 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-initrd + +cmd=( + docker run + --restart=always + --name srv-http-initrd + -p 11303:11300 + -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-os b/container/srv-http/start-os new file mode 100755 index 0000000..bb344cf --- /dev/null +++ b/container/srv-http/start-os @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-os + +cmd=( + docker run + --restart=always + --name srv-http-os + -p 11304:11300 + -v /srv/os:/srv/os:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-pub b/container/srv-http/start-pub new file mode 100755 index 0000000..dc1864c --- /dev/null +++ b/container/srv-http/start-pub @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-pub + +cmd=( + docker run + --restart=always + --name srv-http-pub + -p 11302:11300 + -v /srv/pub:/srv/pub:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-result b/container/srv-http/start-result new file mode 100755 index 0000000..f9ecc97 --- /dev/null +++ b/container/srv-http/start-result @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-result + +cmd=( + docker run + --restart=always + --name srv-http-result + -p 11300:11300 + -v /srv/result:/srv/result:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] lib/es_jobs: remove unnecessary null in raw.stats
by Wang Chenglong 01 Mar '21

01 Mar '21

before: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: [null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,"{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found\"}", null,null,null,null,null,null,null,null,null,null,null] after: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: ["{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped ifno files are found\"}"] Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/es_jobs.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index 2d2f3bf..c5c0f0d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -104,7 +104,7 @@ class ESJobs def set_default_value(result, stats, metrics) left_metrics = metrics - stats.keys - left_metrics.each { |metric| result['raw.stats'][metric] << nil } + left_metrics.each { |metric| result['raw.stats'][metric] } stats.each do |key, value| result['raw.stats'][key] << value -- 2.23.0

1 0

[PATCH v5 compass-ci] doc/stats-rule.md
by Wang Chenglong 01 Mar '21

01 Mar '21

[Why] The rules to follow when writing the stats script. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/stats-rule.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 doc/stats-rule.md diff --git a/doc/stats-rule.md b/doc/stats-rule.md new file mode 100644 index 0000000..9e7af07 --- /dev/null +++ b/doc/stats-rule.md @@ -0,0 +1,46 @@ +${LKP_SERVER}/stats/${script} stat�� +========================================== + +## ��es�� + +1.��error_id�� + - original: {"changed": false, "failures": ["No package yum-plugin-priorities available."], "msg": "Failed to + install some of the specified packages", "rc": 1, "results": []} + + - error_id: error.No-package-yum-plugin-priorities-available: 1 + +2.�� + - ��error_id�� + # �� + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + end + + - ��jobs��'��'��es��error_id.��stats/${script}�� + +3.��error message�� + # original error message + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + end + +4.��error_id�� + - ��'-'�� + - ��'-',��. + - ��'#'. + - �� : , ! ��. + + def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/-+/, '-') # ��'-', �� + line.gsub!(/\b-[a-z][0-9]\.[0-9]\.[0-9]+/, "#") # dgraph-linux-amd64-v0.8.2.tar.gz + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line + end -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: delete the field submit_token
by Li Ping 01 Mar '21

01 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..33456ca 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -403,6 +403,7 @@ class Job @hash.delete("my_uuid") @hash.delete("my_token") + @hash.delete("submit_token") end private def is_valid_account?(account_info) -- 2.23.0

1 0

[PATCH lkp-tests] jobs/iso2rootfs.yaml: change some field to level-1 parameter.
by Li Ping 01 Mar '21

01 Mar '21

[why] the related fields when submit test.yaml to be level-1 parameter is easy for scheduler to delete, like submit_name: submit_email: submit_token: and the required fields when submit test.yaml not have much to do with the iso2rootfs. Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/iso2rootfs.yaml | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 55c4345fa..22651d25e 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -51,23 +51,23 @@ iso2rootfs: # http://1.1.1.1//dailybuilds/1970-01-01-00-00-00/openEuler-20.03-LTS-aarch64… dailybuild_iso_url_file: - ################# - # submit test yaml related fields - ################# +################# +# submit test yaml related fields +################# - # three required fields when submit test yaml - submit_name: - submit_email: - submit_token: +# three required fields when submit test yaml +submit_name: +submit_email: +submit_token: - # submit target tests yaml - ## 1. You can add as many jobs as you like. - ## 2. The following three fields is required for every test job. - test1_yaml: - test1_os_mount: - test1_testbox: - ## the following three fields is required when your job.yaml and - ## script for this test are from the internet. - test1_git_url: - test1_git_yaml: - test1_git_script: +# submit target tests yaml +## 1. You can add as many jobs as you like. +## 2. The following three fields is required for every test job. +test1_yaml: +test1_os_mount: +test1_testbox: +## the following three fields is required when your job.yaml and +## script for this test are from the internet. +test1_git_url: +test1_git_yaml: +test1_git_script: -- 2.23.0

1 0

[PATCH compass-ci] container: add get_job_stats function in assist-result
by Cao Xueliang 01 Mar '21

01 Mar '21

We can expose assist-result host/port to the SUT, then the SUT can get the job stats. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 11 +++++++++++ container/assist-result/views/get_job_stats.rb | 10 ++++++++++ 2 files changed, 21 insertions(+) create mode 100644 container/assist-result/views/get_job_stats.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb index 077a427..02956aa 100755 --- a/container/assist-result/routes.rb +++ b/container/assist-result/routes.rb @@ -7,6 +7,7 @@ require 'json' require 'sinatra' require_relative './views/get_job_yaml' +require_relative './views/get_job_stats' require_relative './views/check_job_credible' configure do @@ -34,3 +35,13 @@ post '/check_job_credible' do return [200, {'credible' => result}.to_json] end + +get '/get_job_stats/:job_id' do + begin + result = get_job_stats(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_stats.rb b/container/assist-result/views/get_job_stats.rb new file mode 100644 index 0000000..d294df3 --- /dev/null +++ b/container/assist-result/views/get_job_stats.rb @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_stats(job_id) + content = ESQuery.new.query_by_id(job_id) + return content['stats'] +end -- 2.23.0

2 2

[PATCH v4 compass-ci] doc/stats-rule.md
by Wang Chenglong 01 Mar '21

01 Mar '21

[Why] The rules to follow when writing the stats script. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/stats-rule.md | 48 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 doc/stats-rule.md diff --git a/doc/stats-rule.md b/doc/stats-rule.md new file mode 100644 index 0000000..ccddbf0 --- /dev/null +++ b/doc/stats-rule.md @@ -0,0 +1,48 @@ +${LKP_SERVER}/stats/${script} stat�� +========================================== + +## ��es�� + +1.��error_id�� + - original: {"changed": false, "failures": ["No package yum-plugin-priorities available."], "msg": "Failed to + install some of the specified packages", "rc": 1, "results": []} + + - error_id: error.No-package-yum-plugin-priorities-available: 1 + +2.�� + - ��error_id�� + # �� + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + end + + - ��jobs��'��'��es��error_id.��stats/${script}�� + +3.��error message�� + # original error message + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + end + +4.��error_id�� + - ��'-'�� + - ��'-',��. + - ��'#'. + - ��: , ! {%%} ��. + + def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/-+/, '-') # ��'-', �� + line.gsub!(/\b-[a-z][0-9]\.[0-9]\.[0-9]+/, "#") # dgraph-linux-amd64-v0.8.2.tar.gz + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/{%/, ' ') + line.gsub!(/%}/, ' ') + line + end -- 2.23.0

2 2

[PATCH compass-ci 1/2] sparrow/4-docker/buildall: check whether skip ssh-r
by Liu Yinsi 01 Mar '21

01 Mar '21

[why] for user locally deploy compass-ci, ssh-r is not exists, so check when ssh-r already exists, skip ssh-r. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index bb77a1f..c9edbe9 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,7 +56,8 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && exit + docker ps | grep -wq "ssh_r" + [ "$?" == "0" ] && [ "$container_name" == 'ssh-r' ] && exit [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run -- 2.23.0

2 2