- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] jobs/iso2rootfs.yaml: change some field to level-1 parameter.
by Li Ping 01 Mar '21

01 Mar '21

[why] the related fields when submit test.yaml to be level-1 parameter is easy for scheduler to delete, like submit_name: submit_email: submit_token: and the required fields when submit test.yaml not have much to do with the iso2rootfs. Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/iso2rootfs.yaml | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 55c4345fa..22651d25e 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -51,23 +51,23 @@ iso2rootfs: # http://1.1.1.1//dailybuilds/1970-01-01-00-00-00/openEuler-20.03-LTS-aarch64… dailybuild_iso_url_file: - ################# - # submit test yaml related fields - ################# +################# +# submit test yaml related fields +################# - # three required fields when submit test yaml - submit_name: - submit_email: - submit_token: +# three required fields when submit test yaml +submit_name: +submit_email: +submit_token: - # submit target tests yaml - ## 1. You can add as many jobs as you like. - ## 2. The following three fields is required for every test job. - test1_yaml: - test1_os_mount: - test1_testbox: - ## the following three fields is required when your job.yaml and - ## script for this test are from the internet. - test1_git_url: - test1_git_yaml: - test1_git_script: +# submit target tests yaml +## 1. You can add as many jobs as you like. +## 2. The following three fields is required for every test job. +test1_yaml: +test1_os_mount: +test1_testbox: +## the following three fields is required when your job.yaml and +## script for this test are from the internet. +test1_git_url: +test1_git_yaml: +test1_git_script: -- 2.23.0

1 0

[PATCH compass-ci] container: add get_job_stats function in assist-result
by Cao Xueliang 01 Mar '21

01 Mar '21

We can expose assist-result host/port to the SUT, then the SUT can get the job stats. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 11 +++++++++++ container/assist-result/views/get_job_stats.rb | 10 ++++++++++ 2 files changed, 21 insertions(+) create mode 100644 container/assist-result/views/get_job_stats.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb index 077a427..02956aa 100755 --- a/container/assist-result/routes.rb +++ b/container/assist-result/routes.rb @@ -7,6 +7,7 @@ require 'json' require 'sinatra' require_relative './views/get_job_yaml' +require_relative './views/get_job_stats' require_relative './views/check_job_credible' configure do @@ -34,3 +35,13 @@ post '/check_job_credible' do return [200, {'credible' => result}.to_json] end + +get '/get_job_stats/:job_id' do + begin + result = get_job_stats(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_stats.rb b/container/assist-result/views/get_job_stats.rb new file mode 100644 index 0000000..d294df3 --- /dev/null +++ b/container/assist-result/views/get_job_stats.rb @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_stats(job_id) + content = ESQuery.new.query_by_id(job_id) + return content['stats'] +end -- 2.23.0

2 2

[PATCH v4 compass-ci] doc/stats-rule.md
by Wang Chenglong 01 Mar '21

01 Mar '21

[Why] The rules to follow when writing the stats script. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/stats-rule.md | 48 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 doc/stats-rule.md diff --git a/doc/stats-rule.md b/doc/stats-rule.md new file mode 100644 index 0000000..ccddbf0 --- /dev/null +++ b/doc/stats-rule.md @@ -0,0 +1,48 @@ +${LKP_SERVER}/stats/${script} stat�� +========================================== + +## ��es�� + +1.��error_id�� + - original: {"changed": false, "failures": ["No package yum-plugin-priorities available."], "msg": "Failed to + install some of the specified packages", "rc": 1, "results": []} + + - error_id: error.No-package-yum-plugin-priorities-available: 1 + +2.�� + - ��error_id�� + # �� + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + end + + - ��jobs��'��'��es��error_id.��stats/${script}�� + +3.��error message�� + # original error message + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + end + +4.��error_id�� + - ��'-'�� + - ��'-',��. + - ��'#'. + - ��: , ! {%%} ��. + + def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/-+/, '-') # ��'-', �� + line.gsub!(/\b-[a-z][0-9]\.[0-9]\.[0-9]+/, "#") # dgraph-linux-amd64-v0.8.2.tar.gz + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/{%/, ' ') + line.gsub!(/%}/, ' ') + line + end -- 2.23.0

2 2

[PATCH compass-ci 1/2] sparrow/4-docker/buildall: check whether skip ssh-r
by Liu Yinsi 01 Mar '21

01 Mar '21

[why] for user locally deploy compass-ci, ssh-r is not exists, so check when ssh-r already exists, skip ssh-r. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index bb77a1f..c9edbe9 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,7 +56,8 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && exit + docker ps | grep -wq "ssh_r" + [ "$?" == "0" ] && [ "$container_name" == 'ssh-r' ] && exit [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run -- 2.23.0

2 2

[PATCH v3 compass-ci] container/assign-account: optimize config login permission
by Luan Shengde 01 Mar '21

01 Mar '21

optimize account login permission setting: before: set enable login: has key: enable_login key value: true set disable login has key: enable_login key value: false or has no key: enable_login now: set enable login: has key: enable_login key value: true set disable login: has key: enable_login key value: false keep the default login permission: has no key: enable_login Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 7e0a5a7..2922f81 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -136,7 +136,7 @@ class AccountStorage def permit_login_config(login_name) if @data.key?('enable_login') && @data['enable_login'] %x(usermod -s /usr/bin/zsh #{login_name}) - else + elsif @data.key?('enable_login') && !@data['enable_login'] %x(usermod -s /sbin/nologin #{login_name}) end end -- 2.23.0

1 0

[PATCH compass-ci] add check runtime upper limit
by Luan Shengde 01 Mar '21

01 Mar '21

add time limit for borrowing machine. the max runtime is limited to no more than 10 days. case runtime/sleep beyond the limit or both of them was set empty value. it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 52965bc..63ec8fb 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -409,6 +410,29 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time(runtime/sleep) cannot exceed 10 days. Consider re-borrow.\n" + + if @hash["pp"]["sleep"].as_i? + sleep_run_time = @hash["pp"]["sleep"] + elsif @hash["pp"]["sleep"].as_h? + sleep_run_time = @hash["pp"]["sleep"]["runtime"] + else + notice_msg = "\nPlease set runtime/sleep first for the job yaml and retry." + raise notice_msg + end + + raise error_msg if sleep_run_time.as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH compass-ci] container/scheduler: install util-linux to use uuidgen
by Wu Zhende 26 Feb '21

26 Feb '21

submit job.yaml in scheduler docker needs command "uuidgen" to generate submit_id Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/scheduler/Dockerfile b/container/scheduler/Dockerfile index 0ad813c..7fb8e35 100644 --- a/container/scheduler/Dockerfile +++ b/container/scheduler/Dockerfile @@ -8,7 +8,7 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN apk update RUN apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' 'util-linux' RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ -- 2.23.0

1 0

[PATCH compass-ci] lib/es_jobs: remove unnecessary null in raw.stats
by Wang Chenglong 26 Feb '21

26 Feb '21

before: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: [null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,"{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found\"}", null,null,null,null,null,null,null,null,null,null,null] after: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: ["{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped ifno files are found\"}"] Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/es_jobs.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index 2d2f3bf..c5c0f0d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -104,7 +104,7 @@ class ESJobs def set_default_value(result, stats, metrics) left_metrics = metrics - stats.keys - left_metrics.each { |metric| result['raw.stats'][metric] << nil } + left_metrics.each { |metric| result['raw.stats'][metric] } stats.each do |key, value| result['raw.stats'][key] << value -- 2.23.0

1 0

[PATCH compass-ci] job.cr: add has_key? function in Job class
by Cao Xueliang 26 Feb '21

26 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 4 ++++ src/scheduler/find_job_boot.cr | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..52965bc 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -332,6 +332,10 @@ class Job @hash.[key]? end + def has_key?(key : String) + @hash.has_key?(key) + end + def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" raise "Should not use []= update #{key}, use update_#{key}" diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 88db69e..8dd22a8 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,7 +160,7 @@ class Sched end private def get_boot_ipxe(job : Job) - return job["ipxe_response"] if job["suite"] == "install-iso" && job["ipxe_response"]? + return job["ipxe_response"] if job["suite"] == "install-iso" && job.has_key?("ipxe_response") response = "#!ipxe\n\n" -- 2.23.0

1 0

[PATCH v8 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 26 Feb '21

26 Feb '21

add time limit for borrowing machine. the max runtime is limited to no more than 10 days. case runtime/sleep beyond the limit or both of them was set empty value. it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..91d28fa 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -405,6 +406,29 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time(runtime/sleep) cannot exceed 10 days. Consider re-borrow.\n" + + if @hash["pp"]["sleep"].as_i? + sleep_run_time = @hash["pp"]["sleep"] + elsif @hash["pp"]["sleep"].as_h? + sleep_run_time = @hash["pp"]["sleep"]["runtime"] + else + notice_msg = "\nPlease set runtime/sleep first for the job yaml and retry." + raise notice_msg + end + + raise error_msg if sleep_run_time.as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0