January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests] reset machine testbox by job
by Zhang Dewan 20 Jan '21

20 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++ tests/reset-tbox | 54 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..4d43850f --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: 172.168.131.2:5001/reset-tbox +queue: reset-tbox + +reset-tbox: + host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..1ca17bfd --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,54 @@ +#!/bin/bash +# host_nameset + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +FileCheck() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +TboxCheck() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +ResetTbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +ResultCheck() +{ + if [ $? -ne 0 ] ; then + echo "Failed to reboot the testbox: $host_name" + fi + echo "done." +} + +main() +{ + FileCheck + TboxCheck + ResetTbox + ResultCheck +} +main + + + -- 2.23.0

3 4

[PATCH v3 lkp-tests] tests: add ansible_tests script
by Zhang Yale 20 Jan '21

20 Jan '21

[ideology] 1. This script can test hundreds of the ansible roles in upstream git repo. 2. ansible roles are supported for the EL platform which has 11257 results in ansible-galaxy official website. 3. For testing ansible git repo, those cannot be tested in a unified way, I mainly test ansible roles git repo, and these roles repo will be more and more. [why] Command "ansible-playbook site.yml" can run in localhost, and only need to create an entry YAML file in which the value of hosts is localhost, for example "site.yml", created in the directory at the same level as the roles directory, which can invoke roles in it. And it dose not need to create a host file in $ansible_path. ``` site.yml --- - hosts: localhost roles: - roles: ansible-role-nginx ``` ansible-role-nginx which is an role in ~/.ansible/roles, can install from galaxy or git clone from github. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/ansible_test | 92 ++++++++++++++++++++++++++-------------------- 1 file changed, 53 insertions(+), 39 deletions(-) diff --git a/tests/ansible_test b/tests/ansible_test index 09e5ca9e..703f5c55 100755 --- a/tests/ansible_test +++ b/tests/ansible_test @@ -1,59 +1,73 @@ #!/bin/bash -# username -# rolename -# ansible_host -# ansible_user -# ansible_python_interpreter - -[ -n "$username" ] || echo "username is empty" -[ -n "$rolename" ] || echo "rolename is empty" -[ -n "$ansible_host" ] || $ansible_host=localhost -[ -n "$ansible_user" ] || $ansible_user=root -[ -n "$ansible_python_interpreter" ] || $ansible_python_interpreter=/user/bin/python3 - -build_mutual_trust() +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() { - echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa - cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys - ssh -o "StrictHostKeyChecking=no" -o "ExitOnForwardFailure=yes" -o "passwordAuthentication=no" root@$ansible__host 'chmod 600 ~/.ssh/authorized_keys' + ansible_host=localhost + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible } install_ansible() { - pip3 install ansible - ansible localhost -m ping - #ansible all -m authorized_key -a "user=root key='{{ lookup('file', '/root/.ssh/id_rsa.pub') }}' path=/root/.ssh/authorized_keys manage_dir=no" --ask-pass -c paramiko - [ $? -eq 0 ] && echo "Ansible is successfully deployed!" + [[ "$os" == "openeuler" ]] && pip3 install ansible + [[ "$os" == "centos" ]] && yum install -y ansible + [[ "$os" == "debian" ]] && apt-get install -y ansible + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] && { + echo "Ansible is successfully deployed!" + } + } } download_ansible_role() { - ansible-galaxy install $username.$rolename - [ $? -eq 0 ] && echo "Please check whether your rolename is correct!" + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone https://github.com/$ansible_repo.git || exit + # git clone git://$GIT_SERVER/$ansible_repo.git || exit + # ansible-galaxy install $username.$rolename + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + fi } ansible_playbook_run() { - ansible_path=/root/.ansible - [ -d "$ansible_path" ] || mkdir -p /root/.ansible - cat <<-EOF > "$ansible_path"/hosts - $ansible_host ansible_user=$ansible_user ansible_python_interpreter=$ansible_python_interpreter - EOF - [ $? -eq 0 ] && echo "Hostfile has been generated!" - - cat <<-EOF > "$ansible_path"/$rolename.yml + cat <<-EOF > "$ansible_path"/site.yml - hosts: $ansible_host - remote_user: $ansible_user roles: - - role: $username.$rolename + - role: $benchmark EOF - [ $? -eq 0 ] && echo "Playbook has been generated!" + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml - ansible-playbook "$ansible_path"/$rolename.yml - [ $? -eq 0 ] && echo "Your playbook is running successfully!" + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "playbook.run.fail: 1" && exit 1 +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run } -#build_mutual_trust -install_ansible -download_ansible_role -ansible_playbook_run +main -- 2.23.0

1 1

[PATCH compass-ci] sparrow: register default accounts for services in sparrow
by Cao Xueliang 20 Jan '21

20 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 10 ++++++++++ sparrow/5-build/register-accounts | 14 ++++++++++++++ sparrow/install-tiny | 1 + 3 files changed, 25 insertions(+) create mode 100755 sparrow/5-build/register-accounts diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 4bafa80..2ba9662 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -60,6 +60,16 @@ export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +mkdir -p /etc/compass-ci/register +cat > /etc/compass-ci/register/register.yaml <<'EOF' +delimiter: + my_email: delimiter@localhost + my_name: delimiter +auto-submit: + my_email: auto-submit@localhost + my_name: auto-submit +EOF + source /etc/os-release path=$(dirname ${BASH_SOURCE[0]})/os/${ID} diff --git a/sparrow/5-build/register-accounts b/sparrow/5-build/register-accounts new file mode 100755 index 0000000..41126f9 --- /dev/null +++ b/sparrow/5-build/register-accounts @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['CCI_SRC']}/lib/es_client" +require_relative "#{ENV['CCI_SRC']}/lib/build_my_info_client" + +content = YAML.load_file('/etc/compass-ci/register/register.yaml') +content.each do |key, value| + build_my_info = BuildMyInfo.new(value['my_email'], value['my_name'], ENV['lab']) + build_my_info.store_account_info +end diff --git a/sparrow/install-tiny b/sparrow/install-tiny index a291549..eb9b3cb 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -19,4 +19,5 @@ cd $CCI_SRC/sparrow || exit . /etc/profile.d/compass.sh 4-docker/buildall 5-build/ipxe +5-build/register-accounts 7-systemd/systemd-setup -- 2.23.0

2 2

[PATCH lkp-tests] adaptation/debian: add pgbench dependency
by Hu Xuejiao 20 Jan '21

20 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/centos | 1 + distro/adaptation/debian | 1 + 2 files changed, 2 insertions(+) diff --git a/distro/adaptation/centos b/distro/adaptation/centos index 2cfe3877..8fbae666 100644 --- a/distro/adaptation/centos +++ b/distro/adaptation/centos @@ -335,3 +335,4 @@ liboath-dev: liboath-devel lsb-release: redhat-lsb linux-base: openssh-client: openssh-clients +postgresql-server: postgresql diff --git a/distro/adaptation/debian b/distro/adaptation/debian index 90260f1d..15706cf6 100644 --- a/distro/adaptation/debian +++ b/distro/adaptation/debian @@ -21,3 +21,4 @@ libperl5: libperl5.24 perl-modules-5: perl-modules-5.24 libgcc-dev: libgcc-6-dev openjdk: openjdk-8-jdk +postgresql-server-dev-all: postgresql -- 2.23.0

1 0

[PATCH compass-ci 1/2] sparrow: register accounts in sparrow
by Cao Xueliang 20 Jan '21

20 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 10 ++++++++++ sparrow/5-build/register-accounts | 14 ++++++++++++++ 2 files changed, 24 insertions(+) create mode 100755 sparrow/5-build/register-accounts diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 4bafa80..2ba9662 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -60,6 +60,16 @@ export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +mkdir -p /etc/compass-ci/register +cat > /etc/compass-ci/register/register.yaml <<'EOF' +delimiter: + my_email: delimiter@localhost + my_name: delimiter +auto-submit: + my_email: auto-submit@localhost + my_name: auto-submit +EOF + source /etc/os-release path=$(dirname ${BASH_SOURCE[0]})/os/${ID} diff --git a/sparrow/5-build/register-accounts b/sparrow/5-build/register-accounts new file mode 100755 index 0000000..41126f9 --- /dev/null +++ b/sparrow/5-build/register-accounts @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['CCI_SRC']}/lib/es_client" +require_relative "#{ENV['CCI_SRC']}/lib/build_my_info_client" + +content = YAML.load_file('/etc/compass-ci/register/register.yaml') +content.each do |key, value| + build_my_info = BuildMyInfo.new(value['my_email'], value['my_name'], ENV['lab']) + build_my_info.store_account_info +end -- 2.23.0

1 1

[PATCH v2 compass-ci] scheduler: respond an '{}' when 'vt' is non-existent or empty
by Ren Wen 20 Jan '21

20 Jan '21

In job.yaml -> response case1: no 'vt' -> {} case2: vt: -> {} case3: vt: domain: <domain.xml> -> {"domain":"<domain.xml>"} Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 3f446b2..b7f3d75 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -174,6 +174,9 @@ class Sched _kernel_params = job["kernel_params"]? _kernel_params = _kernel_params.as_a.map(&.to_s).join(" ") if _kernel_params + _vt = job["vt"]? + _vt = Hash(String, String).new unless (_vt && _vt != nil) + return { "job_id" => job.id, "kernel_uri" => job.kernel_uri, @@ -181,7 +184,7 @@ class Sched "kernel_params" => _kernel_params, "result_root" => job.result_root, "LKP_SERVER" => job["LKP_SERVER"]?, - "vt" => job["vt"]?, + "vt" => _vt, "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", "SRV_HTTP_HOST" => SRV_HTTP_HOST, "SRV_HTTP_PORT" => SRV_HTTP_PORT, -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] qemu/kvm.sh: optimize the prompt when the error occurs
by Yu Chuan 20 Jan '21

20 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- providers/qemu/kvm.sh | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 35678fbd44ef..f8246e828359 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,6 +6,9 @@ : ${nr_cpu:=1} : ${memory:=1G} +: ${LKP_SRC:="/c/lkp-tests"} + +source ${LKP_SRC}/lib/log.sh check_logfile() { @@ -52,15 +55,23 @@ parse_ipxe_script() check_option_value() { + [ -n "$kernel" ] || { + log_info "Can not find job for current hostname: $hostname." + exit 0 + } + [ -s "$kernel" ] || { - echo "The kernel does not exist: $kernel" - exit + log_error "Can not find kernel file or kernel file is empty: $kernel." + exit 1 } # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit + [ -n "$initrds" ] || { + log_error "The current initrds is null." + exit 1 + } } set_initrd() -- 2.23.0

2 1

[PATCH compass-ci 2/2] delimiter: Change DELIMITER_ACCOUNT to DELIMITER_EMAIL
by Cao Xueliang 20 Jan '21

20 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/constants.rb | 2 +- src/delimiter/utils.rb | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index a2e13fe..95d1acd 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -6,7 +6,7 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' GIT_MIRROR_HOST = ENV['GIT_MIRROR_HOST'] || '172.17.0.1' MONITOR_HOST = ENV['MONITOR_HOST'] || '172.17.0.1' MONITOR_PORT = ENV['MONITOR_PORT'] || '11310' -DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'caoxl78320(a)163.com' +DELIMITER_EMAIL = ENV['DELIMITER_EMAIL'] || 'delimiter@localhost' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 8cf2a84..2c6549c 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -122,17 +122,17 @@ module Utils end def get_account_info - ESQuery.new(index: 'accounts').query_by_id(DELIMITER_ACCONUT) + ESQuery.new(index: 'accounts').query_by_id(DELIMITER_EMAIL) end def init_job_content(job_id) job_yaml = AssistResult.new.get_job_yaml(job_id) - raise "get job yaml failed, job id: #{job_id} !" unless job_yaml + raise "get job yaml failed, job id: #{job_id}" unless job_yaml job = JSON.parse job_yaml account_info = get_account_info - raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info + raise "query delimiter account info failed: #{DELIMITER_EMAIL}" unless account_info record_jobs(job['id'], job['upstream_commit']) -- 2.23.0

1 0

[PATCH v9 compass-ci] lib/es_jobs.rb: optimize es-jobs output
by Lu Kaiyi 20 Jan '21

20 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archlinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archlinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.summary.any_error=1: 24 kvcount.summary.any_fail=1: 25 kvcount.summary.any_stderr=1: 44 kvcount.summary.any_warning=1: 23 kvcount.summary.success=1: 1 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 ... raw.id.[job_state=failed]: ["crystal.122405", "crystal.122406", "crystal.122331", "crystal.122365", "crystal.122404", "crystal.122438", "crystal.122607", "crystal.122370", "crystal.122375", "crystal.122410", "crystal.122900", "crystal.122906", "crystal.122271", "crystal.122451", "crystal.122835", "crystal.122834", "crystal.122232", "crystal.122403", "crystal.122453", "crystal.122886"] ... raw.id.[summary.any_error=1]: ["crystal.122436", "crystal.122476", "crystal.122803", "crystal.122883", "crystal.122910", "crystal.122438", "crystal.122452", "crystal.122895", "crystal.122446", "crystal.122685", "crystal.122687", "crystal.122833", "crystal.122451", "crystal.122690", "crystal.122719", "crystal.122882", "crystal.122890", "crystal.122453", "crystal.122443", "crystal.122435", "crystal.122432", "crystal.122688", "crystal.122689", "crystal.122886"] ... sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0, null, null, null, null, null, null, null, null, 524288.0, 524288.0, null, null, null, null, null, 516608.0, null, 1048576.0, null, null, null, null, null, null, null, 524288.0, null, null, null, null, 2104832.0, 1572864.0, null, null, 2097152.0, null, null, null, null, null, null, null, null, null] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... max.stats.sched_debug.cfs_rq:/.load.stddev: 2104832.0 max.stats.softirqs.CPU1.NET_RX: 6 max.stats.slabinfo.kmalloc-512.active_objs: 1616 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 201 +++++++++++++++++++++++++++---------------------- 1 file changed, 111 insertions(+), 90 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..9cb95a4 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -3,137 +3,158 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo summary.success + summary.any_fail summary.any_error summary.any_stderr summary.any_warning].freeze + +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' # deal jobs search from es class ESJobs def initialize(es_query, my_refine = [], fields = [], stats_filter = []) - @es_query = es_query - @es = ESQuery.new(ES_HOST, ES_PORT) + @jobs = query_jobs_from_es(es_query) @refine = my_refine @fields = fields @stats_filter = stats_filter - @stats_filter_result = {} @refine_jobs = [] - @jobs = {} - @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' - } - set_defaults - deal_jobs + set_jobs_summary end - def set_defaults - query_result = @es.multi_field_query(@es_query) - query_result['hits']['hits'].each do |job| - @jobs[job['_id']] = job['_source'] - end - - @stats = { - 'stats.count' => Hash.new(0), - 'stats.sum' => Hash.new(0), - 'stats.avg' => Hash.new(0) - } - @result = {} - @fields.each do |field| - @result[field] = [] - end + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs end - def add_result_fields(job, level) - return unless @refine.include?(level) || @refine.include?(-1) + def set_job_summary(stats, job) + summary_result = '' + if stats.keys.any? { |stat| stat.match(/warn/i) } + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + if stats.keys.any? { |stat| stat.match(/stderr/i) } + job['summary.any_stderr'] = 1 + summary_result = 'stderr' + end + if stats.keys.any? { |stat| stat.match(/error|nr_fail=0/i) } + job['summary.any_error'] = 1 + summary_result = 'error' + end + if stats.keys.any? { |stat| stat.match(/fail/i) } && + stats.keys.all? { |stat| !stat.match(/nr_fail=0/i) } + job['summary.any_fail'] = 1 + summary_result = 'fail' + end + return unless summary_result.empty? - @refine_jobs << job['id'] - @fields.each do |field| - value = job[field] - if value - value = job['id'] + '.' + value if field == 'job_state' - @result[field] << value - end + job['summary.success'] = 1 + end - next unless job['stats'] + # set jobs summary fields information in place + def set_jobs_summary + @jobs.each do |job| + stats = job['stats'] + next unless stats - @result[field] << job['stats'][field] if job['stats'][field] + set_job_summary(stats, job) end end - def deal_jobs - stats_count = Hash.new(0) - stats_jobs = {} + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats - @jobs.each do |job_id, job| - level = deal_stats(job) - add_result_fields(job, level) - - stat_key = @stats_level[level] - stat_jobs_key = stat_key + '_jobs' - - stats_count[stat_key] += 1 - stats_jobs[stat_jobs_key] ||= [] - stats_jobs[stat_jobs_key] << job_id + metrics.concat(stats.keys) end + metrics.uniq! + end - @stats['stats.count'].merge!(stats_count) - @stats['stats.count'].merge!(stats_jobs) + def initialize_result_hash(metrics) + result = { + 'kvcount' => {}, + 'raw.id' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} + } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result end - def deal_stats(job, level = 0) - return 1 unless job['stats'] + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } - job['stats'].each do |key, value| - match_stats_filter(key, value, job['id']) - calculate_stat(key, value) - level = get_stat_level(key, level) + stats.each do |key, value| + result['raw.stats'][key] << value end - return level end - def match_stats_filter(key, value, job_id) - @stats_filter.each do |filter| - next unless key.include?(filter) + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - key = job_id + '.' + key - @stats_filter_result[key] = value - - break + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + result['raw.id']["[#{keyword}=#{job[keyword]}]"] ||= [] + result['raw.id']["[#{keyword}=#{job[keyword]}]"] << job['id'] end end - def calculate_stat(key, value) - if function_stat?(key) - return unless @fields.include?('stats.sum') - - @stats['stats.sum'][key] += value - else - return unless @fields.include?('stats.avg') - - @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end end end - def get_stat_level(stat, level) - return level if level >= 3 - return 3 if stat.match(/error|fail/i) - return 2 if stat.match(/warn/i) + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end - return 0 + stats_count(result) + result end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def output_yaml(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + if value.is_a? Hash + output_yaml(prefix_key, value) + else + puts "#{prefix_key}: #{value}".gsub('nil', 'null') + end end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def output + @result = query_jobs_state(@jobs) + @result['kvcount'] = @result['kvcount'].sort.to_h + @result['raw.id'] = @result['raw.id'].sort.to_h + output_yaml('', @result) end end -- 2.23.0

1 0

[PATCH v2 compass-ci] answerback-email: optimize
by Luan Shengde 20 Jan '21

20 Jan '21

[why] when define a constant, the constant may already defined in required files, when execute the script, my throw warnings: ./answerback-email.rb:32: warning: already initialized constant SEND_MAIL_HOST ~/compass-ci/lib/constants.rb:11: warning: previous definition of SEND_MAIL_HOST was here ./answerback-email.rb:33: warning: already initialized constant SEND_MAIL_PORT ~/compass-ci/lib/constants.rb:12: warning: previous definition of SEND_MAIL_PORT was here Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index a99306b..8dceee1 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -27,11 +27,11 @@ names = Set.new %w[ defaults = relevant_defaults(names) -JUMPER_HOST = defaults['JUMPER_HOST'] -JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -LAB = defaults['lab'] +JUMPER_HOST ||= defaults['JUMPER_HOST'] +JUMPER_PORT ||= defaults['JUMPER_PORT'] || 29999 +SEND_MAIL_HOST ||= defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT ||= defaults['SEND_MAIL_PORT'] || 49000 +LAB ||= defaults['lab'] TOOL_SERVER = 'z9' my_info = { -- 2.23.0

1 0