- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] lib/yaml.rb: load_yaml_merge add arg
by Wei Jihui 15 Jan '21

15 Jan '21

add template_context for load_yaml_merge, it is used by load_yaml. when template_context is nil, there is not impact for load_yaml_merge. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/yaml.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/yaml.rb b/lib/yaml.rb index 9c9b8088..4290d594 100755 --- a/lib/yaml.rb +++ b/lib/yaml.rb @@ -52,13 +52,13 @@ def load_yaml_with_flock(file, timeout = nil) end end -def load_yaml_merge(files) +def load_yaml_merge(files, template_context = nil) all = {} files.each do |file| next unless File.size? file begin - yaml = load_yaml(file) + yaml = load_yaml(file, template_context) all.update(yaml) rescue StandardError => e warn "#{e.class.name}: #{e.message.split("\n").first}: #{file}" -- 2.23.0

1 0

[PATCH v1 compass-ci] lib/job.cr: fix failed to upload package
by Cao Xueliang 15 Jan '21

15 Jan '21

[why] If the JSON::Any type pkg_style value is nil, we don't set the default upload path. [how] If the JSON::Any type pkg_style value is nil, we set it to empty hash that will get a default upload path. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1d25236..07bfda0 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -256,7 +256,9 @@ class Job pkg_style = @hash[item]? break if pkg_style end - return nil unless pkg_style && pkg_style != nil + return nil unless pkg_style + + pkg_style = JSON.parse("{}") if pkg_style == nil tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] -- 2.23.0

3 3

[PATCH compass-ci 2/2] lib/compare_error_messages.rb: add the credibility check of job
by Lin Jiaxin 15 Jan '21

15 Jan '21

defect: may match the wrong result error_id="build-pkg.pixz.c:warning:'subsuf'defined-but-not-used[-Wunused-function]" filename="pixz.c" correct result: gcc -DHAVE_CONFIG_H -I. -I.. -D_FORTIFY_SOURCE=2 -pthread -Wall -Wno-unknown-pragmas -march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -MT pixz-pixz.o -MD -MP -MF .deps/pixz-pixz.Tpo -c -o pixz-pixz.o `test -f 'pixz.c' || echo './'`pixz.c ^^^^^^ wrong result: mv -f .deps/pixz.cpu.Tpo .deps/pixz.cpu.Po ^^^^^^ Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 0b51fc3..a959c61 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -8,6 +8,39 @@ require_relative 'es_query' require_relative 'error_messages' require_relative "#{ENV['LKP_SRC']}/lib/common" +# previous_job_id's error_ids include error_id +# If later_job_id's output include error filename that extract from error_id, +# the later_job_id is credible. +def credible?(previous_job_id, later_job_id, error_id) + es = ESQuery.new + + previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') + later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') + + return false if filenames_check(previous_result_file, later_result_file, error_id).value?(false) + + return true +end + +def filenames_check(previous_result_file, later_result_file, error_id) + filenames = [] + filenames_check = Hash.new { |h, k| h[k] = false } + + error_lines = ErrorMessages.new(previous_result_file).obtain_error_messages_by_error_id(error_id, true) + error_lines.each do |error_line| + filenames << $1 if error_line =~ /(.*)(:.*){2}: (error|warning):/ + end + + File.open(later_result_file).each_line do |line| + filenames.each do |filename| + filenames_check[filename] + filenames_check[filename] = true if line.include?(filename) + end + end + + return filenames_check +end + def get_compare_result(previous_job_id, later_job_id) es = ESQuery.new -- 2.23.0

2 1

[PATCH lab-z9] add a117 host info
by Bai Jing 15 Jan '21

15 Jan '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- hosts/taishan200-2280-2s64p-256g--a117 | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a117 diff --git a/hosts/taishan200-2280-2s64p-256g--a117 b/hosts/taishan200-2280-2s64p-256g--a117 new file mode 100644 index 0000000..0f10495 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a117 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b57d1 + - /dev/disk/by-id/scsi-350000399886b720d +mac_addr: + - 44:67:47:85:f4:0b + - 44:67:47:85:f4:0c + - 44:67:47:85:f4:0d + - 44:67:47:85:f4:0e + - cc:05:77:ee:b3:ff + - cc:05:77:ee:b4:00 + - cc:05:77:ee:b4:01 + - cc:05:77:ee:b4:02 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.10 -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container - assist-cci: build & start script
by Ren Wen 15 Jan '21

15 Jan '21

Mount '/srv/cci' and upload config to dir: 'libvirt-xml'. Use port 'ASSIST_CCI_PORT' or '11392' by default. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/build | 5 ++++ container/assist-cci/start | 47 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) create mode 100755 container/assist-cci/build create mode 100755 container/assist-cci/start diff --git a/container/assist-cci/build b/container/assist-cci/build new file mode 100755 index 0000000..f82de7f --- /dev/null +++ b/container/assist-cci/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-cci . diff --git a/container/assist-cci/start b/container/assist-cci/start new file mode 100755 index 0000000..b65f814 --- /dev/null +++ b/container/assist-cci/start @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ASSIST_CCI_HOST + ASSIST_CCI_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +ASSIST_CCI_PORT = defaults['ASSIST_CCI_PORT'] || 11392 +SRV_CCI_PATH = '/srv/cci' +LIBVIRT_XML_PATH = "#{SRV_CCI_PATH}/libvirt-xml" +docker_rm 'assist-cci' + +cmd = %w[ + docker run + --name assist-cci + --restart=always + -d +] + env + %W[ + -e LIBVIRT_XML_PATH=#{LIBVIRT_XML_PATH} + -e ASSIST_CCI_PORT=#{ASSIST_CCI_PORT} + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -p #{ASSIST_CCI_PORT}:#{ASSIST_CCI_PORT} + -v /etc/localtime:/etc/localtime:ro + -v #{SRV_CCI_PATH}:#{SRV_CCI_PATH} + -w #{DEFAULT_CCI}/container/assist-cci/ + assist-cci +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

2 2

[PATCH v2 compass-ci 1/3] container - assist-cci: Dockerfile
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/Dockerfile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 container/assist-cci/Dockerfile diff --git a/container/assist-cci/Dockerfile b/container/assist-cci/Dockerfile new file mode 100644 index 0000000..16bba06 --- /dev/null +++ b/container/assist-cci/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER 15991987063(a)163.com + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install activesupport json yaml sinatra puma -- 2.23.0

3 4

[PATCH compass-ci] auto_submit: don't submit job with too old commit
by Li Yuanchao 15 Jan '21

15 Jan '21

If the last commit of a repository is too old, such as 'two months ago', that means the repository is not active. We don't care such repositories. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index ecc4d70..5648d59 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -31,6 +31,9 @@ class AutoSubmit def submit(newrefs_info, submit_argv) newrefs_info['new_refs']['heads'].each do |branch, commit_id| + inactive_time = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git log --pretty=format:"%cr" -1 #{commit_id}) + next if inactive_time =~ /(month|year)/ + real_argvs = Array.new(submit_argv) real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") real_argvs.push("upstream_commit=#{commit_id}") -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: fix "build-pkg" value is nil can't upload package
by Cao Xueliang 15 Jan '21

15 Jan '21

[why] If the build-pkg value is nil, we didn't set the default upload path. [how] If the build-pkg value is nil, we set it to empty hash, that will get a default upload path. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1d25236..07bfda0 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -256,7 +256,9 @@ class Job pkg_style = @hash[item]? break if pkg_style end - return nil unless pkg_style && pkg_style != nil + return nil unless pkg_style + + pkg_style = JSON.parse("{}") if pkg_style == nil tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] -- 2.23.0

3 4

[PATCH compass-ci] delimiter: create a delimiter account when start delimiter container
by Cao Xueliang 15 Jan '21

15 Jan '21

Before: Use compass-ci-robot(a)qq.com as the bisect job subqueue, wait to consume will take a long time. After: User delimiter@localhost as the bisect job subqueue, wait to consume will take a short time. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/Dockerfile | 2 +- container/delimiter/start | 2 ++ src/delimiter.rb | 4 ++++ src/delimiter/constants.rb | 3 ++- src/delimiter/utils.rb | 6 +++--- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/container/delimiter/Dockerfile b/container/delimiter/Dockerfile index 93c52c6..ab7519d 100644 --- a/container/delimiter/Dockerfile +++ b/container/delimiter/Dockerfile @@ -10,4 +10,4 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN umask 002 && \ echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ - gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket terminal-table mail + gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket terminal-table mail io-console diff --git a/container/delimiter/start b/container/delimiter/start index 669923b..2a68df3 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -7,6 +7,8 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ + ES_HOST + ES_PORT TASKQUEUE_HOST TASKQUEUE_PORT ] diff --git a/src/delimiter.rb b/src/delimiter.rb index bd40775..898fa0e 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -2,9 +2,13 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../lib/build_my_info_client' require_relative './delimiter/delimiter' begin + build_my_info = BuildMyInfo.new(DELIMITER_EMAIL, DELIMITER_NAME, ENV['lab']) + build_my_info.store_account_info + delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 7b14558..23b5053 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -6,7 +6,8 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' GIT_MIRROR_HOST = ENV['GIT_MIRROR_HOST'] || '172.17.0.1' MONITOR_HOST = ENV['MONITOR_HOST'] || '172.17.0.1' MONITOR_PORT = ENV['MONITOR_PORT'] || '11310' -DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'caoxl78320(a)163.com' +DELIMITER_EMAIL = ENV['DELIMITER_EMAIL'] || 'delimiter@localhost' +DELIMITER_NAME = ENV['DELIMITER_NAME'] || 'delimiter' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 61fe6ab..aaad394 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -118,17 +118,17 @@ module Utils end def get_account_info - ESQuery.new(index: 'accounts').query_by_id(DELIMITER_ACCONUT) + ESQuery.new(index: 'accounts').query_by_id(DELIMITER_EMAIL) end def init_job_content(job_id) job_yaml = AssistResult.new.get_job_yaml(job_id) - raise "get job yaml failed, job id: #{job_id} !" unless job_yaml + raise "get job yaml failed, job id: #{job_id}" unless job_yaml job = JSON.parse job_yaml account_info = get_account_info - raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info + raise "query delimiter account info failed: #{DELIMITER_EMAIL}" unless account_info record_jobs(job['id'], job['upstream_commit']) -- 2.23.0

2 6

[PATCH v2 compass-ci 3/3] container - assist-cci: upload function
by Ren Wen 15 Jan '21

15 Jan '21

Upload an XML template of libvirt. Response is the modified filename. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/routes.rb | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/assist-cci/routes.rb diff --git a/container/assist-cci/routes.rb b/container/assist-cci/routes.rb new file mode 100755 index 0000000..0e71a75 --- /dev/null +++ b/container/assist-cci/routes.rb @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'digest/md5' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_CCI_PORT'] +end + +LIBVIRT_XML_PATH = ENV['LIBVIRT_XML_PATH'] +VERSION = '0.1' + +get '/' do + "version: #{VERSION}" +end + +post '/upload' do + if params[:file] && (tempfile = params[:file][:tempfile]) && (filename = params[:file][:filename]) + dirname = File.basename(filename, '.xml') + content = tempfile.read + md5_filename = Digest::MD5.hexdigest(content) + '.xml' + + dir_path = "#{LIBVIRT_XML_PATH}/#{dirname}" + Dir.exist?(dir_path) || Dir.mkdir(dir_path) + + target = "#{dir_path}/#{md5_filename}" + File.open(target, 'wb') { |f| f.write content } + + File.join(dirname, md5_filename) + else + '' + end +end -- 2.23.0

1 0

[PATCH compass-ci] scheduler: change 'templates' to 'vt'
by Ren Wen 15 Jan '21

15 Jan '21

Use 'vt' (virt) instead of 'templates', the latter is not clear enough. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 625c74c..152d549 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -181,7 +181,7 @@ class Sched "kernel_params" => _kernel_params, "result_root" => job.result_root, "LKP_SERVER" => job["LKP_SERVER"]?, - "templates" => job["templates"]?, + "vt" => job["vt"]?, "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", }.to_json end -- 2.23.0

1 0

[PATCH v4 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archLinux [output] --- kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archLinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 sum.stats.last_state.test.iperf.exit_code.127: 12 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: "[524288.0, nil, nil, nil, nil, nil, nil, nil, nil, 524288.0, 524288.0, nil, nil, nil, nil, nil, 516608.0, nil, 1048576.0, nil, nil, nil, nil, nil, nil, nil, 524288.0, nil, nil, nil, nil, 2104832.0, 1572864.0, nil, nil, 2097152.0, nil, nil, nil, nil, nil, nil, nil, nil, nil]" ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 105 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 98 insertions(+), 7 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..90e4302 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,106 @@ class ESJobs return 0 end - def output + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) result = { - 'stats.count' => @stats['stats.count'] + 'kvcount' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] + + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + end + end + + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = value.to_s + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + set_default_value(result, stats, metrics) + kvcount(result, job) end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @result['kvcount'] = @result['kvcount'].to_a.sort.to_h + @results = {} + compact_hash('', @result) + puts @results.to_yaml end end -- 2.23.0

2 1

[PATCH compass-ci] scheduler: boot.libvirt: response 'SRV_HTTP_PORT'
by Ren Wen 15 Jan '21

15 Jan '21

For caller to download libvirt config from '/srv/cci/libvirt-xml'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/scheduler/start | 2 ++ src/scheduler/constants.cr | 2 ++ src/scheduler/find_job_boot.cr | 1 + 3 files changed, 5 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 70a3496..d8f10dc 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -25,6 +25,7 @@ names = Set.new %w[ SSHR_PORT SSHR_PORT_BASE SSHR_PORT_LEN + SRV_HTTP_PORT lab ] @@ -35,6 +36,7 @@ defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SRV_HTTP_PORT'] ||= '11300' env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1146b21..377ea72 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -29,6 +29,8 @@ LAB_REPO = "lab-z9" SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" +SRV_HTTP_PORT = (ENV.has_key?("SRV_HTTP_PORT") ? ENV["SRV_HTTP_PORT"] : "11300") + INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 625c74c..3a8be59 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -183,6 +183,7 @@ class Sched "LKP_SERVER" => job["LKP_SERVER"]?, "templates" => job["templates"]?, "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", + "SRV_HTTP_PORT" => SRV_HTTP_PORT }.to_json end -- 2.23.0

1 0

[PATCH compass-ci 2/3] container - libvirt-helper: build & start script
by Ren Wen 15 Jan '21

15 Jan '21

Mount '/srv/cci/libvirt-xml' and upload config. Use port 11392 by defaut. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/libvirt-helper/build | 5 ++++ container/libvirt-helper/start | 46 ++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) create mode 100755 container/libvirt-helper/build create mode 100755 container/libvirt-helper/start diff --git a/container/libvirt-helper/build b/container/libvirt-helper/build new file mode 100755 index 0000000..1d216ea --- /dev/null +++ b/container/libvirt-helper/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t libvirt-helper . diff --git a/container/libvirt-helper/start b/container/libvirt-helper/start new file mode 100755 index 0000000..6222eaa --- /dev/null +++ b/container/libvirt-helper/start @@ -0,0 +1,46 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIBVIRT_HELPER_HOST + LIBVIRT_HELPER_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +LIBVIRT_HELPER_PORT = defaults['LIBVIRT_HELPER_PORT'] || 11392 +LIBVIRT_PATH = '/srv/cci/libvirt-xml' +docker_rm 'libvirt-helper' + +cmd = %w[ + docker run + --name libvirt-helper + --restart=always + -d +] + env + %W[ + -e LIBVIRT_PATH=#{LIBVIRT_PATH} + -e LIBVIRT_HELPER_PORT=#{LIBVIRT_HELPER_PORT} + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -p #{LIBVIRT_HELPER_PORT}:#{LIBVIRT_HELPER_PORT} + -v /etc/localtime:/etc/localtime:ro + -v #{LIBVIRT_PATH}:#{LIBVIRT_PATH} + -w #{DEFAULT_CCI}/container/libvirt-helper/ + libvirt-helper +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./libvirt_app.rb'] + +system(*cmd) -- 2.23.0

2 2

[PATCH v3 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430 [output] --- kvcount.suite=iperf: 2 kvcount.os=openeuler: 2 kvcount.arch=aarch64: 2 kvcount.job_state=finished: 2 kvcount.tbox_group=vm-2p16g: 2 kvcount.submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430: 2 sum.stats.kmsg.timestamp:last: 2 sum.stats.dmesg.timestamp:last: 2 raw.stats.kmsg.timestamp:last: "[75.256233, 48.990905]" raw.stats.iperf.tcp.receiver.bps: "[34017924155.510155, nil]" raw.stats.dmesg.timestamp:last: "[75.256233, 48.990905]" raw.stats.iperf.tcp.sender.bps: "[34073687935.01113, nil]" raw.stats.iperf.udp.bps: "[nil, 1048573.3083402428]" avg.stats.iperf.tcp.receiver.bps: 34017924155.510155 avg.stats.iperf.tcp.sender.bps: 34073687935.01113 avg.stats.iperf.udp.bps: 1048573.3083402428 max.stats.iperf.tcp.receiver.bps: 34017924155.510155 max.stats.iperf.tcp.sender.bps: 34073687935.01113 max.stats.iperf.udp.bps: 1048573.3083402428 min.stats.iperf.tcp.receiver.bps: 34017924155.510155 min.stats.iperf.tcp.sender.bps: 34073687935.01113 min.stats.iperf.udp.bps: 1048573.3083402428 Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 109 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 100 insertions(+), 9 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..31a57a9 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,106 @@ class ESJobs return 0 end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) + result = {} + result['kvcount'] = {} + result['sum.stats'] = {} + result['raw.stats'] = {} + result['avg.stats'] = {} + result['max.stats'] = {} + result['min.stats'] = {} + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + job_metrics = stats.keys + left_metrics = metrics - job_metrics + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = value.to_s + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end + + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @results = {} + compact_hash('', @result) + puts @results.to_yaml + #puts JSON.pretty_generate(@results) end end -- 2.23.0

2 1

[PATCH compass-ci] container/delimiter: auto push image to local docker hub
by Cao Xueliang 15 Jan '21

15 Jan '21

Before: Manual push image to local docker hub. After: Auto push image to local docker hub, others can use the latest image after deploy. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/build | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/container/delimiter/build b/container/delimiter/build index fc1af95..7d9b437 100755 --- a/container/delimiter/build +++ b/container/delimiter/build @@ -2,4 +2,20 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +load_cci_defaults + docker build -t delimiter . + +push_image() +{ + local local_docker_hub="$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT" + local src_tag=delimiter:latest + local dst_tag="$local_docker_hub/$src_tag" + + docker tag "$src_tag" "$dst_tag" + docker push "$dst_tag" +} + +push_image -- 2.23.0

1 1

[PATCH v1 compass-ci] container/delimiter: auto push image to local docker hub
by Cao Xueliang 15 Jan '21

15 Jan '21

Before: Manual push image to local docker hub. After: Auto push image to local docker hub, others can use the latest image after deploy. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/defconfig.sh | 10 ++++++++++ container/delimiter/build | 6 ++++++ 2 files changed, 16 insertions(+) diff --git a/container/defconfig.sh b/container/defconfig.sh index 97d924d..4a5f5fa 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -31,3 +31,13 @@ set_es_indices() { find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; } + +push_image() +{ + local local_docker_hub="$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT" + local src_tag=$1 + local dst_tag="$local_docker_hub/$src_tag" + + docker tag "$src_tag" "$dst_tag" + docker push "$dst_tag" +} diff --git a/container/delimiter/build b/container/delimiter/build index fc1af95..f7f53e7 100755 --- a/container/delimiter/build +++ b/container/delimiter/build @@ -2,4 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +load_cci_defaults + docker build -t delimiter . + +push_image delimiter:latest -- 2.23.0

1 0

[PATCH compass-ci 1/3] container - libvirt-helper: Dockerfile
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/libvirt-helper/Dockerfile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 container/libvirt-helper/Dockerfile diff --git a/container/libvirt-helper/Dockerfile b/container/libvirt-helper/Dockerfile new file mode 100644 index 0000000..638ebd4 --- /dev/null +++ b/container/libvirt-helper/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER 15991987063(a)163.com + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of output results [example] es-jobs submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430 [output] {"kvcount.suite=iperf"=>2, "kvcount.os=openeuler"=>2, "kvcount.arch=aarch64"=>2, "kvcount.job_state=finished"=>2, "kvcount.tbox_group=vm-2p16g"=>2, "kvcount.submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430"=>2, "sum.stats.kmsg.timestamp:last"=>2, "sum.stats.dmesg.timestamp:last"=>2, "raw.stats.kmsg.timestamp:last"=>[75.256233, 48.990905], "raw.stats.iperf.tcp.receiver.bps"=>[34017924155.510155, nil], "raw.stats.dmesg.timestamp:last"=>[75.256233, 48.990905], "raw.stats.iperf.tcp.sender.bps"=>[34073687935.01113, nil], "raw.stats.iperf.udp.bps"=>[nil, 1048573.3083402428], "avg.stats.iperf.tcp.receiver.bps"=>34017924155.510155, "avg.stats.iperf.tcp.sender.bps"=>34073687935.01113, "avg.stats.iperf.udp.bps"=>1048573.3083402428, "max.stats.iperf.tcp.receiver.bps"=>34017924155.510155, "max.stats.iperf.tcp.sender.bps"=>34073687935.01113, "max.stats.iperf.udp.bps"=>1048573.3083402428, "min.stats.iperf.tcp.receiver.bps"=>34017924155.510155, "min.stats.iperf.tcp.sender.bps"=>34073687935.01113, "min.stats.iperf.udp.bps"=>1048573.3083402428} Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 115 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 102 insertions(+), 13 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..754578d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,6 +2,7 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -18,10 +19,10 @@ class ESJobs @refine_jobs = [] @jobs = {} @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' + 0 => 'success', + 1 => 'unknown', + 2 => 'warning', + 3 => 'has_error' } set_defaults deal_jobs @@ -123,17 +124,105 @@ class ESJobs return 0 end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) + result = {} + result['kvcount'] = {} + result['sum.stats'] = {} + result['raw.stats'] = {} + result['avg.stats'] = {} + result['max.stats'] = {} + result['min.stats'] = {} + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + job_metrics = stats.keys + left_metrics = metrics - job_metrics + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end + + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @results = {} + compact_hash('', @result) + pp @results + #puts JSON.pretty_generate(@results) end end -- 2.23.0

2 1

[PATCH compass-ci] sbin/run_job: kill process after running qemu/docker
by Hu Xuejiao 15 Jan '21

15 Jan '21

[why] 1.host-info is adapted in physical machine, so it replaced to ebizzy.yaml 2.after running multi-docker and multi-qemu, these processes need to be deleted Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sbin/run_job b/sbin/run_job index 325fa9f..24d35cc 100755 --- a/sbin/run_job +++ b/sbin/run_job @@ -24,7 +24,7 @@ submit_one_job() submit_job() { - yaml_args=("$cci_makepkg_file" "host-info.yaml") + yaml_args=("$cci_makepkg_file" "ebizzy.yaml") for yaml in "${yaml_args[@]}" do submit_one_job & @@ -75,3 +75,6 @@ run_docker() run_qemu run_docker wait + +kill $(cat pid) +kill $(cat $CCI_SRC/providers/dc.pid) -- 2.23.0

2 1

[PATCH v4 compass-ci] tool/es-jobs: a new tool for summarizing jobs information
by Wu Zhende 15 Jan '21

15 Jan '21

Use es-jobs to summarizes information about a group of related jobs. [Example] es-jobs submit_id=xxx -f 'job_state,dmesg.timestamp:last' -r '-1' -s 'on_fail' means: 1. query the job whose submit_id is xxx 2. I want to get job_state,dmesg.timestamp:last 3. don't refine jobs 4. -s 'on_fail' to get data in stats which contain 'on_fail' [Output] { "job_state": [ "crystal.605716.failed" ], "dmesg.timestamp:last": [ 657.181408 ], "stats_filter_result": { "crystal.605716.openeuler_docker.\u001b[0mwordpress_build_on_fail": 1 }, "stats.count": { "stats.has_error": 1, "stats.has_error_jobs": [ "crystal.605716" ] } } Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 139 +++++++++++++++++++++++++++++++++++++++++++++++++ sbin/es-jobs | 55 +++++++++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 lib/es_jobs.rb create mode 100755 sbin/es-jobs diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb new file mode 100644 index 0000000..a90b1ba --- /dev/null +++ b/lib/es_jobs.rb @@ -0,0 +1,139 @@ +# SPDX-License-Identifier: GPL-2.0-only +# frozen_string_literal: true + +LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/stats" +require_relative './es_query' + +# deal jobs search from es +class ESJobs + def initialize(es_query, my_refine = [], fields = [], stats_filter = []) + @es_query = es_query + @es = ESQuery.new(ES_HOST, ES_PORT) + @refine = my_refine + @fields = fields + @stats_filter = stats_filter + @stats_filter_result = {} + @refine_jobs = [] + @jobs = {} + @stats_level = { + 0 => 'stats.success', + 1 => 'stats.unknown', + 2 => 'stats.warning', + 3 => 'stats.has_error' + } + set_defaults + deal_jobs + end + + def set_defaults + query_result = @es.multi_field_query(@es_query) + query_result['hits']['hits'].each do |job| + @jobs[job['_id']] = job['_source'] + end + + @stats = { + 'stats.count' => Hash.new(0), + 'stats.sum' => Hash.new(0), + 'stats.avg' => Hash.new(0) + } + @result = {} + @fields.each do |field| + @result[field] = [] + end + end + + def add_result_fields(job, level) + return unless @refine.include?(level) || @refine.include?(-1) + + @refine_jobs << job['id'] + @fields.each do |field| + value = job[field] + if value + value = job['id'] + '.' + value if field == 'job_state' + @result[field] << value + end + + next unless job['stats'] + + @result[field] << job['stats'][field] if job['stats'][field] + end + end + + def deal_jobs + stats_count = Hash.new(0) + stats_jobs = {} + + @jobs.each do |job_id, job| + level = deal_stats(job) + add_result_fields(job, level) + + stat_key = @stats_level[level] + stat_jobs_key = stat_key + '_jobs' + + stats_count[stat_key] += 1 + stats_jobs[stat_jobs_key] ||= [] + stats_jobs[stat_jobs_key] << job_id + end + + @stats['stats.count'].merge!(stats_count) + @stats['stats.count'].merge!(stats_jobs) + end + + def deal_stats(job, level = 0) + return 1 unless job['stats'] + + job['stats'].each do |key, value| + match_stats_filter(key, value, job['id']) + calculate_stat(key, value) + level = get_stat_level(key, level) + end + return level + end + + def match_stats_filter(key, value, job_id) + @stats_filter.each do |filter| + next unless key.include?(filter) + + key = job_id + '.' + key + @stats_filter_result[key] = value + + break + end + end + + def calculate_stat(key, value) + if function_stat?(key) + return unless @fields.include?('stats.sum') + + @stats['stats.sum'][key] += value + else + return unless @fields.include?('stats.avg') + + @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + end + end + + def get_stat_level(stat, level) + return level if level >= 3 + return 3 if stat.match(/error|fail/i) + return 2 if stat.match(/warn/i) + + return 0 + end + + def output + result = { + 'stats.count' => @stats['stats.count'] + } + + @stats.each do |key, value| + result[key] = value if @fields.include?(key) + end + + @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? + @result.merge!(result) + puts JSON.pretty_generate(@result) + end +end diff --git a/sbin/es-jobs b/sbin/es-jobs new file mode 100755 index 0000000..c055fe9 --- /dev/null +++ b/sbin/es-jobs @@ -0,0 +1,55 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'optparse' +require 'json' + +require_relative '../lib/es_jobs' + +def parse_argv + items = {} + ARGV.each do |item| + key, value = item.split('=') + if key && value + value_list = value.split(',') + items[key] = value_list.length > 1 ? value_list : value + end + end + items +end +opt_refine = [-1] +opt_fields = [] +opt_stats_filter = [] +opt_parser = OptionParser.new do |opts| + opts.banner = 'Usage: es-jobs [options] search_key1=val1[,val2..] ..' + + opts.separator 'search_key can be submit_id, group_id' + opts.separator 'How to use -r' + opts.separator 'Like es-jobs submit_id=xxx -r "0,1,2,3"' + opts.separator '-1 means not refine, is the default value' + opts.separator ' 0 means stats.succes, refine the jobs whose test cases are all successfully executed' + opts.separator ' 1 means stats.unknown, refine the jobs without the stats' + opts.separator ' 2 means stats.warning, refine the jobs with warnings in tese cases' + opts.separator ' 3 means stats.has_error, refine the jobs with errors in tese cases' + + opts.on('-r fields', '--refine fields', 'refine jobs') do |fields| + opt_refine = fields.split(',').map(&:to_i) + end + + opts.on('-f fields', '--fields fields', 'fields you want to see') do |fields| + opt_fields = fields.split(',') + end + + opts.on('-s fields', '--stats-filter fields', 'return data contains fields in stats') do |fields| + opt_stats_filter = fields.split(',') + end +end + +opt_parser.parse!(ARGV) +items = parse_argv +raise 'Please enter a query' if items.empty? + +es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter) +es_jobs.output -- 2.23.0

2 7

[PATCH compass-ci] use cd "$xxx" for safety reason
by Liu Yinsi 15 Jan '21

15 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/docker2rootfs/common | 4 ++-- sparrow/4-docker/buildall | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index e908079..c26bbc8 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -96,7 +96,7 @@ load_docker_img() cp_rootfs() { docker cp -a rootfs-docker:/tmp/$1 /tmp/ - cd $2 + cd "$2" zcat /tmp/$1 |cpio -idmv } @@ -148,7 +148,7 @@ create_links_vmlinuz_initrd() create_get_initrd echo "Creating links to initrd.lkp and vmlinuz..." - cd $ROOTFS_DIR + cd "$ROOTFS_DIR" ln -fs $ROOTFS_INITRD_LKP initrd.lkp cd $ROOTFS_DIR/boot ln -fs $ROOTFS_VMLINUZ vmlinuz diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index ca4406b..f24226c 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -30,7 +30,7 @@ do_one() [ -n "$LKP_SRC" ] && log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( - cd $container + cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then -- 2.23.0

2 4

[PATCH compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of output results [example] es-jobs submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430 [output before] { "stats.count": { "stats.unknown": 2, "stats.success": 2, "stats.unknown_jobs": [ "crystal.613986", "crystal.613985" ], "stats.success_jobs": [ "crystal.597093", "crystal.597090" ] } } [output after] { "stats.count.unknown": 2, "stats.count.success": 2, "stats.count.unknown_jobs": [ "crystal.613986", "crystal.613985" ], "stats.count.success_jobs": [ "crystal.597093", "crystal.597090" ] } Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..c536fe3 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -18,10 +18,10 @@ class ESJobs @refine_jobs = [] @jobs = {} @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' + 0 => 'success', + 1 => 'unknown', + 2 => 'warning', + 3 => 'has_error' } set_defaults deal_jobs @@ -123,6 +123,22 @@ class ESJobs return 0 end + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + def output result = { 'stats.count' => @stats['stats.count'] @@ -134,6 +150,8 @@ class ESJobs @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? @result.merge!(result) - puts JSON.pretty_generate(@result) + @results = {} + compact_hash('', @result) + puts JSON.pretty_generate(@results) end end -- 2.23.0

2 3

[PATCH compass-ci] scheduler: fix "no job now" when consuming jobs within docker
by Ren Wen 14 Jan '21

14 Jan '21

[Error] "message":"Missing required job key: 'my_token'. [Why] The field 'ipxe_kernel_params' doesn't need to be initialized when consuming the job. [How] Remove 'ipxe_kernel_params' from 'initialized_keys'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8377bab..1d25236 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -427,6 +427,7 @@ class Job "kernel_version", "kernel_uri", "kernel_params", + "ipxe_kernel_params", "linux_vmlinuz_path"] end -- 2.23.0

1 1

[PATCH v2 compass-ci] scheduler: fix "no job now" when consuming jobs within docker
by Ren Wen 14 Jan '21

14 Jan '21

[Error] "message":"Missing required job key: 'my_token'. [Why] The field 'ipxe_kernel_params' doesn't need to be initialized when consuming the job, otherwise, will do the account check. [How] Remove 'ipxe_kernel_params' from 'initialized_keys'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8377bab..1d25236 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -427,6 +427,7 @@ class Job "kernel_version", "kernel_uri", "kernel_params", + "ipxe_kernel_params", "linux_vmlinuz_path"] end -- 2.23.0

1 0