- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] auto_submit: don't submit job with too old commit
by Li Yuanchao 15 Jan '21

15 Jan '21

If the last commit of a repository is too old, such as 'two months ago', that means the repository is not active. We don't care such repositories. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index ecc4d70..5648d59 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -31,6 +31,9 @@ class AutoSubmit def submit(newrefs_info, submit_argv) newrefs_info['new_refs']['heads'].each do |branch, commit_id| + inactive_time = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git log --pretty=format:"%cr" -1 #{commit_id}) + next if inactive_time =~ /(month|year)/ + real_argvs = Array.new(submit_argv) real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") real_argvs.push("upstream_commit=#{commit_id}") -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: fix "build-pkg" value is nil can't upload package
by Cao Xueliang 15 Jan '21

15 Jan '21

[why] If the build-pkg value is nil, we didn't set the default upload path. [how] If the build-pkg value is nil, we set it to empty hash, that will get a default upload path. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1d25236..07bfda0 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -256,7 +256,9 @@ class Job pkg_style = @hash[item]? break if pkg_style end - return nil unless pkg_style && pkg_style != nil + return nil unless pkg_style + + pkg_style = JSON.parse("{}") if pkg_style == nil tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] -- 2.23.0

3 4

[PATCH compass-ci] delimiter: create a delimiter account when start delimiter container
by Cao Xueliang 15 Jan '21

15 Jan '21

Before: Use compass-ci-robot(a)qq.com as the bisect job subqueue, wait to consume will take a long time. After: User delimiter@localhost as the bisect job subqueue, wait to consume will take a short time. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/Dockerfile | 2 +- container/delimiter/start | 2 ++ src/delimiter.rb | 4 ++++ src/delimiter/constants.rb | 3 ++- src/delimiter/utils.rb | 6 +++--- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/container/delimiter/Dockerfile b/container/delimiter/Dockerfile index 93c52c6..ab7519d 100644 --- a/container/delimiter/Dockerfile +++ b/container/delimiter/Dockerfile @@ -10,4 +10,4 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN umask 002 && \ echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ - gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket terminal-table mail + gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket terminal-table mail io-console diff --git a/container/delimiter/start b/container/delimiter/start index 669923b..2a68df3 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -7,6 +7,8 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ + ES_HOST + ES_PORT TASKQUEUE_HOST TASKQUEUE_PORT ] diff --git a/src/delimiter.rb b/src/delimiter.rb index bd40775..898fa0e 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -2,9 +2,13 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../lib/build_my_info_client' require_relative './delimiter/delimiter' begin + build_my_info = BuildMyInfo.new(DELIMITER_EMAIL, DELIMITER_NAME, ENV['lab']) + build_my_info.store_account_info + delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 7b14558..23b5053 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -6,7 +6,8 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' GIT_MIRROR_HOST = ENV['GIT_MIRROR_HOST'] || '172.17.0.1' MONITOR_HOST = ENV['MONITOR_HOST'] || '172.17.0.1' MONITOR_PORT = ENV['MONITOR_PORT'] || '11310' -DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'caoxl78320(a)163.com' +DELIMITER_EMAIL = ENV['DELIMITER_EMAIL'] || 'delimiter@localhost' +DELIMITER_NAME = ENV['DELIMITER_NAME'] || 'delimiter' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 61fe6ab..aaad394 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -118,17 +118,17 @@ module Utils end def get_account_info - ESQuery.new(index: 'accounts').query_by_id(DELIMITER_ACCONUT) + ESQuery.new(index: 'accounts').query_by_id(DELIMITER_EMAIL) end def init_job_content(job_id) job_yaml = AssistResult.new.get_job_yaml(job_id) - raise "get job yaml failed, job id: #{job_id} !" unless job_yaml + raise "get job yaml failed, job id: #{job_id}" unless job_yaml job = JSON.parse job_yaml account_info = get_account_info - raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info + raise "query delimiter account info failed: #{DELIMITER_EMAIL}" unless account_info record_jobs(job['id'], job['upstream_commit']) -- 2.23.0

2 6

[PATCH v2 compass-ci 3/3] container - assist-cci: upload function
by Ren Wen 15 Jan '21

15 Jan '21

Upload an XML template of libvirt. Response is the modified filename. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/routes.rb | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/assist-cci/routes.rb diff --git a/container/assist-cci/routes.rb b/container/assist-cci/routes.rb new file mode 100755 index 0000000..0e71a75 --- /dev/null +++ b/container/assist-cci/routes.rb @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'digest/md5' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_CCI_PORT'] +end + +LIBVIRT_XML_PATH = ENV['LIBVIRT_XML_PATH'] +VERSION = '0.1' + +get '/' do + "version: #{VERSION}" +end + +post '/upload' do + if params[:file] && (tempfile = params[:file][:tempfile]) && (filename = params[:file][:filename]) + dirname = File.basename(filename, '.xml') + content = tempfile.read + md5_filename = Digest::MD5.hexdigest(content) + '.xml' + + dir_path = "#{LIBVIRT_XML_PATH}/#{dirname}" + Dir.exist?(dir_path) || Dir.mkdir(dir_path) + + target = "#{dir_path}/#{md5_filename}" + File.open(target, 'wb') { |f| f.write content } + + File.join(dirname, md5_filename) + else + '' + end +end -- 2.23.0

1 0

[PATCH compass-ci] scheduler: change 'templates' to 'vt'
by Ren Wen 15 Jan '21

15 Jan '21

Use 'vt' (virt) instead of 'templates', the latter is not clear enough. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 625c74c..152d549 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -181,7 +181,7 @@ class Sched "kernel_params" => _kernel_params, "result_root" => job.result_root, "LKP_SERVER" => job["LKP_SERVER"]?, - "templates" => job["templates"]?, + "vt" => job["vt"]?, "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", }.to_json end -- 2.23.0

1 0

[PATCH v4 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archLinux [output] --- kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archLinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 sum.stats.last_state.test.iperf.exit_code.127: 12 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: "[524288.0, nil, nil, nil, nil, nil, nil, nil, nil, 524288.0, 524288.0, nil, nil, nil, nil, nil, 516608.0, nil, 1048576.0, nil, nil, nil, nil, nil, nil, nil, 524288.0, nil, nil, nil, nil, 2104832.0, 1572864.0, nil, nil, 2097152.0, nil, nil, nil, nil, nil, nil, nil, nil, nil]" ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 105 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 98 insertions(+), 7 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..90e4302 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,106 @@ class ESJobs return 0 end - def output + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) result = { - 'stats.count' => @stats['stats.count'] + 'kvcount' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] + + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + end + end + + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = value.to_s + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + set_default_value(result, stats, metrics) + kvcount(result, job) end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @result['kvcount'] = @result['kvcount'].to_a.sort.to_h + @results = {} + compact_hash('', @result) + puts @results.to_yaml end end -- 2.23.0

2 1

[PATCH compass-ci] scheduler: boot.libvirt: response 'SRV_HTTP_PORT'
by Ren Wen 15 Jan '21

15 Jan '21

For caller to download libvirt config from '/srv/cci/libvirt-xml'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/scheduler/start | 2 ++ src/scheduler/constants.cr | 2 ++ src/scheduler/find_job_boot.cr | 1 + 3 files changed, 5 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 70a3496..d8f10dc 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -25,6 +25,7 @@ names = Set.new %w[ SSHR_PORT SSHR_PORT_BASE SSHR_PORT_LEN + SRV_HTTP_PORT lab ] @@ -35,6 +36,7 @@ defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SRV_HTTP_PORT'] ||= '11300' env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1146b21..377ea72 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -29,6 +29,8 @@ LAB_REPO = "lab-z9" SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" +SRV_HTTP_PORT = (ENV.has_key?("SRV_HTTP_PORT") ? ENV["SRV_HTTP_PORT"] : "11300") + INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 625c74c..3a8be59 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -183,6 +183,7 @@ class Sched "LKP_SERVER" => job["LKP_SERVER"]?, "templates" => job["templates"]?, "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", + "SRV_HTTP_PORT" => SRV_HTTP_PORT }.to_json end -- 2.23.0

1 0

[PATCH compass-ci 2/3] container - libvirt-helper: build & start script
by Ren Wen 15 Jan '21

15 Jan '21

Mount '/srv/cci/libvirt-xml' and upload config. Use port 11392 by defaut. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/libvirt-helper/build | 5 ++++ container/libvirt-helper/start | 46 ++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) create mode 100755 container/libvirt-helper/build create mode 100755 container/libvirt-helper/start diff --git a/container/libvirt-helper/build b/container/libvirt-helper/build new file mode 100755 index 0000000..1d216ea --- /dev/null +++ b/container/libvirt-helper/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t libvirt-helper . diff --git a/container/libvirt-helper/start b/container/libvirt-helper/start new file mode 100755 index 0000000..6222eaa --- /dev/null +++ b/container/libvirt-helper/start @@ -0,0 +1,46 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIBVIRT_HELPER_HOST + LIBVIRT_HELPER_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +LIBVIRT_HELPER_PORT = defaults['LIBVIRT_HELPER_PORT'] || 11392 +LIBVIRT_PATH = '/srv/cci/libvirt-xml' +docker_rm 'libvirt-helper' + +cmd = %w[ + docker run + --name libvirt-helper + --restart=always + -d +] + env + %W[ + -e LIBVIRT_PATH=#{LIBVIRT_PATH} + -e LIBVIRT_HELPER_PORT=#{LIBVIRT_HELPER_PORT} + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -p #{LIBVIRT_HELPER_PORT}:#{LIBVIRT_HELPER_PORT} + -v /etc/localtime:/etc/localtime:ro + -v #{LIBVIRT_PATH}:#{LIBVIRT_PATH} + -w #{DEFAULT_CCI}/container/libvirt-helper/ + libvirt-helper +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./libvirt_app.rb'] + +system(*cmd) -- 2.23.0

2 2

[PATCH v3 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430 [output] --- kvcount.suite=iperf: 2 kvcount.os=openeuler: 2 kvcount.arch=aarch64: 2 kvcount.job_state=finished: 2 kvcount.tbox_group=vm-2p16g: 2 kvcount.submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430: 2 sum.stats.kmsg.timestamp:last: 2 sum.stats.dmesg.timestamp:last: 2 raw.stats.kmsg.timestamp:last: "[75.256233, 48.990905]" raw.stats.iperf.tcp.receiver.bps: "[34017924155.510155, nil]" raw.stats.dmesg.timestamp:last: "[75.256233, 48.990905]" raw.stats.iperf.tcp.sender.bps: "[34073687935.01113, nil]" raw.stats.iperf.udp.bps: "[nil, 1048573.3083402428]" avg.stats.iperf.tcp.receiver.bps: 34017924155.510155 avg.stats.iperf.tcp.sender.bps: 34073687935.01113 avg.stats.iperf.udp.bps: 1048573.3083402428 max.stats.iperf.tcp.receiver.bps: 34017924155.510155 max.stats.iperf.tcp.sender.bps: 34073687935.01113 max.stats.iperf.udp.bps: 1048573.3083402428 min.stats.iperf.tcp.receiver.bps: 34017924155.510155 min.stats.iperf.tcp.sender.bps: 34073687935.01113 min.stats.iperf.udp.bps: 1048573.3083402428 Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 109 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 100 insertions(+), 9 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..31a57a9 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,106 @@ class ESJobs return 0 end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) + result = {} + result['kvcount'] = {} + result['sum.stats'] = {} + result['raw.stats'] = {} + result['avg.stats'] = {} + result['max.stats'] = {} + result['min.stats'] = {} + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + job_metrics = stats.keys + left_metrics = metrics - job_metrics + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = value.to_s + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end + + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @results = {} + compact_hash('', @result) + puts @results.to_yaml + #puts JSON.pretty_generate(@results) end end -- 2.23.0

2 1

[PATCH compass-ci] container/delimiter: auto push image to local docker hub
by Cao Xueliang 15 Jan '21

15 Jan '21

Before: Manual push image to local docker hub. After: Auto push image to local docker hub, others can use the latest image after deploy. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/build | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/container/delimiter/build b/container/delimiter/build index fc1af95..7d9b437 100755 --- a/container/delimiter/build +++ b/container/delimiter/build @@ -2,4 +2,20 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +load_cci_defaults + docker build -t delimiter . + +push_image() +{ + local local_docker_hub="$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT" + local src_tag=delimiter:latest + local dst_tag="$local_docker_hub/$src_tag" + + docker tag "$src_tag" "$dst_tag" + docker push "$dst_tag" +} + +push_image -- 2.23.0

1 1

[PATCH v1 compass-ci] container/delimiter: auto push image to local docker hub
by Cao Xueliang 15 Jan '21

15 Jan '21

Before: Manual push image to local docker hub. After: Auto push image to local docker hub, others can use the latest image after deploy. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/defconfig.sh | 10 ++++++++++ container/delimiter/build | 6 ++++++ 2 files changed, 16 insertions(+) diff --git a/container/defconfig.sh b/container/defconfig.sh index 97d924d..4a5f5fa 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -31,3 +31,13 @@ set_es_indices() { find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; } + +push_image() +{ + local local_docker_hub="$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT" + local src_tag=$1 + local dst_tag="$local_docker_hub/$src_tag" + + docker tag "$src_tag" "$dst_tag" + docker push "$dst_tag" +} diff --git a/container/delimiter/build b/container/delimiter/build index fc1af95..f7f53e7 100755 --- a/container/delimiter/build +++ b/container/delimiter/build @@ -2,4 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +load_cci_defaults + docker build -t delimiter . + +push_image delimiter:latest -- 2.23.0

1 0

[PATCH compass-ci 1/3] container - libvirt-helper: Dockerfile
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/libvirt-helper/Dockerfile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 container/libvirt-helper/Dockerfile diff --git a/container/libvirt-helper/Dockerfile b/container/libvirt-helper/Dockerfile new file mode 100644 index 0000000..638ebd4 --- /dev/null +++ b/container/libvirt-helper/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER 15991987063(a)163.com + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of output results [example] es-jobs submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430 [output] {"kvcount.suite=iperf"=>2, "kvcount.os=openeuler"=>2, "kvcount.arch=aarch64"=>2, "kvcount.job_state=finished"=>2, "kvcount.tbox_group=vm-2p16g"=>2, "kvcount.submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430"=>2, "sum.stats.kmsg.timestamp:last"=>2, "sum.stats.dmesg.timestamp:last"=>2, "raw.stats.kmsg.timestamp:last"=>[75.256233, 48.990905], "raw.stats.iperf.tcp.receiver.bps"=>[34017924155.510155, nil], "raw.stats.dmesg.timestamp:last"=>[75.256233, 48.990905], "raw.stats.iperf.tcp.sender.bps"=>[34073687935.01113, nil], "raw.stats.iperf.udp.bps"=>[nil, 1048573.3083402428], "avg.stats.iperf.tcp.receiver.bps"=>34017924155.510155, "avg.stats.iperf.tcp.sender.bps"=>34073687935.01113, "avg.stats.iperf.udp.bps"=>1048573.3083402428, "max.stats.iperf.tcp.receiver.bps"=>34017924155.510155, "max.stats.iperf.tcp.sender.bps"=>34073687935.01113, "max.stats.iperf.udp.bps"=>1048573.3083402428, "min.stats.iperf.tcp.receiver.bps"=>34017924155.510155, "min.stats.iperf.tcp.sender.bps"=>34073687935.01113, "min.stats.iperf.udp.bps"=>1048573.3083402428} Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 115 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 102 insertions(+), 13 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..754578d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,6 +2,7 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -18,10 +19,10 @@ class ESJobs @refine_jobs = [] @jobs = {} @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' + 0 => 'success', + 1 => 'unknown', + 2 => 'warning', + 3 => 'has_error' } set_defaults deal_jobs @@ -123,17 +124,105 @@ class ESJobs return 0 end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) + result = {} + result['kvcount'] = {} + result['sum.stats'] = {} + result['raw.stats'] = {} + result['avg.stats'] = {} + result['max.stats'] = {} + result['min.stats'] = {} + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + job_metrics = stats.keys + left_metrics = metrics - job_metrics + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end + + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @results = {} + compact_hash('', @result) + pp @results + #puts JSON.pretty_generate(@results) end end -- 2.23.0

2 1

[PATCH compass-ci] sbin/run_job: kill process after running qemu/docker
by Hu Xuejiao 15 Jan '21

15 Jan '21

[why] 1.host-info is adapted in physical machine, so it replaced to ebizzy.yaml 2.after running multi-docker and multi-qemu, these processes need to be deleted Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sbin/run_job b/sbin/run_job index 325fa9f..24d35cc 100755 --- a/sbin/run_job +++ b/sbin/run_job @@ -24,7 +24,7 @@ submit_one_job() submit_job() { - yaml_args=("$cci_makepkg_file" "host-info.yaml") + yaml_args=("$cci_makepkg_file" "ebizzy.yaml") for yaml in "${yaml_args[@]}" do submit_one_job & @@ -75,3 +75,6 @@ run_docker() run_qemu run_docker wait + +kill $(cat pid) +kill $(cat $CCI_SRC/providers/dc.pid) -- 2.23.0

2 1

[PATCH v4 compass-ci] tool/es-jobs: a new tool for summarizing jobs information
by Wu Zhende 15 Jan '21

15 Jan '21

Use es-jobs to summarizes information about a group of related jobs. [Example] es-jobs submit_id=xxx -f 'job_state,dmesg.timestamp:last' -r '-1' -s 'on_fail' means: 1. query the job whose submit_id is xxx 2. I want to get job_state,dmesg.timestamp:last 3. don't refine jobs 4. -s 'on_fail' to get data in stats which contain 'on_fail' [Output] { "job_state": [ "crystal.605716.failed" ], "dmesg.timestamp:last": [ 657.181408 ], "stats_filter_result": { "crystal.605716.openeuler_docker.\u001b[0mwordpress_build_on_fail": 1 }, "stats.count": { "stats.has_error": 1, "stats.has_error_jobs": [ "crystal.605716" ] } } Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 139 +++++++++++++++++++++++++++++++++++++++++++++++++ sbin/es-jobs | 55 +++++++++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 lib/es_jobs.rb create mode 100755 sbin/es-jobs diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb new file mode 100644 index 0000000..a90b1ba --- /dev/null +++ b/lib/es_jobs.rb @@ -0,0 +1,139 @@ +# SPDX-License-Identifier: GPL-2.0-only +# frozen_string_literal: true + +LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/stats" +require_relative './es_query' + +# deal jobs search from es +class ESJobs + def initialize(es_query, my_refine = [], fields = [], stats_filter = []) + @es_query = es_query + @es = ESQuery.new(ES_HOST, ES_PORT) + @refine = my_refine + @fields = fields + @stats_filter = stats_filter + @stats_filter_result = {} + @refine_jobs = [] + @jobs = {} + @stats_level = { + 0 => 'stats.success', + 1 => 'stats.unknown', + 2 => 'stats.warning', + 3 => 'stats.has_error' + } + set_defaults + deal_jobs + end + + def set_defaults + query_result = @es.multi_field_query(@es_query) + query_result['hits']['hits'].each do |job| + @jobs[job['_id']] = job['_source'] + end + + @stats = { + 'stats.count' => Hash.new(0), + 'stats.sum' => Hash.new(0), + 'stats.avg' => Hash.new(0) + } + @result = {} + @fields.each do |field| + @result[field] = [] + end + end + + def add_result_fields(job, level) + return unless @refine.include?(level) || @refine.include?(-1) + + @refine_jobs << job['id'] + @fields.each do |field| + value = job[field] + if value + value = job['id'] + '.' + value if field == 'job_state' + @result[field] << value + end + + next unless job['stats'] + + @result[field] << job['stats'][field] if job['stats'][field] + end + end + + def deal_jobs + stats_count = Hash.new(0) + stats_jobs = {} + + @jobs.each do |job_id, job| + level = deal_stats(job) + add_result_fields(job, level) + + stat_key = @stats_level[level] + stat_jobs_key = stat_key + '_jobs' + + stats_count[stat_key] += 1 + stats_jobs[stat_jobs_key] ||= [] + stats_jobs[stat_jobs_key] << job_id + end + + @stats['stats.count'].merge!(stats_count) + @stats['stats.count'].merge!(stats_jobs) + end + + def deal_stats(job, level = 0) + return 1 unless job['stats'] + + job['stats'].each do |key, value| + match_stats_filter(key, value, job['id']) + calculate_stat(key, value) + level = get_stat_level(key, level) + end + return level + end + + def match_stats_filter(key, value, job_id) + @stats_filter.each do |filter| + next unless key.include?(filter) + + key = job_id + '.' + key + @stats_filter_result[key] = value + + break + end + end + + def calculate_stat(key, value) + if function_stat?(key) + return unless @fields.include?('stats.sum') + + @stats['stats.sum'][key] += value + else + return unless @fields.include?('stats.avg') + + @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + end + end + + def get_stat_level(stat, level) + return level if level >= 3 + return 3 if stat.match(/error|fail/i) + return 2 if stat.match(/warn/i) + + return 0 + end + + def output + result = { + 'stats.count' => @stats['stats.count'] + } + + @stats.each do |key, value| + result[key] = value if @fields.include?(key) + end + + @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? + @result.merge!(result) + puts JSON.pretty_generate(@result) + end +end diff --git a/sbin/es-jobs b/sbin/es-jobs new file mode 100755 index 0000000..c055fe9 --- /dev/null +++ b/sbin/es-jobs @@ -0,0 +1,55 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'optparse' +require 'json' + +require_relative '../lib/es_jobs' + +def parse_argv + items = {} + ARGV.each do |item| + key, value = item.split('=') + if key && value + value_list = value.split(',') + items[key] = value_list.length > 1 ? value_list : value + end + end + items +end +opt_refine = [-1] +opt_fields = [] +opt_stats_filter = [] +opt_parser = OptionParser.new do |opts| + opts.banner = 'Usage: es-jobs [options] search_key1=val1[,val2..] ..' + + opts.separator 'search_key can be submit_id, group_id' + opts.separator 'How to use -r' + opts.separator 'Like es-jobs submit_id=xxx -r "0,1,2,3"' + opts.separator '-1 means not refine, is the default value' + opts.separator ' 0 means stats.succes, refine the jobs whose test cases are all successfully executed' + opts.separator ' 1 means stats.unknown, refine the jobs without the stats' + opts.separator ' 2 means stats.warning, refine the jobs with warnings in tese cases' + opts.separator ' 3 means stats.has_error, refine the jobs with errors in tese cases' + + opts.on('-r fields', '--refine fields', 'refine jobs') do |fields| + opt_refine = fields.split(',').map(&:to_i) + end + + opts.on('-f fields', '--fields fields', 'fields you want to see') do |fields| + opt_fields = fields.split(',') + end + + opts.on('-s fields', '--stats-filter fields', 'return data contains fields in stats') do |fields| + opt_stats_filter = fields.split(',') + end +end + +opt_parser.parse!(ARGV) +items = parse_argv +raise 'Please enter a query' if items.empty? + +es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter) +es_jobs.output -- 2.23.0

2 7

[PATCH compass-ci] use cd "$xxx" for safety reason
by Liu Yinsi 15 Jan '21

15 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/docker2rootfs/common | 4 ++-- sparrow/4-docker/buildall | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index e908079..c26bbc8 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -96,7 +96,7 @@ load_docker_img() cp_rootfs() { docker cp -a rootfs-docker:/tmp/$1 /tmp/ - cd $2 + cd "$2" zcat /tmp/$1 |cpio -idmv } @@ -148,7 +148,7 @@ create_links_vmlinuz_initrd() create_get_initrd echo "Creating links to initrd.lkp and vmlinuz..." - cd $ROOTFS_DIR + cd "$ROOTFS_DIR" ln -fs $ROOTFS_INITRD_LKP initrd.lkp cd $ROOTFS_DIR/boot ln -fs $ROOTFS_VMLINUZ vmlinuz diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index ca4406b..f24226c 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -30,7 +30,7 @@ do_one() [ -n "$LKP_SRC" ] && log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( - cd $container + cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then -- 2.23.0

2 4

[PATCH compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of output results [example] es-jobs submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430 [output before] { "stats.count": { "stats.unknown": 2, "stats.success": 2, "stats.unknown_jobs": [ "crystal.613986", "crystal.613985" ], "stats.success_jobs": [ "crystal.597093", "crystal.597090" ] } } [output after] { "stats.count.unknown": 2, "stats.count.success": 2, "stats.count.unknown_jobs": [ "crystal.613986", "crystal.613985" ], "stats.count.success_jobs": [ "crystal.597093", "crystal.597090" ] } Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..c536fe3 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -18,10 +18,10 @@ class ESJobs @refine_jobs = [] @jobs = {} @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' + 0 => 'success', + 1 => 'unknown', + 2 => 'warning', + 3 => 'has_error' } set_defaults deal_jobs @@ -123,6 +123,22 @@ class ESJobs return 0 end + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + def output result = { 'stats.count' => @stats['stats.count'] @@ -134,6 +150,8 @@ class ESJobs @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? @result.merge!(result) - puts JSON.pretty_generate(@result) + @results = {} + compact_hash('', @result) + puts JSON.pretty_generate(@results) end end -- 2.23.0

2 3

[PATCH compass-ci] scheduler: fix "no job now" when consuming jobs within docker
by Ren Wen 14 Jan '21

14 Jan '21

[Error] "message":"Missing required job key: 'my_token'. [Why] The field 'ipxe_kernel_params' doesn't need to be initialized when consuming the job. [How] Remove 'ipxe_kernel_params' from 'initialized_keys'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8377bab..1d25236 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -427,6 +427,7 @@ class Job "kernel_version", "kernel_uri", "kernel_params", + "ipxe_kernel_params", "linux_vmlinuz_path"] end -- 2.23.0

1 1

[PATCH v2 compass-ci] scheduler: fix "no job now" when consuming jobs within docker
by Ren Wen 14 Jan '21

14 Jan '21

[Error] "message":"Missing required job key: 'my_token'. [Why] The field 'ipxe_kernel_params' doesn't need to be initialized when consuming the job, otherwise, will do the account check. [How] Remove 'ipxe_kernel_params' from 'initialized_keys'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8377bab..1d25236 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -427,6 +427,7 @@ class Job "kernel_version", "kernel_uri", "kernel_params", + "ipxe_kernel_params", "linux_vmlinuz_path"] end -- 2.23.0

1 0

[PATCH v7 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 14 Jan '21

14 Jan '21

[Error Msg] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [Root Cause] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= [Why] even network interfaces ready, but routing not, get ip by routing failed, so add a check export_ip_mac to ensure network is ok. [how] before: network interfaces "up"|"1" -> network is ok after: network interfaces "up"|"1" && export_ip_mac ok -> network is ok Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..2969544a6 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -91,8 +91,8 @@ network_ok() for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && export_ip_mac && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && export_ip_mac && return 0 done is_clearlinux && { @@ -296,6 +296,8 @@ export_ip_mac() export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') fi + [ -n "$PUB_NIC" ] || return 1 + export PUB_MAC=$(cat /sys/class/net/$PUB_NIC/address) [ -n "$SCHED_HOST" ] && PUB_MAC=$(echo "$PUB_MAC" | tr : -) } @@ -746,7 +748,6 @@ boot_init() setup_network run_ntpdate - export_ip_mac announce_bootup -- 2.23.0

4 8

help star
by Wu Fengguang 14 Jan '21

14 Jan '21

Xueliang is our help star, thanks Xueliang! caoxl@crystal ~/code/todo/bin% ./help-star ../people/*/credit/2021-01-14.md 24.6 Wu Fengguang 16.2 Cao Xueliang 16.1 Yu Chuan 12.2 Luan Shengde 11.8 Wei Jihui 11.7 Liu Shaofei 10.2 Zhang Dewan 8.9 Xiao Shenwei 8.5 Zhang Yale 8.3 Liu Yinsi 7.6 Wu Zhende 7.2 Ren Wen 6.9 Wang Yong 6.5 Lu Weitao 5.2 Lin Jiaxin 5.0 Li Ping 4.7 Bai Jing 4.6 Xu Xijian 4.3 Cui Yili 2.4 Li Yuanchao 2.4 Wang Chenglong 2.2 Du Kaitian 1.3 Lu Kaiyi 1.1 Hu Xuejiao 0.0 Tong Qunfeng 0.0 Wang Shuai 0.0 Cui Xucui

1 0

[PATCH v5 lkp-tests] stats: add ansible_test stats script
by Zhang Yale 14 Jan '21

14 Jan '21

[Why] Output needs to be displayed in the stats.json file, so this output need to processed as a key-value pair. [input] PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 playbook_run_on_fail: 1 [output] ansible_test.total.nr_ok: 2 ansible_test.total.nr_changed: 0 ansible_test.total.nr_failed: 1 ansible_test.total.nr_skipped: 0 ansible_test.total.nr_rescued: 0 ansible_test.total.nr_ignored: 0 playbook_run_on_fail: 1 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 stats/ansible_test diff --git a/stats/ansible_test b/stats/ansible_test new file mode 100755 index 00000000..4c7d07b3 --- /dev/null +++ b/stats/ansible_test @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + case line.chomp! + when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ + puts "ansible_test.total.nr_ok: #{$1}" + puts "ansible_test.total.nr_changed: #{$2}" + puts "ansible_test.total.nr_failed: #{$4}" + puts "ansible_test.total.nr_skipped: #{$5}" + puts "ansible_test.total.nr_rescued: #{$6}" + puts "ansible_test.total.nr_ignored: #{$7}" + when /playbook_run_on_fail/ + puts line + end +end -- 2.23.0

3 4

review star
by Wu Fengguang 14 Jan '21

14 Jan '21

Jihui is our review star, thanks Jihui! caoxl@crystal ~/code/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 98 32 52 20 Wei Jihui 71 44 125 19 Wu Fengguang 24 13 23 11 Liu Shaofei 59 16 21 9 Cao Xueliang 12 10 31 8 Yu Chuan 41 10 20 7 Lu Kaiyi 33 7 13 5 Ren Wen 16 7 8 5 Cui Yili 16 6 7 4 Bai Jing 17 3 3 4 Wu Zhende 11 6 10 3 Lin Jiaxin 9 4 7 3 Li Yuanchao 6 4 8 3 Xiao Shenwei 18 6 8 3 Wang Yong 19 6 10 2 Wang Chenglong 14 4 8 2 Lu Weitao 20 4 6 2 Li Ping 29 3 4 2 Xu Xijian 55 4 8 1 Luan Shengde 20 2 2 1 Hu Xuejiao 6 0 0 0 Zhang Dewan 51 1 2 0 Liu Yinsi 30 3 2 0 Zhang Yale

1 0

git count
by Wu Fengguang 14 Jan '21

14 Jan '21

Yinsi is our code star, thanks Yinsi! 12 Liu Yinsi 10 Luan Shengde 9 Lu Kaiyi 8 Hu Xuejiao 5 Cao Xueliang 5 Xu Xijian 5 Ren Wen 4 Lu Weitao 4 Wei Jihui 4 Wu Fengguang 4 Wu Zhende 3 Liu Shaofei 3 Wang Chenglong 3 Li Yuanchao 2 Bai Jing 2 Xiao Shenwei 2 Zhang Yale 2 Cui Yili 2 Wang Yong 2 Lin Jiaxin 1 Yu Chuan 0 Tong Qunfeng 0 Du Kaitian 0 Sun Yukui 0 Zhang Yu 0 Zhang Dewan 0 Li Ping 0 Shi Zhichao

1 0

[PATCH compass-ci 1/3] sbin/auto_submit: fix failed to submit job in container
by Liu Yinsi 14 Jan '21

14 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. [how] store auto submit info to ES to pass verification when submit jobs in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..c121ae0 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -6,6 +6,7 @@ require 'bunny' require 'json' require 'yaml' +require_relative '../lib/build_my_info_client' # receive message and auto submit job class AutoSubmit @@ -82,5 +83,15 @@ class AutoSubmit end end +option = { + my_name: 'Auto Submit Robot', + my_email: 'autosubmit@localhost', + lab: 'autosubmit' +} + +# fix verification problems when submit jobs by auto-submit service +build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) +build_my_info.config_my_info + auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

2 2