- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests 1/2] reset machine testbox by job
by Zhang Dewan 20 Jan '21

20 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++ tests/reset-tbox | 54 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..4d43850f --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: 172.168.131.2:5001/reset-tbox +queue: reset-tbox + +reset-tbox: + host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..1ca17bfd --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,54 @@ +#!/bin/bash +# host_nameset + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +FileCheck() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +TboxCheck() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +ResetTbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +ResultCheck() +{ + if [ $? -ne 0 ] ; then + echo "Failed to reboot the testbox: $host_name" + fi + echo "done." +} + +main() +{ + FileCheck + TboxCheck + ResetTbox + ResultCheck +} +main + + + -- 2.23.0

3 3

[PATCH v1 compass-ci 4/4] providers/libvirt: add a libvirt provider
by Xiao Shenwei 20 Jan '21

20 Jan '21

[why] libvirt is a vm management tool, it based on a domain.xml file to run a vm, the xml each element will be parsed qemu-kvm parameter. so, we can configure xml to run vms of diff specifications Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/client.rb | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 providers/libvirt/client.rb diff --git a/providers/libvirt/client.rb b/providers/libvirt/client.rb new file mode 100644 index 0000000..d029a35 --- /dev/null +++ b/providers/libvirt/client.rb @@ -0,0 +1,23 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'lib/mylog' +require_relative 'lib/consumer' + +def main(hostname, queues) + puts "cat #{hostname}.log" + logger = Mylog.new("#{hostname}.log") + consumer = Consumer.new(hostname, queues).add_logger(logger) + response = consumer.request_job + + if response.nil? || response['job_id'].empty? + logger.warn('no job now') + puts '----------' + puts 'no job now' + puts '----------' + return + end + + consumer.close +end -- 2.23.0

1 0

[PATCH v1 compass-ci 1/4] providers/libvirt: basic class to save common methods
by Xiao Shenwei 20 Jan '21

20 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/base.rb | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 providers/libvirt/lib/base.rb diff --git a/providers/libvirt/lib/base.rb b/providers/libvirt/lib/base.rb new file mode 100644 index 0000000..2e8a74d --- /dev/null +++ b/providers/libvirt/lib/base.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# save common variables and methods +class Base + attr_reader :logger + + def add_logger(logger) + @logger = logger + self + end +end -- 2.23.0

1 0

[PATCH v10 compass-ci] lib/es_jobs.rb: optimize es-jobs output
by Lu Kaiyi 20 Jan '21

20 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archlinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archlinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.summary.any_error=1: 24 kvcount.summary.any_fail=1: 25 kvcount.summary.any_stderr=1: 44 kvcount.summary.any_warning=1: 23 kvcount.summary.success=1: 1 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 ... raw.id.[job_state=failed]: ["crystal.122405","crystal.122406","crystal.122331","crystal.122365","crystal.122404","crystal.122438","crystal.122607","crystal.122370","crystal.122375","crystal.122410","crystal.122900","crystal.122906","crystal.122271","crystal.122451","crystal.122835","crystal.122834","crystal.122232","crystal.122403","crystal.122453","crystal.122886"] ... raw.id.[summary.any_error=1]: ["crystal.122436","crystal.122476","crystal.122803","crystal.122883","crystal.122910","crystal.122438","crystal.122452","crystal.122895","crystal.122446","crystal.122685","crystal.122687","crystal.122833","crystal.122451","crystal.122690","crystal.122719","crystal.122882","crystal.122890","crystal.122453","crystal.122443","crystal.122435","crystal.122432","crystal.122688","crystal.122689","crystal.122886"] ... sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0,null,null,null,null,null,null,null,null,524288.0,524288.0,null,null,null,null,null,516608.0,null,1048576.0,null,null,null,null,null,null,null,524288.0,null,null,null,null,2104832.0,1572864.0,null,null,2097152.0,null,null,null,null,null,null,null,null,null] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... max.stats.sched_debug.cfs_rq:/.load.stddev: 2104832.0 max.stats.softirqs.CPU1.NET_RX: 6 max.stats.slabinfo.kmalloc-512.active_objs: 1616 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 201 +++++++++++++++++++++++++++---------------------- 1 file changed, 111 insertions(+), 90 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..af0a1a9 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -3,137 +3,158 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo summary.success + summary.any_fail summary.any_error summary.any_stderr summary.any_warning].freeze + +require 'json' require "#{LKP_SRC}/lib/stats" require_relative './es_query' # deal jobs search from es class ESJobs def initialize(es_query, my_refine = [], fields = [], stats_filter = []) - @es_query = es_query - @es = ESQuery.new(ES_HOST, ES_PORT) + @jobs = query_jobs_from_es(es_query) @refine = my_refine @fields = fields @stats_filter = stats_filter - @stats_filter_result = {} @refine_jobs = [] - @jobs = {} - @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' - } - set_defaults - deal_jobs + set_jobs_summary end - def set_defaults - query_result = @es.multi_field_query(@es_query) - query_result['hits']['hits'].each do |job| - @jobs[job['_id']] = job['_source'] - end - - @stats = { - 'stats.count' => Hash.new(0), - 'stats.sum' => Hash.new(0), - 'stats.avg' => Hash.new(0) - } - @result = {} - @fields.each do |field| - @result[field] = [] - end + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs end - def add_result_fields(job, level) - return unless @refine.include?(level) || @refine.include?(-1) + def set_job_summary(stats, job) + summary_result = '' + if stats.keys.any? { |stat| stat.match(/warn/i) } + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + if stats.keys.any? { |stat| stat.match(/stderr/i) } + job['summary.any_stderr'] = 1 + summary_result = 'stderr' + end + if stats.keys.any? { |stat| stat.match(/error|nr_fail=0/i) } + job['summary.any_error'] = 1 + summary_result = 'error' + end + if stats.keys.any? { |stat| stat.match(/fail/i) } && + stats.keys.all? { |stat| !stat.match(/nr_fail=0/i) } + job['summary.any_fail'] = 1 + summary_result = 'fail' + end + return unless summary_result.empty? - @refine_jobs << job['id'] - @fields.each do |field| - value = job[field] - if value - value = job['id'] + '.' + value if field == 'job_state' - @result[field] << value - end + job['summary.success'] = 1 + end - next unless job['stats'] + # set jobs summary fields information in place + def set_jobs_summary + @jobs.each do |job| + stats = job['stats'] + next unless stats - @result[field] << job['stats'][field] if job['stats'][field] + set_job_summary(stats, job) end end - def deal_jobs - stats_count = Hash.new(0) - stats_jobs = {} + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats - @jobs.each do |job_id, job| - level = deal_stats(job) - add_result_fields(job, level) - - stat_key = @stats_level[level] - stat_jobs_key = stat_key + '_jobs' - - stats_count[stat_key] += 1 - stats_jobs[stat_jobs_key] ||= [] - stats_jobs[stat_jobs_key] << job_id + metrics.concat(stats.keys) end + metrics.uniq! + end - @stats['stats.count'].merge!(stats_count) - @stats['stats.count'].merge!(stats_jobs) + def initialize_result_hash(metrics) + result = { + 'kvcount' => {}, + 'raw.id' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} + } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result end - def deal_stats(job, level = 0) - return 1 unless job['stats'] + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } - job['stats'].each do |key, value| - match_stats_filter(key, value, job['id']) - calculate_stat(key, value) - level = get_stat_level(key, level) + stats.each do |key, value| + result['raw.stats'][key] << value end - return level end - def match_stats_filter(key, value, job_id) - @stats_filter.each do |filter| - next unless key.include?(filter) + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - key = job_id + '.' + key - @stats_filter_result[key] = value - - break + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + result['raw.id']["[#{keyword}=#{job[keyword]}]"] ||= [] + result['raw.id']["[#{keyword}=#{job[keyword]}]"] << job['id'] end end - def calculate_stat(key, value) - if function_stat?(key) - return unless @fields.include?('stats.sum') - - @stats['stats.sum'][key] += value - else - return unless @fields.include?('stats.avg') - - @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end end end - def get_stat_level(stat, level) - return level if level >= 3 - return 3 if stat.match(/error|fail/i) - return 2 if stat.match(/warn/i) + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end - return 0 + stats_count(result) + result end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def output_yaml(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + if value.is_a? Hash + output_yaml(prefix_key, value) + else + puts "#{prefix_key}: #{value.to_json}" + end end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def output + @result = query_jobs_state(@jobs) + @result['kvcount'] = @result['kvcount'].sort.to_h + @result['raw.id'] = @result['raw.id'].sort.to_h + output_yaml('', @result) end end -- 2.23.0

2 1

[PATCH v2 compass-ci] create-image: disable selinux
by Wang Chenglong 20 Jan '21

20 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..4056ff1 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,17 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + # disable selinux + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/^.*SELINUX=enforcing/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

2 2

[PATCH compass-ci 2/2] service/scheduler: save testbox info in es
by Wu Zhende 20 Jan '21

20 Jan '21

When a testbox get a job, record its information in es. The deadline field indicates the time when the job executed by the testbox times out. It's a sign that other measures are needed to intervene. Can use curl -XGET http://localhost:9200/testbox/_doc/_search -d '{"query":{"range":{"deadline":{"lt":"now"}}}}' to find all timeout testboxs and jobs Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 19 +++++++++++++++++++ src/scheduler/find_job_boot.cr | 1 + 2 files changed, 20 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 51e0d74..570f066 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -134,6 +134,25 @@ class Sched @log.warn(e) end + def set_tbox_boot_wtmp(job : Job) + time = Time.local + booting_time = time.to_s("%Y-%m-%dT%H:%M:%S") + + runtime = (job["timeout"]? || job["runtime"]?).to_s + runtime = 1800 if runtime.empty? + + # reserve 300 seconds for system startup + deadline = (time + (runtime.to_i32 * 2 + 300).second).to_s("%Y-%m-%dT%H:%M:%S") + hash = { + "job_id" => job["id"], + "state" => "booting", + "booting_time" => booting_time, + "deadline" => deadline + } + + @es.update_tbox(job["testbox"], hash) + end + def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b7f3d75..f8d7caa 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -101,6 +101,7 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + set_tbox_boot_wtmp(job) else # for physical machines spawn { auto_submit_idle_job(host) } -- 2.23.0

2 1

[PATCH compass-ci] create-image: disable selinux
by Wang Chenglong 20 Jan '21

20 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..69a5ddc 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,17 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + # disable selinux + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/SELINUX=enforcing/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

2 2

[PATCH compass-ci 1/2] scheduler/elasticsearch_client.cr: support update all hash
by Wu Zhende 20 Jan '21

20 Jan '21

[Before] Only support update job_id/job_state/history [After] Support update any key-value. Because need save other new key-value in testbox. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index b17d7f8..c1663c9 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -3,6 +3,7 @@ require "yaml" require "json" +require "any_merge" require "elasticsearch-crystal/elasticsearch/api" require "./constants" require "../lib/job" @@ -89,21 +90,30 @@ class Elasticsearch::Client def update_tbox(testbox : String, wtmp_hash : Hash) query = {:index => "testbox", :type => "_doc", :id => testbox} - result = @client.get_source(query) if @client.exists(query) - history = result["history"].as_a? if result.is_a?(JSON::Any) + if @client.exists(query) + result = @client.get_source(query) + raise result unless result.is_a?(JSON::Any) + + history = result["history"].as_a? + result = result.as_h + else + result = wtmp_hash + end + history ||= [] of JSON::Any history << JSON.parse(wtmp_hash.to_json) + history = JSON.parse(history.to_json) + + body = { "history" => history} + body.any_merge!(result) + body.any_merge!(wtmp_hash) @client.create( { :index => "testbox", :type => "_doc", :id => testbox, - :body => { - :state => wtmp_hash["state"], - :job_id => wtmp_hash["job_id"], - :history => history - } + :body => body } ) end -- 2.23.0

1 0

Re: [PATCH lkp-tests 3/4] lib/job.sh: modify the generated file suffix
by Wu Fengguang 20 Jan '21

20 Jan '21

Why ignore? I just applied. On Wed, Jan 20, 2021 at 10:22:01AM +0800, Li Ping wrote: >please ignore this email. > >On Wed, Jan 20, 2021 at 10:20:44AM +0800, Li Ping wrote: >>[why] >>user extract $program result actually generating a YAML file. >>and compass-ci/sbin/result2stats have already support merge yaml file >> >>Signed-off-by: Li Ping <1477412247(a)qq.com> >>--- >> lib/job.sh | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >>diff --git a/lib/job.sh b/lib/job.sh >>index 03c49e0ad..dc65f68d8 100755 >>--- a/lib/job.sh >>+++ b/lib/job.sh >>@@ -318,5 +318,5 @@ run_test() >> run_target_stats() >> { >> local script_name=$1 >>- $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.json >>+ $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.yaml >> } >>-- >>2.23.0 >>

1 0

Re: [PATCH lkp-tests 4/4] sbin/submit: Uppercase the submit option "-l"
by Wu Fengguang 20 Jan '21

20 Jan '21

Avoid using --lftp Check my older reply. On Wed, Jan 20, 2021 at 10:20:45AM +0800, Li Ping wrote: >[why] >"-l" looks like minus one, be not recommended for use. > >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > sbin/submit | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > >diff --git a/sbin/submit b/sbin/submit >index 39cdcf576..51e85f33c 100755 >--- a/sbin/submit >+++ b/sbin/submit >@@ -47,7 +47,7 @@ options = OptionParser.new do |opts| > actions << 'connect' > end > >- opts.on('-l', '--lftp', 'lftp mirror job result') do >+ opts.on('-L', '--lftp', 'lftp mirror job result') do > actions << 'lftp_result' > end > >-- >2.23.0 >

1 0