- Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests 2/3] alter reset-tbox.yaml
by Zhang Dewan 20 Jan '21

20 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml index 4d43850f..216a3852 100644 --- a/jobs/reset-tbox.yaml +++ b/jobs/reset-tbox.yaml @@ -3,7 +3,7 @@ category: functional testbox: dc-8g os_mount: container -docker_image: 172.168.131.2:5001/reset-tbox +docker_image: reset-tbox:latest queue: reset-tbox reset-tbox: -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/3] reset machine testbox by job
by Zhang Dewan 20 Jan '21

20 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++ tests/reset-tbox | 54 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..4d43850f --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: 172.168.131.2:5001/reset-tbox +queue: reset-tbox + +reset-tbox: + host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..1ca17bfd --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,54 @@ +#!/bin/bash +# host_nameset + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +FileCheck() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +TboxCheck() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +ResetTbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +ResultCheck() +{ + if [ $? -ne 0 ] ; then + echo "Failed to reboot the testbox: $host_name" + fi + echo "done." +} + +main() +{ + FileCheck + TboxCheck + ResetTbox + ResultCheck +} +main + + + -- 2.23.0

1 0

[PATCH compass-ci] upload-libvirt/build: add upload-libvirt docker image
by cuiyili 20 Jan '21

20 Jan '21

Add upload-libvirt docker image in order to upload libvirt xml file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt/build | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100755 container/upload-libvirt/build diff --git a/container/upload-libvirt/build b/container/upload-libvirt/build new file mode 100755 index 0000000..bda6a3e --- /dev/null +++ b/container/upload-libvirt/build @@ -0,0 +1,7 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/openresty/openresty +# Copyright (C) 2016-2020 Eric D. Evan Wies +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t upload-libvirt . -- 2.23.0

1 0

[PATCH compass-ci] upload-libvirt/Dockerfile: add upload-libvirt docker image
by cuiyili 20 Jan '21

20 Jan '21

Add upload-libvirt docker image in order to upload libvirt xml file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt/build | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100755 container/upload-libvirt/build diff --git a/container/upload-libvirt/build b/container/upload-libvirt/build new file mode 100755 index 0000000..bda6a3e --- /dev/null +++ b/container/upload-libvirt/build @@ -0,0 +1,7 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/openresty/openresty +# Copyright (C) 2016-2020 Eric D. Evan Wies +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t upload-libvirt . -- 2.23.0

1 0

[PATCH compass-ci] upload-libvirt/Dockerfile: add upload-libvirt docker image
by cuiyili 20 Jan '21

20 Jan '21

Add upload-libvirt docker image in order to upload libvirt xml file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/upload-libvirt/Dockerfile diff --git a/container/upload-libvirt/Dockerfile b/container/upload-libvirt/Dockerfile new file mode 100644 index 0000000..db48bcb --- /dev/null +++ b/container/upload-libvirt/Dockerfile @@ -0,0 +1,14 @@ +# Origin: https://hub.docker.com/r/openresty/openresty +# Copyright (C) 2016-2020 Eric D. Evan Wies +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM openresty/openresty:alpine + +RUN apk add bash +RUN mkdir -p /var/log/nginx/ +RUN adduser lkp -u 1090 -g 1090 -D + +COPY ./openresty.sh /usr/local/bin/ + +CMD ["/usr/local/bin/openresty.sh"] -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: add parameter option -i for submit
by Wei Jihui 20 Jan '21

20 Jan '21

submit -i is for add other yamls content to job.yaml [usage] 1. submit -i 1.yaml -i 2.yaml job.yaml vars in job.yaml can overwrite 1.yaml and 2.yaml 2. submit -i 1.yaml job.yaml -i 2.yaml vars in job.yaml can only overwrite 1.yaml vars in 2.yaml can overwrite job.yaml and 1.yaml 3. submit -i 1.yaml job1.yaml -i 2.yaml job2.yaml -i 3.yaml vars in job1.yaml can only overwrite 1.yaml vars in job2.yaml can overwrite 1.yaml and 2.yaml vars in 2.yaml can overwrite job1.yaml and 1.yaml vars in 3.yaml can overwrite 1.yaml,job1.yaml,2.yaml,job2.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 17 +++++++++++++---- sbin/submit | 30 ++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 631ebfc9..640b25a5 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -215,6 +215,8 @@ class Job attr_accessor :overrides attr_accessor :defaults attr_accessor :cmdline_defaults + attr_accessor :default_yamls + attr_accessor :override_yamls def initialize(job = {}, defaults = {}, overrides = {}, cmdline_defaults={}) @job = job @@ -222,6 +224,8 @@ class Job @overrides = overrides # from command line @cmdline_defaults = cmdline_defaults # from command line @available_programs = {} + @default_yamls = [] + @override_yamls = [] end def source_file_symkey(file) @@ -254,8 +258,11 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.merge!(@overrides) - hash.merge!(fault_reproduction) if hash.has_key?("id") + + revise_hash(hash, load_include_yamls(@default_yamls), false) unless @default_yamls.empty? + revise_hash(hash, load_include_yamls(@override_yamls), true) unless @override_yamls.empty? + revise_hash(hash, @overrides, true) unless @overrides.empty? + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e @@ -286,8 +293,10 @@ class Job @jobfile = jobfile end - def fault_reproduction - wait = load_yaml("#{lkp_src}/jobs/fault_reproduction.yaml", {}) + def load_include_yamls(include_yamls) + include_hash = {} + include_hash[comment_to_symbol("#{include_yamls.join(' ')}")] = nil + include_hash.merge!(load_yaml_merge(include_yamls, {})) end def multi_args(hash) diff --git a/sbin/submit b/sbin/submit index 39cdcf57..e8b17648 100755 --- a/sbin/submit +++ b/sbin/submit @@ -14,6 +14,7 @@ opt_set_key_value = {} opt_cmdline_defaults = {} opt_output_dir = nil opt_auto_define_files = false +opt_include_yamls = [] opt_monitor = false opt_monitor_query = {} opt_my_queue = false @@ -43,6 +44,10 @@ options = OptionParser.new do |opts| opt_auto_define_files = true end + opts.on('-i include.yaml', '--include include.yaml', 'include other job yamls') do |yaml| + opt_include_yamls << yaml + end + opts.on('-c', '--connect', 'auto connect to the host') do actions << 'connect' end @@ -66,6 +71,7 @@ options = OptionParser.new do |opts| end +ARGV_CLONE = ARGV.clone options.parse!(ARGV) seen_job_yaml = false @@ -91,6 +97,19 @@ end job_ids = [] job_hash_list = [] +def prioritize_include_yaml(include_yamls, jobfile) + default_yamls, override_yamls = [], [] + jobfile_index = ARGV_CLONE.index(jobfile) + include_yamls.each do |yaml| + if ARGV_CLONE.index(yaml) < jobfile_index + default_yamls << yaml + else + override_yamls << yaml + end + end + return default_yamls, override_yamls +end + def find_jobfile(jobfile) return jobfile if File.exist?(jobfile) @@ -103,14 +122,25 @@ def find_jobfile(jobfile) exit 1 end +def find_jobfiles(jobfile_list) + search_jobfile_list = [] + jobfile_list.each do |jobfile| + search_jobfile_list << find_jobfile(jobfile) + end + return search_jobfile_list +end + submit_id = %x(uuidgen).chomp puts "submit_id=#{submit_id}" ARGV.each do |jobfile| + default_yamls, override_yamls = prioritize_include_yaml(opt_include_yamls, jobfile) jobfile = find_jobfile(jobfile) jobs = Job2sh.new jobs.cmdline_defaults = opt_cmdline_defaults jobs.overrides = opt_set_key_value + jobs.default_yamls = find_jobfiles(default_yamls) + jobs.override_yamls = find_jobfiles(override_yamls) jobs.load(jobfile, true) || next jobs[:expand_params] = true jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] -- 2.23.0

2 1

[PATCH lkp-tests 2/2] alter reset-tbox.yaml
by Zhang Dewan 20 Jan '21

20 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml index 4d43850f..216a3852 100644 --- a/jobs/reset-tbox.yaml +++ b/jobs/reset-tbox.yaml @@ -3,7 +3,7 @@ category: functional testbox: dc-8g os_mount: container -docker_image: 172.168.131.2:5001/reset-tbox +docker_image: reset-tbox:latest queue: reset-tbox reset-tbox: -- 2.23.0

3 4

[PATCH lkp-tests 1/2] reset machine testbox by job
by Zhang Dewan 20 Jan '21

20 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++ tests/reset-tbox | 54 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..4d43850f --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: 172.168.131.2:5001/reset-tbox +queue: reset-tbox + +reset-tbox: + host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..1ca17bfd --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,54 @@ +#!/bin/bash +# host_nameset + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +FileCheck() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +TboxCheck() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +ResetTbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +ResultCheck() +{ + if [ $? -ne 0 ] ; then + echo "Failed to reboot the testbox: $host_name" + fi + echo "done." +} + +main() +{ + FileCheck + TboxCheck + ResetTbox + ResultCheck +} +main + + + -- 2.23.0

3 3

[PATCH v1 compass-ci 4/4] providers/libvirt: add a libvirt provider
by Xiao Shenwei 20 Jan '21

20 Jan '21

[why] libvirt is a vm management tool, it based on a domain.xml file to run a vm, the xml each element will be parsed qemu-kvm parameter. so, we can configure xml to run vms of diff specifications Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/client.rb | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 providers/libvirt/client.rb diff --git a/providers/libvirt/client.rb b/providers/libvirt/client.rb new file mode 100644 index 0000000..d029a35 --- /dev/null +++ b/providers/libvirt/client.rb @@ -0,0 +1,23 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'lib/mylog' +require_relative 'lib/consumer' + +def main(hostname, queues) + puts "cat #{hostname}.log" + logger = Mylog.new("#{hostname}.log") + consumer = Consumer.new(hostname, queues).add_logger(logger) + response = consumer.request_job + + if response.nil? || response['job_id'].empty? + logger.warn('no job now') + puts '----------' + puts 'no job now' + puts '----------' + return + end + + consumer.close +end -- 2.23.0

1 0

[PATCH v1 compass-ci 1/4] providers/libvirt: basic class to save common methods
by Xiao Shenwei 20 Jan '21

20 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/lib/base.rb | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 providers/libvirt/lib/base.rb diff --git a/providers/libvirt/lib/base.rb b/providers/libvirt/lib/base.rb new file mode 100644 index 0000000..2e8a74d --- /dev/null +++ b/providers/libvirt/lib/base.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# save common variables and methods +class Base + attr_reader :logger + + def add_logger(logger) + @logger = logger + self + end +end -- 2.23.0

1 0

[PATCH v10 compass-ci] lib/es_jobs.rb: optimize es-jobs output
by Lu Kaiyi 20 Jan '21

20 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archlinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archlinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.summary.any_error=1: 24 kvcount.summary.any_fail=1: 25 kvcount.summary.any_stderr=1: 44 kvcount.summary.any_warning=1: 23 kvcount.summary.success=1: 1 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 ... raw.id.[job_state=failed]: ["crystal.122405","crystal.122406","crystal.122331","crystal.122365","crystal.122404","crystal.122438","crystal.122607","crystal.122370","crystal.122375","crystal.122410","crystal.122900","crystal.122906","crystal.122271","crystal.122451","crystal.122835","crystal.122834","crystal.122232","crystal.122403","crystal.122453","crystal.122886"] ... raw.id.[summary.any_error=1]: ["crystal.122436","crystal.122476","crystal.122803","crystal.122883","crystal.122910","crystal.122438","crystal.122452","crystal.122895","crystal.122446","crystal.122685","crystal.122687","crystal.122833","crystal.122451","crystal.122690","crystal.122719","crystal.122882","crystal.122890","crystal.122453","crystal.122443","crystal.122435","crystal.122432","crystal.122688","crystal.122689","crystal.122886"] ... sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0,null,null,null,null,null,null,null,null,524288.0,524288.0,null,null,null,null,null,516608.0,null,1048576.0,null,null,null,null,null,null,null,524288.0,null,null,null,null,2104832.0,1572864.0,null,null,2097152.0,null,null,null,null,null,null,null,null,null] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... max.stats.sched_debug.cfs_rq:/.load.stddev: 2104832.0 max.stats.softirqs.CPU1.NET_RX: 6 max.stats.slabinfo.kmalloc-512.active_objs: 1616 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 201 +++++++++++++++++++++++++++---------------------- 1 file changed, 111 insertions(+), 90 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..af0a1a9 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -3,137 +3,158 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo summary.success + summary.any_fail summary.any_error summary.any_stderr summary.any_warning].freeze + +require 'json' require "#{LKP_SRC}/lib/stats" require_relative './es_query' # deal jobs search from es class ESJobs def initialize(es_query, my_refine = [], fields = [], stats_filter = []) - @es_query = es_query - @es = ESQuery.new(ES_HOST, ES_PORT) + @jobs = query_jobs_from_es(es_query) @refine = my_refine @fields = fields @stats_filter = stats_filter - @stats_filter_result = {} @refine_jobs = [] - @jobs = {} - @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' - } - set_defaults - deal_jobs + set_jobs_summary end - def set_defaults - query_result = @es.multi_field_query(@es_query) - query_result['hits']['hits'].each do |job| - @jobs[job['_id']] = job['_source'] - end - - @stats = { - 'stats.count' => Hash.new(0), - 'stats.sum' => Hash.new(0), - 'stats.avg' => Hash.new(0) - } - @result = {} - @fields.each do |field| - @result[field] = [] - end + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs end - def add_result_fields(job, level) - return unless @refine.include?(level) || @refine.include?(-1) + def set_job_summary(stats, job) + summary_result = '' + if stats.keys.any? { |stat| stat.match(/warn/i) } + job['summary.any_warning'] = 1 + summary_result = 'warning' + end + if stats.keys.any? { |stat| stat.match(/stderr/i) } + job['summary.any_stderr'] = 1 + summary_result = 'stderr' + end + if stats.keys.any? { |stat| stat.match(/error|nr_fail=0/i) } + job['summary.any_error'] = 1 + summary_result = 'error' + end + if stats.keys.any? { |stat| stat.match(/fail/i) } && + stats.keys.all? { |stat| !stat.match(/nr_fail=0/i) } + job['summary.any_fail'] = 1 + summary_result = 'fail' + end + return unless summary_result.empty? - @refine_jobs << job['id'] - @fields.each do |field| - value = job[field] - if value - value = job['id'] + '.' + value if field == 'job_state' - @result[field] << value - end + job['summary.success'] = 1 + end - next unless job['stats'] + # set jobs summary fields information in place + def set_jobs_summary + @jobs.each do |job| + stats = job['stats'] + next unless stats - @result[field] << job['stats'][field] if job['stats'][field] + set_job_summary(stats, job) end end - def deal_jobs - stats_count = Hash.new(0) - stats_jobs = {} + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats - @jobs.each do |job_id, job| - level = deal_stats(job) - add_result_fields(job, level) - - stat_key = @stats_level[level] - stat_jobs_key = stat_key + '_jobs' - - stats_count[stat_key] += 1 - stats_jobs[stat_jobs_key] ||= [] - stats_jobs[stat_jobs_key] << job_id + metrics.concat(stats.keys) end + metrics.uniq! + end - @stats['stats.count'].merge!(stats_count) - @stats['stats.count'].merge!(stats_jobs) + def initialize_result_hash(metrics) + result = { + 'kvcount' => {}, + 'raw.id' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} + } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result end - def deal_stats(job, level = 0) - return 1 unless job['stats'] + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } - job['stats'].each do |key, value| - match_stats_filter(key, value, job['id']) - calculate_stat(key, value) - level = get_stat_level(key, level) + stats.each do |key, value| + result['raw.stats'][key] << value end - return level end - def match_stats_filter(key, value, job_id) - @stats_filter.each do |filter| - next unless key.include?(filter) + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - key = job_id + '.' + key - @stats_filter_result[key] = value - - break + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + result['raw.id']["[#{keyword}=#{job[keyword]}]"] ||= [] + result['raw.id']["[#{keyword}=#{job[keyword]}]"] << job['id'] end end - def calculate_stat(key, value) - if function_stat?(key) - return unless @fields.include?('stats.sum') - - @stats['stats.sum'][key] += value - else - return unless @fields.include?('stats.avg') - - @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end end end - def get_stat_level(stat, level) - return level if level >= 3 - return 3 if stat.match(/error|fail/i) - return 2 if stat.match(/warn/i) + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end - return 0 + stats_count(result) + result end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def output_yaml(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + if value.is_a? Hash + output_yaml(prefix_key, value) + else + puts "#{prefix_key}: #{value.to_json}" + end end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def output + @result = query_jobs_state(@jobs) + @result['kvcount'] = @result['kvcount'].sort.to_h + @result['raw.id'] = @result['raw.id'].sort.to_h + output_yaml('', @result) end end -- 2.23.0

2 1

[PATCH v2 compass-ci] create-image: disable selinux
by Wang Chenglong 20 Jan '21

20 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..4056ff1 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,17 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + # disable selinux + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/^.*SELINUX=enforcing/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

2 2

[PATCH compass-ci 2/2] service/scheduler: save testbox info in es
by Wu Zhende 20 Jan '21

20 Jan '21

When a testbox get a job, record its information in es. The deadline field indicates the time when the job executed by the testbox times out. It's a sign that other measures are needed to intervene. Can use curl -XGET http://localhost:9200/testbox/_doc/_search -d '{"query":{"range":{"deadline":{"lt":"now"}}}}' to find all timeout testboxs and jobs Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 19 +++++++++++++++++++ src/scheduler/find_job_boot.cr | 1 + 2 files changed, 20 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 51e0d74..570f066 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -134,6 +134,25 @@ class Sched @log.warn(e) end + def set_tbox_boot_wtmp(job : Job) + time = Time.local + booting_time = time.to_s("%Y-%m-%dT%H:%M:%S") + + runtime = (job["timeout"]? || job["runtime"]?).to_s + runtime = 1800 if runtime.empty? + + # reserve 300 seconds for system startup + deadline = (time + (runtime.to_i32 * 2 + 300).second).to_s("%Y-%m-%dT%H:%M:%S") + hash = { + "job_id" => job["id"], + "state" => "booting", + "booting_time" => booting_time, + "deadline" => deadline + } + + @es.update_tbox(job["testbox"], hash) + end + def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b7f3d75..f8d7caa 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -101,6 +101,7 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + set_tbox_boot_wtmp(job) else # for physical machines spawn { auto_submit_idle_job(host) } -- 2.23.0

2 1

[PATCH compass-ci] create-image: disable selinux
by Wang Chenglong 20 Jan '21

20 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..69a5ddc 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,17 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + # disable selinux + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/SELINUX=enforcing/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

2 2

[PATCH compass-ci 1/2] scheduler/elasticsearch_client.cr: support update all hash
by Wu Zhende 20 Jan '21

20 Jan '21

[Before] Only support update job_id/job_state/history [After] Support update any key-value. Because need save other new key-value in testbox. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index b17d7f8..c1663c9 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -3,6 +3,7 @@ require "yaml" require "json" +require "any_merge" require "elasticsearch-crystal/elasticsearch/api" require "./constants" require "../lib/job" @@ -89,21 +90,30 @@ class Elasticsearch::Client def update_tbox(testbox : String, wtmp_hash : Hash) query = {:index => "testbox", :type => "_doc", :id => testbox} - result = @client.get_source(query) if @client.exists(query) - history = result["history"].as_a? if result.is_a?(JSON::Any) + if @client.exists(query) + result = @client.get_source(query) + raise result unless result.is_a?(JSON::Any) + + history = result["history"].as_a? + result = result.as_h + else + result = wtmp_hash + end + history ||= [] of JSON::Any history << JSON.parse(wtmp_hash.to_json) + history = JSON.parse(history.to_json) + + body = { "history" => history} + body.any_merge!(result) + body.any_merge!(wtmp_hash) @client.create( { :index => "testbox", :type => "_doc", :id => testbox, - :body => { - :state => wtmp_hash["state"], - :job_id => wtmp_hash["job_id"], - :history => history - } + :body => body } ) end -- 2.23.0

1 0

Re: [PATCH lkp-tests 3/4] lib/job.sh: modify the generated file suffix
by Wu Fengguang 20 Jan '21

20 Jan '21

Why ignore? I just applied. On Wed, Jan 20, 2021 at 10:22:01AM +0800, Li Ping wrote: >please ignore this email. > >On Wed, Jan 20, 2021 at 10:20:44AM +0800, Li Ping wrote: >>[why] >>user extract $program result actually generating a YAML file. >>and compass-ci/sbin/result2stats have already support merge yaml file >> >>Signed-off-by: Li Ping <1477412247(a)qq.com> >>--- >> lib/job.sh | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >>diff --git a/lib/job.sh b/lib/job.sh >>index 03c49e0ad..dc65f68d8 100755 >>--- a/lib/job.sh >>+++ b/lib/job.sh >>@@ -318,5 +318,5 @@ run_test() >> run_target_stats() >> { >> local script_name=$1 >>- $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.json >>+ $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.yaml >> } >>-- >>2.23.0 >>

1 0

Re: [PATCH lkp-tests 4/4] sbin/submit: Uppercase the submit option "-l"
by Wu Fengguang 20 Jan '21

20 Jan '21

Avoid using --lftp Check my older reply. On Wed, Jan 20, 2021 at 10:20:45AM +0800, Li Ping wrote: >[why] >"-l" looks like minus one, be not recommended for use. > >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > sbin/submit | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > >diff --git a/sbin/submit b/sbin/submit >index 39cdcf576..51e85f33c 100755 >--- a/sbin/submit >+++ b/sbin/submit >@@ -47,7 +47,7 @@ options = OptionParser.new do |opts| > actions << 'connect' > end > >- opts.on('-l', '--lftp', 'lftp mirror job result') do >+ opts.on('-L', '--lftp', 'lftp mirror job result') do > actions << 'lftp_result' > end > >-- >2.23.0 >

1 0

[PATCH lkp-tests] pkg: add PKGBUILD for openeuler docker test
by Bai Jing 20 Jan '21

20 Jan '21

[why] The source file is missing in the openEuler image, So we need to add source files for it. Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/openeuler_docker/PKGBUILD | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 pkg/openeuler_docker/PKGBUILD diff --git a/pkg/openeuler_docker/PKGBUILD b/pkg/openeuler_docker/PKGBUILD new file mode 100644 index 000000000..8076350a9 --- /dev/null +++ b/pkg/openeuler_docker/PKGBUILD @@ -0,0 +1,28 @@ +pkgname=openeuler_docker +pkgver=1.0 +pkgrel=1 +pkgdesc="openeuler docker image" +arch=('i386' 'x86_64') +url="https://repo.openeuler.org/openEuler-20.03-LTS/docker_img" +license=('GPL') +source=("https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler…") +md5sums=('6aa300910c9f9040b725939fb80e8121') + +handle_docker_image() { + docker load < openEuler-docker.aarch64.tar.xz + docker run -d --name my-openeuler openeuler-20.03-lts:latest + container_id=$(docker ps -a | grep my-openeuler | awk -F ' ' '{print $1}') + docker cp /etc/yum.repos.d/openEuler.repo $container_id:/etc/yum.repos.d/ + docker commit $container_id openeuler-20.03-lts:v1 + docker save -o openeuler-20.03-lts-v1.tar openeuler-20.03-lts:v1 + docker stop $container_id && docker rm $container_id +} + +build() { + handle_docker_image +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/${pkgname}" + cp "$srcdir/openeuler-20.03-lts-v1.tar" "$pkgdir/lkp/benchmarks/${pkgname}" +} -- 2.23.0

3 4

[PATCH v2 compass-ci] doc: add a doc about catalogs under /srv/git
by Li Yuanchao 20 Jan '21

20 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/about_srv_git.md | 179 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 179 insertions(+) create mode 100644 doc/about_srv_git.md diff --git a/doc/about_srv_git.md b/doc/about_srv_git.md new file mode 100644 index 0000000..6f75b37 --- /dev/null +++ b/doc/about_srv_git.md @@ -0,0 +1,179 @@ + +# 挂载了`/srv/git`目录的容器有： + +``` +auto-submit： + 主要功能为自动提交job。通过消息队列rabbitmq接收git-mirror发送的消息，解析消息内容后提交job。 + +git-daemon： + 作为局域网的git服务端，为/srv/git目录下的仓库提供git clone/git push服务，而不需要原本代码仓的账号密码。 + 通过 "git clone git://localhost/$project" 克隆的仓库即是使用了git-daemon服务。 + +git-mirror： + 监控用户代码仓，对新加的代码仓进行clone，已有的保持更新。 + +remote-git： + 在不能直接访问/srv/git目录的情况下，可以通过curl命令执行需要在/srv/git目录下运行的git命令。 +``` + + +# /srv/git目录下的各目录归属如下： + + +字母a-z是git-mirror监控的用户代码仓的镜像存储目录。 +``` +drwxrwsr-x. 55 lkp committer 4.0K 2020-11-20 03:54 u +drwxrwsr-x. 78 lkp committer 4.0K 2020-12-04 11:24 v +drwxrwsr-x. 75 lkp committer 4.0K 2020-12-04 11:25 i +drwxrwsr-x. 986 lkp committer 36K 2020-12-04 11:37 d +drwxrwsr-x. 1612 lkp committer 60K 2020-12-05 03:50 g +drwxrwsr-x. 827 lkp committer 28K 2020-12-05 04:00 b +drwxrwsr-x. 66 lkp committer 4.0K 2020-12-05 04:05 w +drwxrwsr-x. 32 lkp committer 4.0K 2020-12-05 04:17 y +drwxrwsr-x. 706 lkp committer 28K 2020-12-05 05:49 e +drwxrwsr-x. 837 lkp committer 28K 2020-12-08 22:05 f +drwxrwsr-x. 679 lkp committer 24K 2020-12-10 18:18 k +drwxrwsr-x. 948 lkp committer 36K 2020-12-11 02:31 h +drwxrwsr-x. 384 lkp committer 12K 2020-12-11 03:12 j +drwxrwsr-x. 927 lkp committer 36K 2020-12-15 03:11 n +drwxrwsr-x. 1757 lkp committer 56K 2020-12-15 11:09 l +drwxrwsr-x. 54 lkp committer 4.0K 2020-12-15 14:29 t +drwxrwsr-x. 1018 lkp committer 36K 2020-12-15 15:57 a +drwxrwsr-x. 1344 lkp committer 40K 2020-12-15 22:28 m +drwxrwsr-x. 809 lkp committer 28K 2020-12-16 05:21 o +drwxrwsr-x. 1342 lkp committer 36K 2020-12-21 19:19 c +drwxrwsr-x. 5800 lkp committer 212K 2020-12-22 03:08 p +drwxrwsr-x. 395 lkp committer 12K 2020-12-22 22:16 q +drwxrwsr-x. 1632 lkp committer 64K 2020-12-28 21:25 r +drwxrwsr-x. 445 lkp committer 20K 2020-12-28 23:16 x +drwxrwsr-x. 1796 lkp committer 56K 2021-01-05 09:20 s +drwxrwsr-x. 39 lkp committer 4.0K 2021-01-08 10:48 z +``` + +下面这些用户为lkp:committer，名字是网址的文件夹，是git-mirror监控仓库的子模块。 +``` +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-16 10:57 git.code.sf.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-16 11:22 code.reversed.top +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-20 03:56 source.ffmpeg.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-21 02:46 git.kernel.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-22 04:05 gitlab.univ-nantes.fr +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-02 18:14 git.musl-libc.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 05:56 gitlab.dkrz.de +drwxrwsr-x. 8 lkp committer 4.0K 2020-12-05 06:17 git.sv.gnu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:17 git.ageinghacker.net +drwxrwsr-x. 11 lkp committer 4.0K 2020-12-05 06:39 git.qemu-project.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:44 git.sigxcpu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 06:56 git.osmocom.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-05 07:38 gitlab.linphone.org +drwxrwsr-x. 8 lkp committer 4.0K 2020-12-05 07:52 gitlab.gnome.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-10 14:40 webrtc.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-11 03:33 www.kismetwireless.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-11 03:34 src.fedoraproject.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-16 05:53 git.tuxfamily.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:14 chriso(a)github.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:55 nedko.arnaudov.name +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 17:59 git.gnome.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 18:01 android.googlesource.com +drwxrwsr-x. 5 lkp committer 4.0K 2020-12-17 18:40 gopkg.in +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 19:53 git.cryptomilk.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 20:05 git.dpkg.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-17 20:09 kernel.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-19 14:15 aur(a)archlinux.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-21 05:57 boringssl.googlesource.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 04:44 hub.spigotmc.org +drwxrwsr-x. 14 lkp committer 4.0K 2020-12-22 04:49 repo.or.cz +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 04:52 luajit.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:02 git.assembla.com +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-22 05:07 git.purrdata.net +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:09 brettc(a)github.com +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 05:11 git.teknik.io +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-22 14:40 code.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2020-12-23 01:50 git.qemu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-23 01:51 git.merproject.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-23 01:52 code.qt.io +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-23 01:57 git.tartarus.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-27 22:17 gist.github.com +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-29 00:28 invent.kde.org +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-29 00:55 git.sv.nongnu.org +drwxrwsr-x. 6 lkp committer 4.0K 2020-12-29 02:57 git.savannah.gnu.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-29 02:57 git.xfce.org +drwxrwsr-x. 7 lkp committer 4.0K 2020-12-29 02:58 anongit.freedesktop.org +drwxrwsr-x. 3 lkp committer 4.0K 2020-12-29 02:59 git.ejiek.com +drwxrwsr-x. 32 lkp committer 4.0K 2020-12-29 03:00 gitlab.com +drwxrwsr-x. 5 lkp committer 4.0K 2020-12-30 22:00 git.torproject.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-02 05:58 skia.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-03 05:57 aur(a)aur4.archlinux.org +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-05 10:56 gitlab.freedesktop.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-05 10:59 gitlab.xiph.org +drwxrwsr-x. 3 lkp committer 4.0K 2021-01-05 10:59 git.savannah.nongnu.org +drwxrwsr-x. 4 lkp committer 4.0K 2021-01-05 11:01 git.tukaani.org +drwxrwsr-x. 13 lkp committer 4.0K 2021-01-05 11:04 aur.archlinux.org +drwxrwsr-x. 6 lkp committer 4.0K 2021-01-05 11:07 bitbucket.org +drwxrwsr-x. 1664 lkp committer 40K 2021-01-07 09:08 github.com +drwxrwsr-x. 5 lkp committer 4.0K 2021-01-08 05:03 go.googlesource.com +drwxrwsr-x. 7 lkp committer 4.0K 2021-01-08 05:24 chromium.googlesource.com +drwxrwsr-x. 14 lkp committer 4.0K 2021-01-08 06:00 aur(a)aur.archlinux.org +``` + +下面这些目录是原本由git-mirror监控，但是由于目录结构变更而不再监控的仓库。 +``` +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:30 coccinelle +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:45 oec-hardware +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:45 libbpf +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:46 qemu +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:51 smatch +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-17 11:51 gtk +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-21 11:02 pixz +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-31 11:26 sparse +drwxrwsr-x. 3 lkp committer 4.0K 2020-08-31 11:27 pahole +drwxrwsr-x. 3 lkp committer 4.0K 2020-09-22 15:10 upstream-repos +drwxrwsr-x. 3 lkp committer 4.0K 2020-11-09 19:31 linux +drwxrwsr-x. 24 lkp committer 4.0K 2020-11-10 16:40 submodules +drwxrwsr-x. 4 lkp committer 4.0K 2020-12-02 10:00 compass-ci +``` + +下面的目录是构建的pkgbuild文件仓库和make源码仓库，由构建组维护。 +``` +drwxrwsr-x. 43 lkp team 200K 2020-11-11 15:17 archlinux +drwxrwsr-x. 82 root team 4.0K 2020-12-15 18:12 pkg +drwxrwsr-x. 13 wfg team 4.0K 2020-12-16 12:55 make +``` + +存放rpm包的spec文件，由用户手动创建。 +``` +drwxrwsr-x. 3 root committer 4.0K 2020-12-08 11:15 rpmbuild-pkg +``` + +下面这些是以前调试的时候由用户手动clone的，目前应该没有维护了。 +``` +drwxrwsr-x. 3 zhy committer 4.0K 2020-06-16 11:25 mysql-server +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:28 atomic +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:28 spinlock +drwxrwsr-x. 3 zhy committer 4.0K 2020-07-10 14:29 mutex +drwxrwsr-x. 4 zhy committer 4.0K 2020-08-24 11:52 avx2neon +drwxrwsr-x. 5 liping committer 4.0K 2020-08-24 11:52 AvxToNeon +``` + +下面这个是服务器上各个仓库的源端（compass-ci/lkp-tests/.../upstream-repos）。 +``` +drwxrwsr-x. 3 wfg committer 4.0K 2020-09-14 15:26 gitee.com +``` + +一些测试工具，现在不用了。 +``` +drwxrwxrwx. 3 weijihui weijihui 4.0K 2020-07-27 15:26 intrintesttool +``` + +ansible软件包自动部署的一些样例。 +``` +drwxrwsr-x. 17 root committer 4.0K 2020-12-29 10:33 ansible-examples +``` + +一些比较久远的仓库，似乎不再维护了。 +``` +drwxrwsr-x. 2 root committer 16K 2020-03-27 10:45 lost+found +drwxrwsr-x. 2 wfg committer 4.0K 2020-07-24 16:58 gzip +drwxrwsr-x. 5 wfg committer 4.0K 2020-08-01 11:15 cci +drwxrwsr-x. 5 wfg committer 4.0K 2020-08-04 16:27 lkp-tests +drwxrwsr-x. 4 wfg committer 4.0K 2020-09-19 14:15 crystal-ci +``` -- 2.23.0

2 1

[PATCH lkp-tests] bin/lkp-setup-rootfs: refactor chmod_root_ssh()
by Xu Xijian 20 Jan '21

20 Jan '21

Add blank line after return cmd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- bin/lkp-setup-rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cca1b53be..2c48521f7 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -247,6 +247,7 @@ chmod_root_ssh() { [ -d "/root/.ssh" ] || return [ -f "/root/.ssh/id_rsa" ] || return + chmod 700 /root/.ssh/ chmod 600 /root/.ssh/id_rsa } -- 2.23.0

1 0

[PATCH v2 lkp-tests] reset machine testbox by job
by Zhang Dewan 20 Jan '21

20 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- jobs/reset-tbox.yaml | 10 ++++++++ tests/reset-tbox | 54 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 64 insertions(+) create mode 100644 jobs/reset-tbox.yaml create mode 100755 tests/reset-tbox diff --git a/jobs/reset-tbox.yaml b/jobs/reset-tbox.yaml new file mode 100644 index 00000000..4d43850f --- /dev/null +++ b/jobs/reset-tbox.yaml @@ -0,0 +1,10 @@ +suite: reset-tbox +category: functional + +testbox: dc-8g +os_mount: container +docker_image: 172.168.131.2:5001/reset-tbox +queue: reset-tbox + +reset-tbox: + host_name: diff --git a/tests/reset-tbox b/tests/reset-tbox new file mode 100755 index 00000000..1ca17bfd --- /dev/null +++ b/tests/reset-tbox @@ -0,0 +1,54 @@ +#!/bin/bash +# host_nameset + +account_file=/etc/lab/.account.info +hosts_file=/etc/lab/hosts_file +command_file=/usr/bin/ipmitool + +FileCheck() +{ + for i in $account_file $hosts_file $command_file + do + [ -f $i ] || { + echo "File not found: $i" + exit 0 + } + done + source $account_file +} + +TboxCheck() +{ + ibmc_ip=$(grep -w $host_name $hosts_file | awk '{print $1}') + + [[ "$ibmc_ip" =~ ^[0-9]+.* ]] || { + echo "Testbox not found: $host_name" + exit 1 + } +} + +ResetTbox() +{ + echo "reboot the tesetbox..." + ipmitool -I lanplus -H $ibmc_ip -U $iBMC_user -P $iBMC_passwd power reset +} + +ResultCheck() +{ + if [ $? -ne 0 ] ; then + echo "Failed to reboot the testbox: $host_name" + fi + echo "done." +} + +main() +{ + FileCheck + TboxCheck + ResetTbox + ResultCheck +} +main + + + -- 2.23.0

3 4

[PATCH v3 lkp-tests] tests: add ansible_tests script
by Zhang Yale 20 Jan '21

20 Jan '21

[ideology] 1. This script can test hundreds of the ansible roles in upstream git repo. 2. ansible roles are supported for the EL platform which has 11257 results in ansible-galaxy official website. 3. For testing ansible git repo, those cannot be tested in a unified way, I mainly test ansible roles git repo, and these roles repo will be more and more. [why] Command "ansible-playbook site.yml" can run in localhost, and only need to create an entry YAML file in which the value of hosts is localhost, for example "site.yml", created in the directory at the same level as the roles directory, which can invoke roles in it. And it dose not need to create a host file in $ansible_path. ``` site.yml --- - hosts: localhost roles: - roles: ansible-role-nginx ``` ansible-role-nginx which is an role in ~/.ansible/roles, can install from galaxy or git clone from github. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/ansible_test | 92 ++++++++++++++++++++++++++-------------------- 1 file changed, 53 insertions(+), 39 deletions(-) diff --git a/tests/ansible_test b/tests/ansible_test index 09e5ca9e..703f5c55 100755 --- a/tests/ansible_test +++ b/tests/ansible_test @@ -1,59 +1,73 @@ #!/bin/bash -# username -# rolename -# ansible_host -# ansible_user -# ansible_python_interpreter - -[ -n "$username" ] || echo "username is empty" -[ -n "$rolename" ] || echo "rolename is empty" -[ -n "$ansible_host" ] || $ansible_host=localhost -[ -n "$ansible_user" ] || $ansible_user=root -[ -n "$ansible_python_interpreter" ] || $ansible_python_interpreter=/user/bin/python3 - -build_mutual_trust() +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() { - echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa - cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys - ssh -o "StrictHostKeyChecking=no" -o "ExitOnForwardFailure=yes" -o "passwordAuthentication=no" root@$ansible__host 'chmod 600 ~/.ssh/authorized_keys' + ansible_host=localhost + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible } install_ansible() { - pip3 install ansible - ansible localhost -m ping - #ansible all -m authorized_key -a "user=root key='{{ lookup('file', '/root/.ssh/id_rsa.pub') }}' path=/root/.ssh/authorized_keys manage_dir=no" --ask-pass -c paramiko - [ $? -eq 0 ] && echo "Ansible is successfully deployed!" + [[ "$os" == "openeuler" ]] && pip3 install ansible + [[ "$os" == "centos" ]] && yum install -y ansible + [[ "$os" == "debian" ]] && apt-get install -y ansible + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] && { + echo "Ansible is successfully deployed!" + } + } } download_ansible_role() { - ansible-galaxy install $username.$rolename - [ $? -eq 0 ] && echo "Please check whether your rolename is correct!" + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone https://github.com/$ansible_repo.git || exit + # git clone git://$GIT_SERVER/$ansible_repo.git || exit + # ansible-galaxy install $username.$rolename + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + fi } ansible_playbook_run() { - ansible_path=/root/.ansible - [ -d "$ansible_path" ] || mkdir -p /root/.ansible - cat <<-EOF > "$ansible_path"/hosts - $ansible_host ansible_user=$ansible_user ansible_python_interpreter=$ansible_python_interpreter - EOF - [ $? -eq 0 ] && echo "Hostfile has been generated!" - - cat <<-EOF > "$ansible_path"/$rolename.yml + cat <<-EOF > "$ansible_path"/site.yml - hosts: $ansible_host - remote_user: $ansible_user roles: - - role: $username.$rolename + - role: $benchmark EOF - [ $? -eq 0 ] && echo "Playbook has been generated!" + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml - ansible-playbook "$ansible_path"/$rolename.yml - [ $? -eq 0 ] && echo "Your playbook is running successfully!" + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "playbook.run.fail: 1" && exit 1 +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run } -#build_mutual_trust -install_ansible -download_ansible_role -ansible_playbook_run +main -- 2.23.0

1 1

[PATCH compass-ci] sparrow: register default accounts for services in sparrow
by Cao Xueliang 20 Jan '21

20 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 10 ++++++++++ sparrow/5-build/register-accounts | 14 ++++++++++++++ sparrow/install-tiny | 1 + 3 files changed, 25 insertions(+) create mode 100755 sparrow/5-build/register-accounts diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 4bafa80..2ba9662 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -60,6 +60,16 @@ export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +mkdir -p /etc/compass-ci/register +cat > /etc/compass-ci/register/register.yaml <<'EOF' +delimiter: + my_email: delimiter@localhost + my_name: delimiter +auto-submit: + my_email: auto-submit@localhost + my_name: auto-submit +EOF + source /etc/os-release path=$(dirname ${BASH_SOURCE[0]})/os/${ID} diff --git a/sparrow/5-build/register-accounts b/sparrow/5-build/register-accounts new file mode 100755 index 0000000..41126f9 --- /dev/null +++ b/sparrow/5-build/register-accounts @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['CCI_SRC']}/lib/es_client" +require_relative "#{ENV['CCI_SRC']}/lib/build_my_info_client" + +content = YAML.load_file('/etc/compass-ci/register/register.yaml') +content.each do |key, value| + build_my_info = BuildMyInfo.new(value['my_email'], value['my_name'], ENV['lab']) + build_my_info.store_account_info +end diff --git a/sparrow/install-tiny b/sparrow/install-tiny index a291549..eb9b3cb 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -19,4 +19,5 @@ cd $CCI_SRC/sparrow || exit . /etc/profile.d/compass.sh 4-docker/buildall 5-build/ipxe +5-build/register-accounts 7-systemd/systemd-setup -- 2.23.0

2 2

[PATCH lkp-tests] adaptation/debian: add pgbench dependency
by Hu Xuejiao 20 Jan '21

20 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/centos | 1 + distro/adaptation/debian | 1 + 2 files changed, 2 insertions(+) diff --git a/distro/adaptation/centos b/distro/adaptation/centos index 2cfe3877..8fbae666 100644 --- a/distro/adaptation/centos +++ b/distro/adaptation/centos @@ -335,3 +335,4 @@ liboath-dev: liboath-devel lsb-release: redhat-lsb linux-base: openssh-client: openssh-clients +postgresql-server: postgresql diff --git a/distro/adaptation/debian b/distro/adaptation/debian index 90260f1d..15706cf6 100644 --- a/distro/adaptation/debian +++ b/distro/adaptation/debian @@ -21,3 +21,4 @@ libperl5: libperl5.24 perl-modules-5: perl-modules-5.24 libgcc-dev: libgcc-6-dev openjdk: openjdk-8-jdk +postgresql-server-dev-all: postgresql -- 2.23.0

1 0

[PATCH compass-ci 1/2] sparrow: register accounts in sparrow
by Cao Xueliang 20 Jan '21

20 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 10 ++++++++++ sparrow/5-build/register-accounts | 14 ++++++++++++++ 2 files changed, 24 insertions(+) create mode 100755 sparrow/5-build/register-accounts diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 4bafa80..2ba9662 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -60,6 +60,16 @@ export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +mkdir -p /etc/compass-ci/register +cat > /etc/compass-ci/register/register.yaml <<'EOF' +delimiter: + my_email: delimiter@localhost + my_name: delimiter +auto-submit: + my_email: auto-submit@localhost + my_name: auto-submit +EOF + source /etc/os-release path=$(dirname ${BASH_SOURCE[0]})/os/${ID} diff --git a/sparrow/5-build/register-accounts b/sparrow/5-build/register-accounts new file mode 100755 index 0000000..41126f9 --- /dev/null +++ b/sparrow/5-build/register-accounts @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require_relative "#{ENV['CCI_SRC']}/lib/es_client" +require_relative "#{ENV['CCI_SRC']}/lib/build_my_info_client" + +content = YAML.load_file('/etc/compass-ci/register/register.yaml') +content.each do |key, value| + build_my_info = BuildMyInfo.new(value['my_email'], value['my_name'], ENV['lab']) + build_my_info.store_account_info +end -- 2.23.0

1 1