- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 18 Jan '21

18 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: [ -n "$initrds" ] || exit after: [ -n "$initrds" ] && cat $initrds > initrd || exit [why] 1. before execute cat command, check wether initrds is null. 2. initrd is a brief file name, it can be used directly, so remove $initrd, just us initrd. 3. set_initrd() is unnecessary, so remove. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: default queue for libvirt
by Ren Wen 18 Jan '21

18 Jan '21

testbox : vt-2p8g -> tbox_group : vt-2p8g -> default queue : vt-2p8g.<arch> Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 03ca5af..7d08610 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -310,7 +310,7 @@ class Job # set default value self["queue"] = tbox_group - if tbox_group.to_s.starts_with?(/(vm|dc)-/) + if tbox_group.to_s.starts_with?(/(vm|dc|vt)-/) self["queue"] = "#{tbox_group}.#{arch}" end end -- 2.23.0

1 0

[PATCH compass-ci] scheduler: default queue for libvirt type
by Ren Wen 18 Jan '21

18 Jan '21

testbox : libvirt-2p8g -> default queue : libvirt-2p8g.<arch> Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1d25236..5667842 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -306,7 +306,7 @@ class Job # set default value self["queue"] = tbox_group - if tbox_group.to_s.starts_with?(/(vm|dc)-/) + if tbox_group.to_s.starts_with?(/(vm|dc|libvirt)-/) self["queue"] = "#{tbox_group}.#{arch}" end end -- 2.23.0

2 2

[PATCH v2 lkp-tests] hosts: add new host file for libvirt
by Xiao Shenwei 18 Jan '21

18 Jan '21

[why] libvirt use a xml file to start vm <os> <kernel>xxx</kernel> <initrd>xxx</initrd> <cmdline>xxx</cmdline> </os> so, we can submit job use os_mount=initramfs to get kernel and initrd absolute path. Minimum memory limit: 16G Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/vt-2p16g | 5 +++++ hosts/vt-2p32g | 5 +++++ 2 files changed, 10 insertions(+) create mode 100644 hosts/vt-2p16g create mode 100644 hosts/vt-2p32g diff --git a/hosts/vt-2p16g b/hosts/vt-2p16g new file mode 100644 index 000000000..aecbb7948 --- /dev/null +++ b/hosts/vt-2p16g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 16G diff --git a/hosts/vt-2p32g b/hosts/vt-2p32g new file mode 100644 index 000000000..aa7c8c065 --- /dev/null +++ b/hosts/vt-2p32g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 32G -- 2.23.0

1 0

[PATCH lkp-tests] jobs: split fault_reproduction.yaml
by Wei Jihui 18 Jan '21

18 Jan '21

[why] user can choose when to ssh, use ssh.yaml is ssh when task normal finish use ssh-on-fail.yaml is ssh when task is failed use ssh-always is ssh always whether task is failed or not. [usage] submit -i ssh.yaml job.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/ssh-always.yaml | 4 ++++ jobs/ssh-on-fail.yaml | 12 ++++++++++++ jobs/{fault_reproduction.yaml => ssh.yaml} | 4 ---- 3 files changed, 16 insertions(+), 4 deletions(-) create mode 100644 jobs/ssh-always.yaml create mode 100644 jobs/ssh-on-fail.yaml rename jobs/{fault_reproduction.yaml => ssh.yaml} (80%) diff --git a/jobs/ssh-always.yaml b/jobs/ssh-always.yaml new file mode 100644 index 00000000..0166fe12 --- /dev/null +++ b/jobs/ssh-always.yaml @@ -0,0 +1,4 @@ +# ssh to testbox and sleep always + +<< : jobs/ssh.yaml +<< : jobs/ssh-on-fail.yaml diff --git a/jobs/ssh-on-fail.yaml b/jobs/ssh-on-fail.yaml new file mode 100644 index 00000000..5f28d001 --- /dev/null +++ b/jobs/ssh-on-fail.yaml @@ -0,0 +1,12 @@ +ssh_pub_key: + <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + +on_fail: + sshd: + sleep: 6h diff --git a/jobs/fault_reproduction.yaml b/jobs/ssh.yaml similarity index 80% rename from jobs/fault_reproduction.yaml rename to jobs/ssh.yaml index 46a1e042..11e5b175 100644 --- a/jobs/fault_reproduction.yaml +++ b/jobs/ssh.yaml @@ -8,7 +8,3 @@ ssh_pub_key: %> sshd: sleep: 6h - -on_fail: - sshd: - sleep: 6h -- 2.23.0

3 4

[PATCH v2 compass-ci] lib/job.cr: enable the account check function
by Wu Zhende 18 Jan '21

18 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 03ca5af..db9616b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -392,15 +392,15 @@ class Job private def check_account_info error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" + account_info = @es.get_account(self["my_email"]) - # raise account_info unless account_info.is_a?(JSON::Any) + raise error_msg unless account_info.is_a?(JSON::Any) - # @account_info = account_info.as_h - @account_info = account_info.as_h if account_info.is_a?(JSON::Any) + @account_info = account_info.as_h - # raise error_msg if @account_info["found"]? == false - # raise error_msg unless self["my_name"] == @account_info["my_name"].to_s - # raise error_msg unless self["my_token"] == @account_info["my_token"] + raise error_msg if @account_info["found"]? == false + raise error_msg if self["my_name"] != @account_info["my_name"].to_s + raise error_msg if self["my_token"] != @account_info["my_token"] @hash.delete("my_uuid") @hash.delete("my_token") -- 2.23.0

2 1

[PATCH compass-ci] lib/job.cr: auto install cgz with lkp
by Wei Jihui 18 Jan '21

18 Jan '21

[why] tests need some common softwares which at lkp-tests/distro/depend/lkp. initramfs-common just for initramfs os, it is not enough. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 07bfda0..ae1f170 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -556,7 +556,7 @@ class Job private def get_depends_initrd(program_params, initrd_deps_arr, initrd_pkg_arr) initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - program_params["initramfs-common"] = JSON::Any.new("") if os_mount == "initramfs" + program_params["lkp"] = JSON::Any.new("") program_params.keys.each do |program| if program =~ /^(.*)-\d+$/ -- 2.23.0

2 1

[PATCH lkp-tests] distro/depends: merge initramfs-common to lkp
by Wei Jihui 18 Jan '21

18 Jan '21

all type os will use distro/depends/lkp Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/initramfs-common | 4 ---- distro/depends/lkp | 3 +++ 2 files changed, 3 insertions(+), 4 deletions(-) delete mode 100644 distro/depends/initramfs-common diff --git a/distro/depends/initramfs-common b/distro/depends/initramfs-common deleted file mode 100644 index 5f766da6..00000000 --- a/distro/depends/initramfs-common +++ /dev/null @@ -1,4 +0,0 @@ -time -wget -perf -which diff --git a/distro/depends/lkp b/distro/depends/lkp index 82c67184..4b51c803 100644 --- a/distro/depends/lkp +++ b/distro/depends/lkp @@ -1,4 +1,7 @@ bc +git +wget +perf gawk time kmod -- 2.23.0

2 1

[PATCH lkp-tests] sbin/submit: add parameter option -i for submit
by Wei Jihui 18 Jan '21

18 Jan '21

[usage] 1. submit -i include.yaml job.yaml this will add include.yaml content to job.yaml, and priority is higher than job.yaml 2. submit -i 1.yaml -i 2.yaml job.yaml this will add 1.yaml and 2.yaml to job.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 12 ++++++++---- sbin/submit | 14 ++++++++++++++ 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 631ebfc9..1650eb2f 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -215,6 +215,7 @@ class Job attr_accessor :overrides attr_accessor :defaults attr_accessor :cmdline_defaults + attr_accessor :include_yamls def initialize(job = {}, defaults = {}, overrides = {}, cmdline_defaults={}) @job = job @@ -222,6 +223,7 @@ class Job @overrides = overrides # from command line @cmdline_defaults = cmdline_defaults # from command line @available_programs = {} + @include_yamls = [] end def source_file_symkey(file) @@ -254,8 +256,8 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end - hash.merge!(@overrides) - hash.merge!(fault_reproduction) if hash.has_key?("id") + hash.merge!(@overrides) unless @overrides.empty? + hash.merge!(load_include_yamls) unless @include_yamls.empty? @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e @@ -286,8 +288,10 @@ class Job @jobfile = jobfile end - def fault_reproduction - wait = load_yaml("#{lkp_src}/jobs/fault_reproduction.yaml", {}) + def load_include_yamls + include_hash = {} + include_hash[comment_to_symbol('auto generated by load_include_yamls()')] = nil + include_hash.merge!(load_yaml_merge(@include_yamls, {})) end def multi_args(hash) diff --git a/sbin/submit b/sbin/submit index 39cdcf57..6eca3b6e 100755 --- a/sbin/submit +++ b/sbin/submit @@ -14,6 +14,7 @@ opt_set_key_value = {} opt_cmdline_defaults = {} opt_output_dir = nil opt_auto_define_files = false +opt_include_yamls = [] opt_monitor = false opt_monitor_query = {} opt_my_queue = false @@ -43,6 +44,10 @@ options = OptionParser.new do |opts| opt_auto_define_files = true end + opts.on('-i include.yaml', '--include include.yaml', 'include other job yamls') do |yaml| + opt_include_yamls << yaml + end + opts.on('-c', '--connect', 'auto connect to the host') do actions << 'connect' end @@ -103,6 +108,14 @@ def find_jobfile(jobfile) exit 1 end +def find_jobfiles(jobfile_list) + search_jobfile_list = [] + jobfile_list.each do |jobfile| + search_jobfile_list << find_jobfile(jobfile) + end + return search_jobfile_list +end + submit_id = %x(uuidgen).chomp puts "submit_id=#{submit_id}" @@ -111,6 +124,7 @@ ARGV.each do |jobfile| jobs = Job2sh.new jobs.cmdline_defaults = opt_cmdline_defaults jobs.overrides = opt_set_key_value + jobs.include_yamls = find_jobfiles(opt_include_yamls) jobs.load(jobfile, true) || next jobs[:expand_params] = true jobs['testbox'] = opt_set_key_value['testbox'] if opt_set_key_value['testbox'] -- 2.23.0

3 2

[PATCH v2 compass-ci 3/3] docker/docker.rb: add docker serial log
by Wu Zhende 18 Jan '21

18 Jan '21

Generate serial log file. And add the start and end flags into it. It's for generate docker's dmsg file in result root. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 8e489c6..e1ab057 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -18,6 +18,7 @@ names = Set.new %w[ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 +LOG_DIR = '/srv/cci/serial/logs' def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -82,12 +83,35 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, + { 'hostname' => hostname, + 'docker_image' => docker_image, + 'load_path' => load_path, + 'log_dir' => "#{LOG_DIR}/#{hostname}" + }, ENV['CCI_SRC'] + '/providers/docker/run.sh' ) clean_dir(load_path) end +def record_runtime_log(hostname, start_time: Time.new, is_start: true, hash: {}) + log_file = "#{LOG_DIR}/#{hostname}" + if is_start + File.open(log_file, 'w') do |f| + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep(2) + f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting DOCKER" + f.puts "\n#{hash['job']}" + end + return start_time + end + + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal DOCKER duration: #{duration} minutes" + end +end + def main(hostname, queues) set_host2queues(hostname, queues) url = get_url hostname @@ -95,10 +119,14 @@ def main(hostname, queues) hash = parse_response url return del_host2queues(hostname) if hash.nil? + start_time = record_runtime_log(hostname, hash: hash) + load_path = build_load_path(hostname) load_initrds(load_path, hash) start_container(hostname, load_path, hash) + del_host2queues(hostname) + record_runtime_log(hostname, start_time: start_time, is_start: false) end def loop_main(hostname, queues) -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] docker/run.sh: collect docker serial logs
by Wu Zhende 18 Jan '21

18 Jan '21

Save docker logs to /srv/cci/serial/logs/$hostname. So fluentd can tail this file and send logs to monitoring service. In this way can generate docker's dmsg file Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/run.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 4d53db7..2810641 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -7,6 +7,7 @@ : ${docker_image:="centos:7"} : ${load_path:="${HOME}/jobs"} : ${hostname:="dc-1g-1"} +: ${log_dir:="/srv/cci/serial/logs/$hostname"} if [[ $hostname =~ ^(.*)-[0-9]+$ ]]; then tbox_group=${BASH_REMATCH[1]} @@ -38,4 +39,4 @@ cmd=( /root/sbin/entrypoint.sh ) -"${cmd[@]}" +"${cmd[@]}" 2>&1 | tee -a "$log_dir" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] monitoring/parse_serial_logs.cr: add docker patterns
by Wu Zhende 18 Jan '21

18 Jan '21

Add the start and end flags of the Docker startup mode. Use to generate docker's dmsg file. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/parse_serial_logs.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 5a54849..0589523 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -25,6 +25,7 @@ require "json" class SerialParser START_PATTERNS = [ "starting QEMU", + "starting DOCKER", "Start PXE over IPv4", "iPXE initialising devices", "Open Source Network Boot Firmware", @@ -34,6 +35,7 @@ class SerialParser END_PATTERNS = [ "Total QEMU duration: ", + "Total DOCKER duration: ", "No job now", "Restarting system", ] -- 2.23.0

1 0

[PATCH v7 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 18 Jan '21

18 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archLinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archLinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 sum.stats.last_state.test.iperf.exit_code.127: 12 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0, null, null, null, null, null, null, null, null, 524288.0, 524288.0, null, null, null, null, null, 516608.0, null, 1048576.0, null, null, null, null, null, null, null, 524288.0, null, null, null, null, 2104832.0, 1572864.0, null, null, 2097152.0, null, null, null, null, null, null, null, null, null] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... crystal.122454.summary.success: 1 crystal.122454.summary.result: success crystal.122690.summary.any_warning: 1 crystal.122690.summary.any_stderr: 1 crystal.122690.summary.any_error: 1 crystal.122690.summary.result: error ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 126 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 119 insertions(+), 7 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..c8a5b6c 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo].freeze +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,127 @@ class ESJobs return 0 end - def output + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(metrics) result = { - 'stats.count' => @stats['stats.count'] + 'kvcount' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + stats.each do |key, value| + result['raw.stats'][key] << value end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] + + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + end + end + + def set_jobs_stat(result, stats, job) + summary_result = '' + if stats.keys.any? { |stat| stat.match(/warn/i) } + result["#{job['id']}.summary.any_warning"] = 1 + summary_result = 'warning' + end + if stats.keys.any? { |stat| stat.match(/stderr/i) } + result["#{job['id']}.summary.any_stderr"] = 1 + summary_result = 'stderr' + end + if stats.keys.any? { |stat| stat.match(/error|nr_fail=0/i) } + result["#{job['id']}.summary.any_error"] = 1 + summary_result = 'error' + end + if stats.keys.any? { |stat| stat.match(/fail/i) } && + stats.keys.all? { |stat| !stat.match(/nr_fail=0/i) } + result["#{job['id']}.summary.any_fail"] = 1 + summary_result = 'fail' + end + + if summary_result.empty? + result["#{job['id']}.summary.success"] = 1 + result["#{job['id']}.summary.result"] = 'success' + else + result["#{job['id']}.summary.result"] = summary_result + end + end + + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = '---' + value.inspect + '---' + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_jobs_stat(result, stats, job) + set_default_value(result, stats, metrics) + kvcount(result, job) + end + + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + @result = query_jobs_state((a)jobs.values) + @result['kvcount'] = @result['kvcount'].to_a.sort.to_h + @results = {} + compact_hash('', @result) + yaml = @results.to_yaml.delete_prefix!("---\n") + puts yaml.gsub('nil', 'null').gsub('"---', '').gsub('---"', '') end end -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] my-qemu.sh: add `-h|--help`, `-d|--debug` params for my-qemu.sh
by Yu Chuan 18 Jan '21

18 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- providers/my-qemu.sh | 33 +++++++++++++++++++++++++++++++++ providers/qemu/kvm.sh | 13 +++++++------ providers/qemu/pxe.sh | 8 ++++++-- 3 files changed, 46 insertions(+), 8 deletions(-) diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 8845a346e195..7d72a8fea708 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,6 +2,39 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +source $CCI_SRC/lib/log.sh + +help() +{ + cat <<-'EOF' + Usage: ./my-qemu.sh [OPTION]... + Ask scheduler service whether there is a job, and start a local vm if there is a job. + Scheduler api: http://${SCHED_HOST}:${SCHED_PORT}/boot.ipxe/mac/${mac} + + Mandatory arguments to long options are mandatory for short options too. + -h, --help display this help and exit + -d, --debug open the local vm serial port in the current shell + EOF + + exit 0 +} + +while true +do + [ $# -eq 0 ] && break + case "$1" in + -h|--help) + help;; + -d|--debug) + export DEBUG=true;; + *) + die "Unknown param: $1" + help + exit 1;; + esac + shift +done + [[ $tbox_group ]] || tbox_group=vm-2p8g export hostname=$tbox_group.$USER-$$ diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index a136c8ff45cb..098e8c2edcc7 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -115,11 +115,11 @@ set_qemu() print_message() { - echo SCHED_PORT: $SCHED_PORT - echo kernel: $kernel - echo initrds: $initrds - echo append: $append - echo less $log_file + log_info SCHED_PORT: $SCHED_PORT + log_info kernel: $kernel + log_info initrds: $initrds + log_info append: $append + [ "$DEBUG" == "true" ] || log_info less $log_file sleep 5 } @@ -136,9 +136,10 @@ public_option() -k en-us -no-reboot -nographic - -serial file:${log_file} -monitor null ) + + [ "$DEBUG" == "true" ] || kvm=("${kvm[@]}" -serial file:${log_file}) } individual_option() diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index fa688331964a..ff2c5661d3e4 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -4,6 +4,8 @@ # - nr_cpu # - memory +source $CCI_SRC/lib/log.sh + : ${nr_cpu:=1} : ${memory:=1G} @@ -18,7 +20,7 @@ fi qemu=qemu-system-aarch64 command -v $qemu >/dev/null || qemu=qemu-kvm -echo less $serial_log +[ "$DEBUG" == "true" ] || log_info less $serial_log kvm=( $qemu @@ -33,7 +35,9 @@ kvm=( -k en-us -no-reboot -nographic - -serial file:${serial_log} -monitor null ) + +[ "$DEBUG" == "true" ] || kvm=("${kvm[@]}" -serial file:${serial_log}) + "${kvm[@]}" -- 2.23.0

2 1

[PATCH v6 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 18 Jan '21

18 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archLinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archLinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 sum.stats.last_state.test.iperf.exit_code.127: 12 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0, nil, nil, nil, nil, nil, nil, nil, nil, 524288.0, 524288.0, nil, nil, nil, nil, nil, 516608.0, nil, 1048576.0, nil, nil, nil, nil, nil, nil, nil, 524288.0, nil, nil, nil, nil, 2104832.0, 1572864.0, nil, nil, 2097152.0, nil, nil, nil, nil, nil, nil, nil, nil, nil] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 97 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 90 insertions(+), 7 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..a36a241 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,98 @@ class ESJobs return 0 end - def output + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(metrics) result = { - 'stats.count' => @stats['stats.count'] + 'kvcount' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] + + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + end + end - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = '---' + value.inspect + '---' + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end + + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def output + @result = query_jobs_state((a)jobs.values) + @result['kvcount'] = @result['kvcount'].to_a.sort.to_h + @results = {} + compact_hash('', @result) + yaml = @results.to_yaml.delete_prefix!("---\n") + puts yaml.gsub('"---', '').gsub('---"', '') end end -- 2.23.0

2 1

[PATCH lkp-tests] distro/depends: add ansible_test depends file
by Zhang Yale 18 Jan '21

18 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- distro/depends/ansible_test | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 distro/depends/ansible_test diff --git a/distro/depends/ansible_test b/distro/depends/ansible_test new file mode 100644 index 00000000..5eaf3f3c --- /dev/null +++ b/distro/depends/ansible_test @@ -0,0 +1,5 @@ +python3-pip +gcc +python3-devel +libffi-devel +openssl-devel -- 2.23.0

1 0

[PATCH lkp-tests 3/4] tests: add ansible_tests script
by Zhang Yale 18 Jan '21

18 Jan '21

This script can be invoked to run the ansible command to deploy ansible-roles which you specified in your environment. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/ansible_test | 59 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 00000000..09e5ca9e --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,59 @@ +#!/bin/bash +# username +# rolename +# ansible_host +# ansible_user +# ansible_python_interpreter + +[ -n "$username" ] || echo "username is empty" +[ -n "$rolename" ] || echo "rolename is empty" +[ -n "$ansible_host" ] || $ansible_host=localhost +[ -n "$ansible_user" ] || $ansible_user=root +[ -n "$ansible_python_interpreter" ] || $ansible_python_interpreter=/user/bin/python3 + +build_mutual_trust() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa + cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys + ssh -o "StrictHostKeyChecking=no" -o "ExitOnForwardFailure=yes" -o "passwordAuthentication=no" root@$ansible__host 'chmod 600 ~/.ssh/authorized_keys' +} + +install_ansible() +{ + pip3 install ansible + ansible localhost -m ping + #ansible all -m authorized_key -a "user=root key='{{ lookup('file', '/root/.ssh/id_rsa.pub') }}' path=/root/.ssh/authorized_keys manage_dir=no" --ask-pass -c paramiko + [ $? -eq 0 ] && echo "Ansible is successfully deployed!" +} + +download_ansible_role() +{ + ansible-galaxy install $username.$rolename + [ $? -eq 0 ] && echo "Please check whether your rolename is correct!" +} + +ansible_playbook_run() +{ + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible + cat <<-EOF > "$ansible_path"/hosts + $ansible_host ansible_user=$ansible_user ansible_python_interpreter=$ansible_python_interpreter + EOF + [ $? -eq 0 ] && echo "Hostfile has been generated!" + + cat <<-EOF > "$ansible_path"/$rolename.yml + - hosts: $ansible_host + remote_user: $ansible_user + roles: + - role: $username.$rolename + EOF + [ $? -eq 0 ] && echo "Playbook has been generated!" + + ansible-playbook "$ansible_path"/$rolename.yml + [ $? -eq 0 ] && echo "Your playbook is running successfully!" +} + +#build_mutual_trust +install_ansible +download_ansible_role +ansible_playbook_run -- 2.23.0

1 1

[PATCH lkp-tests 4/4] distro/depends: add vim in lkp
by Zhang Yale 18 Jan '21

18 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- distro/depends/lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/lkp b/distro/depends/lkp index 82c67184..5a8196ac 100644 --- a/distro/depends/lkp +++ b/distro/depends/lkp @@ -6,3 +6,4 @@ gzip which rsync ntpdate +vim -- 2.23.0

1 0

[PATCH lkp-tests 1/4] pkg/turbostat: changed turbostat's source url
by Zhang Yale 18 Jan '21

18 Jan '21

turbostat can not be found in x86_64 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- pkg/turbostat/PKGBUILD | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/turbostat/PKGBUILD b/pkg/turbostat/PKGBUILD index d7e536c7..24adcffe 100644 --- a/pkg/turbostat/PKGBUILD +++ b/pkg/turbostat/PKGBUILD @@ -4,7 +4,7 @@ pkgrel=1 url='https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git' arch=('i386' 'x86_64') license=('GPL') -source=('https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git') +source=("git://$GIT_SERVER/l/linux/linux.git") md5sums=('SKIP') build() -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/compare_error_messages.rb: add the credibility check of job
by Lin Jiaxin 18 Jan '21

18 Jan '21

defect: may match the wrong result error_id="build-pkg.pixz.c:warning:'subsuf'defined-but-not-used[-Wunused-function]" filename="pixz.c" correct result: gcc -DHAVE_CONFIG_H -I. -I.. -D_FORTIFY_SOURCE=2 -pthread -Wall -Wno-unknown-pragmas -march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1, --sort-common,--as-needed,-z,relro,-z,now -MT pixz-pixz.o -MD -MP -MF .deps/pixz-pixz.Tpo -c -o pixz-pixz.o `test -f 'pixz.c' || echo './'`pixz.c ^^^^^^ wrong result: mv -f .deps/pixz.cpu.Tpo .deps/pixz.cpu.Po ^^^^^^ Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 0b51fc3..b3812bb 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -8,6 +8,39 @@ require_relative 'es_query' require_relative 'error_messages' require_relative "#{ENV['LKP_SRC']}/lib/common" +# previous_job_id's error_ids include error_id +# If later_job_id's output include error filename that extract from error_id, +# the later_job_id is credible. +def credible?(previous_job_id, later_job_id, error_id) + es = ESQuery.new + + previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') + later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') + + return false if filenames_check(previous_result_file, later_result_file, error_id).value?(false) + + return true +end + +def filenames_check(previous_result_file, later_result_file, error_id) + filenames = Set.new + filenames_check = Hash.new { |h, k| h[k] = false } + + error_lines = ErrorMessages.new(previous_result_file).obtain_error_messages_by_error_id(error_id, true) + error_lines.each do |error_line| + filenames << $1 if error_line =~ /(.*)(:.*){2}: (error|warning):/ + end + + File.open(later_result_file).each_line do |line| + filenames.each do |filename| + filenames_check[filename] + filenames_check[filename] = true if line.include?(filename) + end + end + + return filenames_check +end + def get_compare_result(previous_job_id, later_job_id) es = ESQuery.new -- 2.23.0

2 1

[PATCH v3 compass-ci] lib/error_messages.rb: obtain_error_messages_by_error_id function add keyword argument
by Lin Jiaxin 18 Jan '21

18 Jan '21

Determine whether to return the error line or error messages by the value of return_error_line Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 3e1564b..d83d890 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -40,12 +40,12 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_error_id(error_id) + def obtain_error_messages_by_error_id(error_id, return_error_line = false) error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| if ('build-pkg.' + build_pkg_error_id(k)) == error_id - error_messages_by_error_id += v.to_a + error_messages_by_error_id += return_error_line ? [k] : v.to_a end end error_messages_by_error_id -- 2.23.0

2 1

[PATCH compass-ci 2/3] qemu/kvm.sh: optimize the prompt when the error occurs
by Yu Chuan 18 Jan '21

18 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- providers/qemu/kvm.sh | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 35678fbd44ef..f1d65272dbf9 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -52,15 +52,23 @@ parse_ipxe_script() check_option_value() { + [ -n "$kernel" ] || { + echo "[ERROR] Can not find job for current hostname: $hostname." + exit 1 + } + [ -s "$kernel" ] || { - echo "The kernel does not exist: $kernel" - exit + echo "[ERROR] Can not find kernel file or kernel file is empty: $kernel." + exit 1 } # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit + [ -n "$initrds" ] || { + echo "[ERROR] Can not find any initrd for current vm." + exit 1 + } } set_initrd() -- 2.23.0

3 4

[PATCH compass-ci] src/lib/web_backend.rb: fix wrong number of arguments
by Lu Weitao 18 Jan '21

18 Jan '21

[Why] wrong number of arguments (given 3, expected 4) 2021-01-18 11:49:46.000000000 +0800 web-backend: 172.17.0.1 - - [18/Jan/2021:03:49:44 UTC] "GET /compare?suite=iperf&dimension=os HTTP/1.1" 500 13 Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/lib/web_backend.rb | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index ea67064..61e3334 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -186,15 +186,15 @@ end def do_get_groups_matrices(must, dimension, total, size, from) job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_hash = combine_group_query_data(job_list, dimension) + matrices, suites_hash, latest_jobs_hash = combine_group_query_data(job_list, dimension) while matrices.empty? from += size break if from > total job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_hash = combine_group_query_data(job_list, dimension) + matrices, suites_hash, latest_jobs_hash = combine_group_query_data(job_list, dimension) end - [matrices, suites_hash] + [matrices, suites_hash, latest_jobs_hash] end def get_groups_matrices(conditions, dimension, must, size, from) @@ -209,11 +209,12 @@ end def get_compare_body(params) dimension, conditions = get_dimension_conditions(params) must = get_es_must(params) - groups_matrices, suites_hash = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) + groups_matrices, suites_hash, latest_jobs_hash = + get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) if !groups_matrices || groups_matrices.empty? body = 'No Data.' else - body = compare_group_matrices(groups_matrices, suites_hash, { no_print: true }) + body = compare_group_matrices(groups_matrices, suites_hash, latest_jobs_hash, { no_print: true }) body = 'No Difference.' if !body || body.empty? end return body -- 2.23.0

1 0

[PATCH compass-ci 3/3] my-qemu.sh: add `-h|--help`, `-d|--debug` params for my-qemu.sh
by Yu Chuan 18 Jan '21

18 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- providers/my-qemu.sh | 32 ++++++++++++++++++++++++++++++++ providers/qemu/kvm.sh | 5 +++-- providers/qemu/pxe.sh | 6 ++++-- 3 files changed, 39 insertions(+), 4 deletions(-) diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 8845a346e195..a8f09a610d3f 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,6 +2,38 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +help() +{ + cat <<-'EOF' + Usage: ./my-qemu.sh [OPTION]... + Ask scheduer service whether there is a job, and start a local vm if there is a job. + Scheduler api: http://${SCHED_HOST}:${SCHED_PORT}/boot.ipxe/mac/${mac} + + Mandatory arguments to long options are mandatory for short options too. + -h, --help display this help and exit + -d, --debug open the local vm serial port in the current shell + EOF + + exit 0 +} + +while true +do + [ $# -eq 0 ] && break + case "$1" in + -h|--help) + help + shift;; + -d|--debug) + export DEBUG=true + shift;; + *) + echo "[ERROR] Unknown param: $1" + help + exit 1;; + esac +done + [[ $tbox_group ]] || tbox_group=vm-2p8g export hostname=$tbox_group.$USER-$$ diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index f1d65272dbf9..a3cbe4248a3a 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -123,7 +123,7 @@ print_message() echo kernel: $kernel echo initrds: $initrds echo append: $append - echo less $log_file + [ "$DEBUG" == "true" ] || echo less $log_file sleep 5 } @@ -140,9 +140,10 @@ public_option() -k en-us -no-reboot -nographic - -serial file:${log_file} -monitor null ) + + [ "$DEBUG" == "true" ] || kvm=("${kvm[@]}" -serial file:${log_file}) } individual_option() diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index fa688331964a..2e205405c793 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -18,7 +18,7 @@ fi qemu=qemu-system-aarch64 command -v $qemu >/dev/null || qemu=qemu-kvm -echo less $serial_log +[ "$DEBUG" == "true" ] || echo less $serial_log kvm=( $qemu @@ -33,7 +33,9 @@ kvm=( -k en-us -no-reboot -nographic - -serial file:${serial_log} -monitor null ) + +[ "$DEBUG" == "true" ] || kvm=("${kvm[@]}" -serial file:${serial_log}) + "${kvm[@]}" -- 2.23.0

2 2

[PATCH compass-ci] sparrow/0-package: add packages
by Liu Yinsi 18 Jan '21

18 Jan '21

container os-cifs and os-nfs will use sudo command, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index b25dade..b8ef769 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -24,6 +24,7 @@ pkgs=( hostname wget curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 4d13413..947cbdc 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -24,6 +24,7 @@ pkgs=( qemu-system-aarch64 qemu-system-x86 qemu-system-riscv64 + sudo ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 04996f5..61a37c4 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -23,6 +23,7 @@ pkgs=( hostname libvirt curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: mount /srv/cci/libvirt-xml to srv-http
by Xiao Shenwei 18 Jan '21

18 Jan '21

[why] libvirt support user define domain.xml file to start kvm the /srv/cci/libvirt-xml will save user's xml file Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index d9f461b..945cde0 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -16,6 +16,7 @@ cmd=( -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/os:/srv/os:ro -v /srv/git/archlinux:/srv/git/archlinux:ro + -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -d srv-http ) -- 2.23.0

1 0

[PATCH compass-ci 1/3] qemu/kvm.sh: fix script will hang when there is no job
by Yu Chuan 18 Jan '21

18 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd When there is no job corresponding to current vm in the remote redis queue, var $initrds will be equal to "". And from `cat --help`, we can know the follow: cat Copy standard input to standard output. So the process will get stuck waiting for input. [How] Function check_option_value() is used to check whether $initrds or $kernel is empty. And $initrds and $kernel are assigned in the function parse_ipxe_script(). So move check_option_value() before set_initrd(). Signed-off-by: Yu Chuan <13186087857(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e9138a7c0..35678fbd44ef 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -197,8 +197,8 @@ write_logfile parse_ipxe_script -set_options check_option_value +set_options print_message -- 2.23.0

2 6

[PATCH v4 compass-ci] docker2rootfs: add unzip vmlinux function
by Wang Chenglong 18 Jan '21

18 Jan '21

[Why] Unpacked vmlinux into vmlinuz format. Signeo-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/common | 33 ++++++++++++++++++++++++++++- container/docker2rootfs/run | 1 + container/docker2rootfs/setup-image | 2 +- 3 files changed, 34 insertions(+), 2 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index c26bbc8..e8c56ba 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -5,6 +5,8 @@ # This file is called by script run running on host # define functions and set host environment variables +. $CCI_SRC/container/qcow2rootfs/bin/common + get_host_aarch() { aarch=$(arch) @@ -133,7 +135,7 @@ get_rootfs_kernel() local vmlinuz_file local vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel @@ -155,3 +157,32 @@ create_links_vmlinuz_initrd() echo "[INFO] Create links to initrd.lkp and vmlinuz success!" } + +unzip_vmlinuz() { + get_rootfs_kernel + + file ${ROOTFS_VMLINUZ_FILE} | grep -q gzip || return + + get_vmlinuz_start_postion ${ROOTFS_VMLINUZ_FILE} + [ $VMLINUZ_START_POSTION -eq -1 ] && return + + local unzip_str + + if [ $VMLINUZ_START_POSTION -eq 0 ]; then + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 | zcat > $ROOTFS_VMLINUZ.tmp" + else + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 skip=$VMLINUZ_START_POSTION | zcat > $ROOTFS_VMLINUZ.tmp" + fi + + echo "Unzipping vmlinuz..." + cd "$ROOTFS_DIR/boot" + $unzip_str + mv $ROOTFS_VMLINUZ.tmp $ROOTFS_VMLINUZ + chmod o+r $ROOTFS_VMLINUZ + + if [ $? -eq 0 ]; then + echo "[INFO] Unzipping vmlinuz success!" + else + echo "[WARNING] Unzipping vmlinuz failed!" + fi +} diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index 8a7b315..cdb19bf 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -31,3 +31,4 @@ start=( cp_rootfs $IMAGE_PACK $2 docker rm -f rootfs-docker create_links_vmlinuz_initrd $2 +unzip_vmlinuz diff --git a/container/docker2rootfs/setup-image b/container/docker2rootfs/setup-image index ffe4937..47bfb7e 100755 --- a/container/docker2rootfs/setup-image +++ b/container/docker2rootfs/setup-image @@ -38,7 +38,7 @@ setup_login() pack_cgz() { echo "Packing package. Please wait." - find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + find ./ ! -path "./tmp/$IMAGE_PACK" | cpio -o -Hnewc | gzip -9 > "./tmp/$IMAGE_PACK" chmod 644 /tmp/${IMAGE_PACK} } -- 2.23.0

1 0

[PATCH v3 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 18 Jan '21

18 Jan '21

before: if execute ./buildall and docker images exist, then ./build after: if execute ./buildall or docker images not exist, then ./build Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index f24226c..77fe43a 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" -ne 0 ] || [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

5 7

[PATCH v2 compass-ci 1/2] lib/error_messages.rb: obtain_error_messages_by_error_id function add keyword argument
by Lin Jiaxin 18 Jan '21

18 Jan '21

Determine whether to return the error line or error messages by the value of eturn_error_line Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 3e1564b..d83d890 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -40,12 +40,12 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_error_id(error_id) + def obtain_error_messages_by_error_id(error_id, return_error_line = false) error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| if ('build-pkg.' + build_pkg_error_id(k)) == error_id - error_messages_by_error_id += v.to_a + error_messages_by_error_id += return_error_line ? [k] : v.to_a end end error_messages_by_error_id -- 2.23.0

2 1

[PATCH lkp-tests] hosts: add new host file for libvirt
by Xiao Shenwei 18 Jan '21

18 Jan '21

[why] libvirt use a xml file to start vm <os> <kernel>xxx</kernel> <initrd>xxx</initrd> <cmdline>xxx</cmdline> </os> so, we can submit job use os_mount=initramfs to get kernel and initrd absolute path. Minimum memory limit: 16G Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/libvirt-2p16g | 5 +++++ hosts/libvirt-2p32g | 5 +++++ 2 files changed, 10 insertions(+) create mode 100644 hosts/libvirt-2p16g create mode 100644 hosts/libvirt-2p32g diff --git a/hosts/libvirt-2p16g b/hosts/libvirt-2p16g new file mode 100644 index 000000000..aecbb7948 --- /dev/null +++ b/hosts/libvirt-2p16g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 16G diff --git a/hosts/libvirt-2p32g b/hosts/libvirt-2p32g new file mode 100644 index 000000000..aa7c8c065 --- /dev/null +++ b/hosts/libvirt-2p32g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 32G -- 2.23.0

1 1

[PATCH v2 compass-ci] sbin/es-find support range query
by Lu Weitao 18 Jan '21

18 Jan '21

[Example] es-find -r/--range "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/sbin/es-find b/sbin/es-find index 17eefe6..c144a8b 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -5,6 +5,7 @@ # Usage: # es-find key=val ... +# es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -29,7 +30,43 @@ def parse_argv items end +# output: +# eg: +# { +# start_time => { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# ... +# } +def prase_range(range) + range_hash = {} + keyword_list = range.split(';') + keyword_list.each do |keyword| + key, value = keyword.split('=') + next unless key && value + + range_hash[key] = assign_range(value) + end + + range_hash +end + +def assign_range(value) + range_value = {} + value_list = value.split(',') + if value_list.length > 1 + range_value[:gte] = value_list[0] + range_value[:lte] = value_list[1] + else + range_value[:gte] = value + end + + range_value +end + options = { nr_jobs: 10, desc_keyword: nil } +range = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -49,6 +86,10 @@ opt_parser = OptionParser.new do |opts| options[:desc_keyword] = d end + opts.on('-r', '--range KEYWORD', 'range by keyword: start_time|end_time|...') do |r| + range = r + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -57,6 +98,8 @@ end opt_parser.parse!(ARGV) items = parse_argv +items[:range] = prase_range(range) if range + es = ESQuery.new(ES_HOST, ES_PORT) query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] docker/run.sh: collect docker serial logs
by Wu Zhende 18 Jan '21

18 Jan '21

Save docker logs to /srv/cci/serial/logs/$hostname. So fluentd can tail this file and send logs to monitoring service. In this way can generate docker's dmsg file Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/run.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 4d53db7..2810641 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -7,6 +7,7 @@ : ${docker_image:="centos:7"} : ${load_path:="${HOME}/jobs"} : ${hostname:="dc-1g-1"} +: ${log_dir:="/srv/cci/serial/logs/$hostname"} if [[ $hostname =~ ^(.*)-[0-9]+$ ]]; then tbox_group=${BASH_REMATCH[1]} @@ -38,4 +39,4 @@ cmd=( /root/sbin/entrypoint.sh ) -"${cmd[@]}" +"${cmd[@]}" 2>&1 | tee -a "$log_dir" -- 2.23.0

2 1

[PATCH compass-ci] lib/job.cr: enable the account check function
by Wu Zhende 18 Jan '21

18 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 07bfda0..af7cb71 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -390,15 +390,15 @@ class Job private def check_account_info error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" + account_info = @es.get_account(self["my_email"]) - # raise account_info unless account_info.is_a?(JSON::Any) + raise error_msg unless account_info.is_a?(JSON::Any) - # @account_info = account_info.as_h - @account_info = account_info.as_h if account_info.is_a?(JSON::Any) + @account_info = account_info.as_h - # raise error_msg if @account_info["found"]? == false - # raise error_msg unless self["my_name"] == @account_info["my_name"].to_s - # raise error_msg unless self["my_token"] == @account_info["my_token"] + raise error_msg if @account_info["found"]? == false + raise error_msg unless self["my_name"] == @account_info["my_name"].to_s + raise error_msg unless self["my_token"] == @account_info["my_token"] @hash.delete("my_uuid") @hash.delete("my_token") -- 2.23.0

2 2

[PATCH v3 compass-ci] docker2rootfs: add unzip vmlinux function
by Wang Chenglong 18 Jan '21

18 Jan '21

[Why] Sometimes, the kernel is in vmlinux format and needs to be unpacked into vmlinuz format. Signeo-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/common | 33 ++++++++++++++++++++++++++++- container/docker2rootfs/run | 1 + container/docker2rootfs/setup-image | 2 +- 3 files changed, 34 insertions(+), 2 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index c26bbc8..e8c56ba 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -5,6 +5,8 @@ # This file is called by script run running on host # define functions and set host environment variables +. $CCI_SRC/container/qcow2rootfs/bin/common + get_host_aarch() { aarch=$(arch) @@ -133,7 +135,7 @@ get_rootfs_kernel() local vmlinuz_file local vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel @@ -155,3 +157,32 @@ create_links_vmlinuz_initrd() echo "[INFO] Create links to initrd.lkp and vmlinuz success!" } + +unzip_vmlinuz() { + get_rootfs_kernel + + file ${ROOTFS_VMLINUZ_FILE} | grep -q gzip || return + + get_vmlinuz_start_postion ${ROOTFS_VMLINUZ_FILE} + [ $VMLINUZ_START_POSTION -eq -1 ] && return + + local unzip_str + + if [ $VMLINUZ_START_POSTION -eq 0 ]; then + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 | zcat > $ROOTFS_VMLINUZ.tmp" + else + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 skip=$VMLINUZ_START_POSTION | zcat > $ROOTFS_VMLINUZ.tmp" + fi + + echo "Unzipping vmlinuz..." + cd "$ROOTFS_DIR/boot" + $unzip_str + mv $ROOTFS_VMLINUZ.tmp $ROOTFS_VMLINUZ + chmod o+r $ROOTFS_VMLINUZ + + if [ $? -eq 0 ]; then + echo "[INFO] Unzipping vmlinuz success!" + else + echo "[WARNING] Unzipping vmlinuz failed!" + fi +} diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index 8a7b315..cdb19bf 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -31,3 +31,4 @@ start=( cp_rootfs $IMAGE_PACK $2 docker rm -f rootfs-docker create_links_vmlinuz_initrd $2 +unzip_vmlinuz diff --git a/container/docker2rootfs/setup-image b/container/docker2rootfs/setup-image index ffe4937..47bfb7e 100755 --- a/container/docker2rootfs/setup-image +++ b/container/docker2rootfs/setup-image @@ -38,7 +38,7 @@ setup_login() pack_cgz() { echo "Packing package. Please wait." - find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + find ./ ! -path "./tmp/$IMAGE_PACK" | cpio -o -Hnewc | gzip -9 > "./tmp/$IMAGE_PACK" chmod 644 /tmp/${IMAGE_PACK} } -- 2.23.0

3 4

[PATCH compass-ci] lib/job.cr: add comment for function set_sshr_info
by Wu Zhende 16 Jan '21

16 Jan '21

Explain why the key with "ssh_pub_key" is used as the entry of the function Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 07bfda0..03ca5af 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -205,6 +205,8 @@ class Job end private def set_sshr_info + # ssh_pub_key will always be set (maybe empty) by submit, + # if sshd is defined anywhere in the job return unless @hash.has_key?("ssh_pub_key") self["sshr_port"] = ENV["SSHR_PORT"] -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] docker/docker.rb: add docker serial log
by Wu Zhende 16 Jan '21

16 Jan '21

Generate serial log file. And add the start and end flags into it. It's for generate docker's dmsg file in result root. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 8e489c6..e1ab057 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -18,6 +18,7 @@ names = Set.new %w[ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 +LOG_DIR = '/srv/cci/serial/logs' def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -82,12 +83,35 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, + { 'hostname' => hostname, + 'docker_image' => docker_image, + 'load_path' => load_path, + 'log_dir' => "#{LOG_DIR}/#{hostname}" + }, ENV['CCI_SRC'] + '/providers/docker/run.sh' ) clean_dir(load_path) end +def record_runtime_log(hostname, start_time: Time.new, is_start: true, hash: {}) + log_file = "#{LOG_DIR}/#{hostname}" + if is_start + File.open(log_file, 'w') do |f| + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep(2) + f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting DOCKER" + f.puts "\n#{hash['job']}" + end + return start_time + end + + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal DOCKER duration: #{duration} minutes" + end +end + def main(hostname, queues) set_host2queues(hostname, queues) url = get_url hostname @@ -95,10 +119,14 @@ def main(hostname, queues) hash = parse_response url return del_host2queues(hostname) if hash.nil? + start_time = record_runtime_log(hostname, hash: hash) + load_path = build_load_path(hostname) load_initrds(load_path, hash) start_container(hostname, load_path, hash) + del_host2queues(hostname) + record_runtime_log(hostname, start_time: start_time, is_start: false) end def loop_main(hostname, queues) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] monitoring/parse_serial_logs.cr: add docker patterns
by Wu Zhende 16 Jan '21

16 Jan '21

Add the start and end flags of the Docker startup mode. Use to generate docker's dmsg file. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/parse_serial_logs.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 5a54849..0589523 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -25,6 +25,7 @@ require "json" class SerialParser START_PATTERNS = [ "starting QEMU", + "starting DOCKER", "Start PXE over IPv4", "iPXE initialising devices", "Open Source Network Boot Firmware", @@ -34,6 +35,7 @@ class SerialParser END_PATTERNS = [ "Total QEMU duration: ", + "Total DOCKER duration: ", "No job now", "Restarting system", ] -- 2.23.0

1 0

[PATCH compass-ci] monitoring/parse_serial_logs.cr: add docker patterns
by Wu Zhende 16 Jan '21

16 Jan '21

Add the start and end flags of the Docker startup mode. Use to generate docker's dmsg file. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/parse_serial_logs.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 5a54849..0589523 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -25,6 +25,7 @@ require "json" class SerialParser START_PATTERNS = [ "starting QEMU", + "starting DOCKER", "Start PXE over IPv4", "iPXE initialising devices", "Open Source Network Boot Firmware", @@ -34,6 +35,7 @@ class SerialParser END_PATTERNS = [ "Total QEMU duration: ", + "Total DOCKER duration: ", "No job now", "Restarting system", ] -- 2.23.0

1 0

[PATCH v2 lkp-tests] daemon/sshd: refactor sshd
by Wei Jihui 16 Jan '21

16 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/sshd | 61 +++++++----------------------------------------- lib/email.sh | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+), 53 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index e9f38006..7ba471a8 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -10,6 +10,7 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/http.sh . $PROJECT_DIR/lib/env.sh +. $PROJECT_DIR/lib/email.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 @@ -36,56 +37,6 @@ run_ssh() fi } -data_success() -{ - data="To: $my_email -Subject: [NOTIFY Compass-CI] Machine application successful - -Dear $my_username, - - Your test machine has been provisioned. - - Login: - ssh root(a)api.compass-ci.openeuler.org -p $port - Due time: - $deadline - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch - -Regards -Compass-CI -" -} - -data_failure() -{ - data="To: $my_email -Subject: [NOTIFY Compass-ci] Machine application failed - -Dear $my_username: - - Sorry to inform you that your application failed. - You may need to wait a moment, or check whether your pub_key exists. - -Regards -Compass-Ci -" -} - -compose_email() -{ - deadline=$(date -d "$runtime seconds" +"%Y-%m-%d %H:%M:%S") - if [ -n "$port" ]; then - data_success - else - data_failure - fi -} - set_port() { [ -n "$my_ssh_pubkey" ] || return @@ -103,9 +54,13 @@ set_port() run_email() { [ -n "$my_email" ] || return 0 - my_username=$(echo "$my_email" | awk -F '@' '{print $1}') - compose_email - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" + + deadline=$(date -d "$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + send_email "borrow_success" + else + send_email "borrow_failed" + fi } run_ssh diff --git a/lib/email.sh b/lib/email.sh index d7c2c44a..a0c99c24 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -15,6 +15,71 @@ Compass-CI " } +job_debug_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Test job debug: $id + +Dear $my_name, + + Your test job is ready to debug and test machine has been provisioned. + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-CI +" +} + +borrow_success_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Machine application successful: $id + +Dear $my_name, + + Your test machine has been provisioned. + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-CI +" +} + +borrow_failed_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Machine application failed: $id + +Dear $my_name: + + Sorry to inform you that your application failed. + You may need to wait a moment, or check whether your pub_key exists. + +Regards +Compass-CI +" +} + send_email() { local subject=$1 -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: replace IP with domain name
by Wu Fengguang 16 Jan '21

16 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- lib/monitor.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index eca70f196..4785477ec 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -107,7 +107,7 @@ class Monitor def lftp_mirror @result_root.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || '124.160.11.58' + srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' srv_http_port = job['SRV_HTTP_PORT'] || '11300' url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" system "lftp -c mirror #{url} >/dev/null 2>&1" -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] container/assist-cci: upload function
by Ren Wen 15 Jan '21

15 Jan '21

Upload an XML template of libvirt. Response is the modified filename. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/routes.rb | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/assist-cci/routes.rb diff --git a/container/assist-cci/routes.rb b/container/assist-cci/routes.rb new file mode 100755 index 0000000..0e71a75 --- /dev/null +++ b/container/assist-cci/routes.rb @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'digest/md5' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_CCI_PORT'] +end + +LIBVIRT_XML_PATH = ENV['LIBVIRT_XML_PATH'] +VERSION = '0.1' + +get '/' do + "version: #{VERSION}" +end + +post '/upload' do + if params[:file] && (tempfile = params[:file][:tempfile]) && (filename = params[:file][:filename]) + dirname = File.basename(filename, '.xml') + content = tempfile.read + md5_filename = Digest::MD5.hexdigest(content) + '.xml' + + dir_path = "#{LIBVIRT_XML_PATH}/#{dirname}" + Dir.exist?(dir_path) || Dir.mkdir(dir_path) + + target = "#{dir_path}/#{md5_filename}" + File.open(target, 'wb') { |f| f.write content } + + File.join(dirname, md5_filename) + else + '' + end +end -- 2.23.0

3 4

[PATCH compass-ci] sbin/auto_submit: fix invalid lab value
by Liu Yinsi 15 Jan '21

15 Jan '21

lab should set by dev env $lab Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index c121ae0..ecc4d70 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,7 +86,7 @@ end option = { my_name: 'Auto Submit Robot', my_email: 'autosubmit@localhost', - lab: 'autosubmit' + lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp } # fix verification problems when submit jobs by auto-submit service -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/0-package: add packages
by Liu Yinsi 15 Jan '21

15 Jan '21

[why] 1. providers/qemu/kvm.sh needs wget, curl, qemu command, so add it. 2. use qemu-system-ARCH packages to replace qemue in debian, refer: https://packages.debian.org/sid/qemu. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 2 ++ sparrow/0-package/os/debian | 6 +++++- sparrow/0-package/os/openEuler | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 8df36c0..b25dade 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -22,6 +22,8 @@ pkgs=( busybox iproute hostname + wget + curl ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index c45e8c2..4d13413 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -13,13 +13,17 @@ pkgs=( docker.io ruby-full ruby-dev - qemu dmidecode lockfile-progs bridge-utils busybox iproute2 hostname + wget + curl + qemu-system-aarch64 + qemu-system-x86 + qemu-system-riscv64 ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 37c74d0..04996f5 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -22,6 +22,7 @@ pkgs=( iproute hostname libvirt + curl ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v5 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archLinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archLinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 sum.stats.last_state.test.iperf.exit_code.127: 12 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0, nil, nil, nil, nil, nil, nil, nil, nil, 524288.0, 524288.0, nil, nil, nil, nil, nil, 516608.0, nil, 1048576.0, nil, nil, nil, nil, nil, nil, nil, 524288.0, nil, nil, nil, nil, 2104832.0, 1572864.0, nil, nil, 2097152.0, nil, nil, nil, nil, nil, nil, nil, nil, nil] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 106 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 99 insertions(+), 7 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..07533ce 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,107 @@ class ESJobs return 0 end - def output + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) result = { - 'stats.count' => @stats['stats.count'] + 'kvcount' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] + + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + end + end + + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = '---' + value.inspect + '---' + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + set_default_value(result, stats, metrics) + kvcount(result, job) end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @result['kvcount'] = @result['kvcount'].to_a.sort.to_h + @results = {} + compact_hash('', @result) + yaml = @results.to_yaml.delete_prefix!("---\n") + puts yaml.gsub('"---', '').gsub('---"', '') end end -- 2.23.0

1 0

[PATCH compass-ci 3/3] container - libvirt-helper: upload function
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/libvirt-helper/libvirt_app.rb | 37 +++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/libvirt-helper/libvirt_app.rb diff --git a/container/libvirt-helper/libvirt_app.rb b/container/libvirt-helper/libvirt_app.rb new file mode 100755 index 0000000..551dfc4 --- /dev/null +++ b/container/libvirt-helper/libvirt_app.rb @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'digest/md5' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['LIBVIRT_HELPER_PORT'] +end + +LIBVIRT_PATH = ENV['LIBVIRT_PATH'] +VERSION = '0.1' + +get '/' do + "version: #{VERSION}" +end + +post '/upload' do + if params[:file] && (tempfile = params[:file][:tempfile]) && (filename = params[:file][:filename]) + dirname = File.basename(filename, '.xml') + content = tempfile.read + md5_filename = Digest::MD5.hexdigest(content) + '.xml' + + dir_path = "#{LIBVIRT_PATH}/#{dirname}" + Dir.exist?(dir_path) || Dir.mkdir(dir_path) + + target = "#{dir_path}/#{md5_filename}" + File.open(target, 'wb') { |f| f.write content } + + File.join(dirname, md5_filename) + else + '' + end +end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/3] container/assist-cci: build & start script
by Ren Wen 15 Jan '21

15 Jan '21

Mount '/srv/cci' and upload config to dir: 'libvirt-xml'. Use port 'ASSIST_CCI_PORT' or '11392' by default. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/build | 5 +++++ container/assist-cci/start | 43 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) create mode 100755 container/assist-cci/build create mode 100755 container/assist-cci/start diff --git a/container/assist-cci/build b/container/assist-cci/build new file mode 100755 index 0000000..f82de7f --- /dev/null +++ b/container/assist-cci/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-cci . diff --git a/container/assist-cci/start b/container/assist-cci/start new file mode 100755 index 0000000..9ac31c1 --- /dev/null +++ b/container/assist-cci/start @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ASSIST_CCI_HOST + ASSIST_CCI_PORT +] + +defaults = relevant_defaults(names) + +ASSIST_CCI_PORT = defaults["ASSIST_CCI_PORT"] || 11392 +SRV_CCI_PATH = '/srv/cci' +LIBVIRT_XML_PATH = "#{SRV_CCI_PATH}/libvirt-xml" +DEFAULT_CCI = '/c/compass-ci' + +defaults['ASSIST_CCI_PORT'] = ASSIST_CCI_PORT +defaults['LIBVIRT_XML_PATH'] = LIBVIRT_XML_PATH +env = docker_env(defaults) + +docker_rm 'assist-cci' + +cmd = %w[ + docker run + --name assist-cci + --restart=always + -d +] + env + %W[ + -p #{ASSIST_CCI_PORT}:#{ASSIST_CCI_PORT} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v /etc/localtime:/etc/localtime:ro + -v #{SRV_CCI_PATH}:#{SRV_CCI_PATH} + -w #{DEFAULT_CCI}/container/assist-cci/ + assist-cci +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] container/assist-cci: Dockerfile
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/Dockerfile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 container/assist-cci/Dockerfile diff --git a/container/assist-cci/Dockerfile b/container/assist-cci/Dockerfile new file mode 100644 index 0000000..49007f7 --- /dev/null +++ b/container/assist-cci/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER Ren Wen <15991987063(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install activesupport json yaml sinatra puma -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: add package
by Liu Yinsi 15 Jan '21

15 Jan '21

[why] 1. providers/qemu/kvm.sh needs wget, curl, qemu command, so add it. 2. current debian 'qemu' package makes no sense anymore, and it is very unlikely one will need all of them together, so delete it, install qemu-system-ARCH packages, can see:https://packages.debian.org/sid/qemu Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 2 ++ sparrow/0-package/os/debian | 6 +++++- sparrow/0-package/os/openEuler | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 8df36c0..b25dade 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -22,6 +22,8 @@ pkgs=( busybox iproute hostname + wget + curl ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index c45e8c2..4d13413 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -13,13 +13,17 @@ pkgs=( docker.io ruby-full ruby-dev - qemu dmidecode lockfile-progs bridge-utils busybox iproute2 hostname + wget + curl + qemu-system-aarch64 + qemu-system-x86 + qemu-system-riscv64 ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 37c74d0..04996f5 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -22,6 +22,7 @@ pkgs=( iproute hostname libvirt + curl ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

2 2

[PATCH compass-ci 1/2] lib/error_messages.rb: obtain_error_messages_by_error_id function add keyword argument
by Lin Jiaxin 15 Jan '21

15 Jan '21

Determine whether to return the error line or error messages by the value of eturn_error_line Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 3e1564b..81e0034 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -40,12 +40,16 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_error_id(error_id) + def obtain_error_messages_by_error_id(error_id, return_error_line = false) error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| if ('build-pkg.' + build_pkg_error_id(k)) == error_id - error_messages_by_error_id += v.to_a + if return_error_line + error_messages_by_error_id << k + else + error_messages_by_error_id += v.to_a + end end end error_messages_by_error_id -- 2.23.0

2 1

[PATCH lkp-tests] lib/yaml.rb: load_yaml_merge add arg
by Wei Jihui 15 Jan '21

15 Jan '21

add template_context for load_yaml_merge, it is used by load_yaml. when template_context is nil, there is not impact for load_yaml_merge. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/yaml.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/yaml.rb b/lib/yaml.rb index 9c9b8088..4290d594 100755 --- a/lib/yaml.rb +++ b/lib/yaml.rb @@ -52,13 +52,13 @@ def load_yaml_with_flock(file, timeout = nil) end end -def load_yaml_merge(files) +def load_yaml_merge(files, template_context = nil) all = {} files.each do |file| next unless File.size? file begin - yaml = load_yaml(file) + yaml = load_yaml(file, template_context) all.update(yaml) rescue StandardError => e warn "#{e.class.name}: #{e.message.split("\n").first}: #{file}" -- 2.23.0

1 0

[PATCH v1 compass-ci] lib/job.cr: fix failed to upload package
by Cao Xueliang 15 Jan '21

15 Jan '21

[why] If the JSON::Any type pkg_style value is nil, we don't set the default upload path. [how] If the JSON::Any type pkg_style value is nil, we set it to empty hash that will get a default upload path. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1d25236..07bfda0 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -256,7 +256,9 @@ class Job pkg_style = @hash[item]? break if pkg_style end - return nil unless pkg_style && pkg_style != nil + return nil unless pkg_style + + pkg_style = JSON.parse("{}") if pkg_style == nil tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] -- 2.23.0

3 3

[PATCH compass-ci 2/2] lib/compare_error_messages.rb: add the credibility check of job
by Lin Jiaxin 15 Jan '21

15 Jan '21

defect: may match the wrong result error_id="build-pkg.pixz.c:warning:'subsuf'defined-but-not-used[-Wunused-function]" filename="pixz.c" correct result: gcc -DHAVE_CONFIG_H -I. -I.. -D_FORTIFY_SOURCE=2 -pthread -Wall -Wno-unknown-pragmas -march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -MT pixz-pixz.o -MD -MP -MF .deps/pixz-pixz.Tpo -c -o pixz-pixz.o `test -f 'pixz.c' || echo './'`pixz.c ^^^^^^ wrong result: mv -f .deps/pixz.cpu.Tpo .deps/pixz.cpu.Po ^^^^^^ Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 0b51fc3..a959c61 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -8,6 +8,39 @@ require_relative 'es_query' require_relative 'error_messages' require_relative "#{ENV['LKP_SRC']}/lib/common" +# previous_job_id's error_ids include error_id +# If later_job_id's output include error filename that extract from error_id, +# the later_job_id is credible. +def credible?(previous_job_id, later_job_id, error_id) + es = ESQuery.new + + previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') + later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') + + return false if filenames_check(previous_result_file, later_result_file, error_id).value?(false) + + return true +end + +def filenames_check(previous_result_file, later_result_file, error_id) + filenames = [] + filenames_check = Hash.new { |h, k| h[k] = false } + + error_lines = ErrorMessages.new(previous_result_file).obtain_error_messages_by_error_id(error_id, true) + error_lines.each do |error_line| + filenames << $1 if error_line =~ /(.*)(:.*){2}: (error|warning):/ + end + + File.open(later_result_file).each_line do |line| + filenames.each do |filename| + filenames_check[filename] + filenames_check[filename] = true if line.include?(filename) + end + end + + return filenames_check +end + def get_compare_result(previous_job_id, later_job_id) es = ESQuery.new -- 2.23.0

2 1

[PATCH lab-z9] add a117 host info
by Bai Jing 15 Jan '21

15 Jan '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- hosts/taishan200-2280-2s64p-256g--a117 | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a117 diff --git a/hosts/taishan200-2280-2s64p-256g--a117 b/hosts/taishan200-2280-2s64p-256g--a117 new file mode 100644 index 0000000..0f10495 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a117 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b57d1 + - /dev/disk/by-id/scsi-350000399886b720d +mac_addr: + - 44:67:47:85:f4:0b + - 44:67:47:85:f4:0c + - 44:67:47:85:f4:0d + - 44:67:47:85:f4:0e + - cc:05:77:ee:b3:ff + - cc:05:77:ee:b4:00 + - cc:05:77:ee:b4:01 + - cc:05:77:ee:b4:02 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.10 -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container - assist-cci: build & start script
by Ren Wen 15 Jan '21

15 Jan '21

Mount '/srv/cci' and upload config to dir: 'libvirt-xml'. Use port 'ASSIST_CCI_PORT' or '11392' by default. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/build | 5 ++++ container/assist-cci/start | 47 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) create mode 100755 container/assist-cci/build create mode 100755 container/assist-cci/start diff --git a/container/assist-cci/build b/container/assist-cci/build new file mode 100755 index 0000000..f82de7f --- /dev/null +++ b/container/assist-cci/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-cci . diff --git a/container/assist-cci/start b/container/assist-cci/start new file mode 100755 index 0000000..b65f814 --- /dev/null +++ b/container/assist-cci/start @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ASSIST_CCI_HOST + ASSIST_CCI_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +ASSIST_CCI_PORT = defaults['ASSIST_CCI_PORT'] || 11392 +SRV_CCI_PATH = '/srv/cci' +LIBVIRT_XML_PATH = "#{SRV_CCI_PATH}/libvirt-xml" +docker_rm 'assist-cci' + +cmd = %w[ + docker run + --name assist-cci + --restart=always + -d +] + env + %W[ + -e LIBVIRT_XML_PATH=#{LIBVIRT_XML_PATH} + -e ASSIST_CCI_PORT=#{ASSIST_CCI_PORT} + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -p #{ASSIST_CCI_PORT}:#{ASSIST_CCI_PORT} + -v /etc/localtime:/etc/localtime:ro + -v #{SRV_CCI_PATH}:#{SRV_CCI_PATH} + -w #{DEFAULT_CCI}/container/assist-cci/ + assist-cci +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

2 2

[PATCH v2 compass-ci 1/3] container - assist-cci: Dockerfile
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/Dockerfile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 container/assist-cci/Dockerfile diff --git a/container/assist-cci/Dockerfile b/container/assist-cci/Dockerfile new file mode 100644 index 0000000..16bba06 --- /dev/null +++ b/container/assist-cci/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER 15991987063(a)163.com + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install activesupport json yaml sinatra puma -- 2.23.0

3 4

[PATCH compass-ci] auto_submit: don't submit job with too old commit
by Li Yuanchao 15 Jan '21

15 Jan '21

If the last commit of a repository is too old, such as 'two months ago', that means the repository is not active. We don't care such repositories. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index ecc4d70..5648d59 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -31,6 +31,9 @@ class AutoSubmit def submit(newrefs_info, submit_argv) newrefs_info['new_refs']['heads'].each do |branch, commit_id| + inactive_time = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git log --pretty=format:"%cr" -1 #{commit_id}) + next if inactive_time =~ /(month|year)/ + real_argvs = Array.new(submit_argv) real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") real_argvs.push("upstream_commit=#{commit_id}") -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: fix "build-pkg" value is nil can't upload package
by Cao Xueliang 15 Jan '21

15 Jan '21

[why] If the build-pkg value is nil, we didn't set the default upload path. [how] If the build-pkg value is nil, we set it to empty hash, that will get a default upload path. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1d25236..07bfda0 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -256,7 +256,9 @@ class Job pkg_style = @hash[item]? break if pkg_style end - return nil unless pkg_style && pkg_style != nil + return nil unless pkg_style + + pkg_style = JSON.parse("{}") if pkg_style == nil tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] -- 2.23.0

3 4

[PATCH compass-ci] delimiter: create a delimiter account when start delimiter container
by Cao Xueliang 15 Jan '21

15 Jan '21

Before: Use compass-ci-robot(a)qq.com as the bisect job subqueue, wait to consume will take a long time. After: User delimiter@localhost as the bisect job subqueue, wait to consume will take a short time. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/Dockerfile | 2 +- container/delimiter/start | 2 ++ src/delimiter.rb | 4 ++++ src/delimiter/constants.rb | 3 ++- src/delimiter/utils.rb | 6 +++--- 5 files changed, 12 insertions(+), 5 deletions(-) diff --git a/container/delimiter/Dockerfile b/container/delimiter/Dockerfile index 93c52c6..ab7519d 100644 --- a/container/delimiter/Dockerfile +++ b/container/delimiter/Dockerfile @@ -10,4 +10,4 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN umask 002 && \ echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ - gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket terminal-table mail + gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket terminal-table mail io-console diff --git a/container/delimiter/start b/container/delimiter/start index 669923b..2a68df3 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -7,6 +7,8 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ + ES_HOST + ES_PORT TASKQUEUE_HOST TASKQUEUE_PORT ] diff --git a/src/delimiter.rb b/src/delimiter.rb index bd40775..898fa0e 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -2,9 +2,13 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require_relative '../lib/build_my_info_client' require_relative './delimiter/delimiter' begin + build_my_info = BuildMyInfo.new(DELIMITER_EMAIL, DELIMITER_NAME, ENV['lab']) + build_my_info.store_account_info + delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 7b14558..23b5053 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -6,7 +6,8 @@ LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' GIT_MIRROR_HOST = ENV['GIT_MIRROR_HOST'] || '172.17.0.1' MONITOR_HOST = ENV['MONITOR_HOST'] || '172.17.0.1' MONITOR_PORT = ENV['MONITOR_PORT'] || '11310' -DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'caoxl78320(a)163.com' +DELIMITER_EMAIL = ENV['DELIMITER_EMAIL'] || 'delimiter@localhost' +DELIMITER_NAME = ENV['DELIMITER_NAME'] || 'delimiter' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 61fe6ab..aaad394 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -118,17 +118,17 @@ module Utils end def get_account_info - ESQuery.new(index: 'accounts').query_by_id(DELIMITER_ACCONUT) + ESQuery.new(index: 'accounts').query_by_id(DELIMITER_EMAIL) end def init_job_content(job_id) job_yaml = AssistResult.new.get_job_yaml(job_id) - raise "get job yaml failed, job id: #{job_id} !" unless job_yaml + raise "get job yaml failed, job id: #{job_id}" unless job_yaml job = JSON.parse job_yaml account_info = get_account_info - raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info + raise "query delimiter account info failed: #{DELIMITER_EMAIL}" unless account_info record_jobs(job['id'], job['upstream_commit']) -- 2.23.0

2 6

[PATCH v2 compass-ci 3/3] container - assist-cci: upload function
by Ren Wen 15 Jan '21

15 Jan '21

Upload an XML template of libvirt. Response is the modified filename. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/routes.rb | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/assist-cci/routes.rb diff --git a/container/assist-cci/routes.rb b/container/assist-cci/routes.rb new file mode 100755 index 0000000..0e71a75 --- /dev/null +++ b/container/assist-cci/routes.rb @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'digest/md5' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_CCI_PORT'] +end + +LIBVIRT_XML_PATH = ENV['LIBVIRT_XML_PATH'] +VERSION = '0.1' + +get '/' do + "version: #{VERSION}" +end + +post '/upload' do + if params[:file] && (tempfile = params[:file][:tempfile]) && (filename = params[:file][:filename]) + dirname = File.basename(filename, '.xml') + content = tempfile.read + md5_filename = Digest::MD5.hexdigest(content) + '.xml' + + dir_path = "#{LIBVIRT_XML_PATH}/#{dirname}" + Dir.exist?(dir_path) || Dir.mkdir(dir_path) + + target = "#{dir_path}/#{md5_filename}" + File.open(target, 'wb') { |f| f.write content } + + File.join(dirname, md5_filename) + else + '' + end +end -- 2.23.0

1 0

[PATCH compass-ci] scheduler: change 'templates' to 'vt'
by Ren Wen 15 Jan '21

15 Jan '21

Use 'vt' (virt) instead of 'templates', the latter is not clear enough. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 625c74c..152d549 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -181,7 +181,7 @@ class Sched "kernel_params" => _kernel_params, "result_root" => job.result_root, "LKP_SERVER" => job["LKP_SERVER"]?, - "templates" => job["templates"]?, + "vt" => job["vt"]?, "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", }.to_json end -- 2.23.0

1 0

[PATCH v4 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archLinux [output] --- kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archLinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 sum.stats.last_state.test.iperf.exit_code.127: 12 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: "[524288.0, nil, nil, nil, nil, nil, nil, nil, nil, 524288.0, 524288.0, nil, nil, nil, nil, nil, 516608.0, nil, 1048576.0, nil, nil, nil, nil, nil, nil, nil, 524288.0, nil, nil, nil, nil, 2104832.0, 1572864.0, nil, nil, 2097152.0, nil, nil, nil, nil, nil, nil, nil, nil, nil]" ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 105 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 98 insertions(+), 7 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..90e4302 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,106 @@ class ESJobs return 0 end - def output + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) result = { - 'stats.count' => @stats['stats.count'] + 'kvcount' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] + + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + end + end + + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = value.to_s + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + set_default_value(result, stats, metrics) + kvcount(result, job) end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @result['kvcount'] = @result['kvcount'].to_a.sort.to_h + @results = {} + compact_hash('', @result) + puts @results.to_yaml end end -- 2.23.0

2 1

[PATCH compass-ci] scheduler: boot.libvirt: response 'SRV_HTTP_PORT'
by Ren Wen 15 Jan '21

15 Jan '21

For caller to download libvirt config from '/srv/cci/libvirt-xml'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/scheduler/start | 2 ++ src/scheduler/constants.cr | 2 ++ src/scheduler/find_job_boot.cr | 1 + 3 files changed, 5 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 70a3496..d8f10dc 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -25,6 +25,7 @@ names = Set.new %w[ SSHR_PORT SSHR_PORT_BASE SSHR_PORT_LEN + SRV_HTTP_PORT lab ] @@ -35,6 +36,7 @@ defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' +defaults['SRV_HTTP_PORT'] ||= '11300' env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1146b21..377ea72 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -29,6 +29,8 @@ LAB_REPO = "lab-z9" SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" +SRV_HTTP_PORT = (ENV.has_key?("SRV_HTTP_PORT") ? ENV["SRV_HTTP_PORT"] : "11300") + INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 625c74c..3a8be59 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -183,6 +183,7 @@ class Sched "LKP_SERVER" => job["LKP_SERVER"]?, "templates" => job["templates"]?, "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", + "SRV_HTTP_PORT" => SRV_HTTP_PORT }.to_json end -- 2.23.0

1 0

[PATCH compass-ci 2/3] container - libvirt-helper: build & start script
by Ren Wen 15 Jan '21

15 Jan '21

Mount '/srv/cci/libvirt-xml' and upload config. Use port 11392 by defaut. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/libvirt-helper/build | 5 ++++ container/libvirt-helper/start | 46 ++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) create mode 100755 container/libvirt-helper/build create mode 100755 container/libvirt-helper/start diff --git a/container/libvirt-helper/build b/container/libvirt-helper/build new file mode 100755 index 0000000..1d216ea --- /dev/null +++ b/container/libvirt-helper/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t libvirt-helper . diff --git a/container/libvirt-helper/start b/container/libvirt-helper/start new file mode 100755 index 0000000..6222eaa --- /dev/null +++ b/container/libvirt-helper/start @@ -0,0 +1,46 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LIBVIRT_HELPER_HOST + LIBVIRT_HELPER_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +LIBVIRT_HELPER_PORT = defaults['LIBVIRT_HELPER_PORT'] || 11392 +LIBVIRT_PATH = '/srv/cci/libvirt-xml' +docker_rm 'libvirt-helper' + +cmd = %w[ + docker run + --name libvirt-helper + --restart=always + -d +] + env + %W[ + -e LIBVIRT_PATH=#{LIBVIRT_PATH} + -e LIBVIRT_HELPER_PORT=#{LIBVIRT_HELPER_PORT} + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -p #{LIBVIRT_HELPER_PORT}:#{LIBVIRT_HELPER_PORT} + -v /etc/localtime:/etc/localtime:ro + -v #{LIBVIRT_PATH}:#{LIBVIRT_PATH} + -w #{DEFAULT_CCI}/container/libvirt-helper/ + libvirt-helper +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./libvirt_app.rb'] + +system(*cmd) -- 2.23.0

2 2

[PATCH v3 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430 [output] --- kvcount.suite=iperf: 2 kvcount.os=openeuler: 2 kvcount.arch=aarch64: 2 kvcount.job_state=finished: 2 kvcount.tbox_group=vm-2p16g: 2 kvcount.submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430: 2 sum.stats.kmsg.timestamp:last: 2 sum.stats.dmesg.timestamp:last: 2 raw.stats.kmsg.timestamp:last: "[75.256233, 48.990905]" raw.stats.iperf.tcp.receiver.bps: "[34017924155.510155, nil]" raw.stats.dmesg.timestamp:last: "[75.256233, 48.990905]" raw.stats.iperf.tcp.sender.bps: "[34073687935.01113, nil]" raw.stats.iperf.udp.bps: "[nil, 1048573.3083402428]" avg.stats.iperf.tcp.receiver.bps: 34017924155.510155 avg.stats.iperf.tcp.sender.bps: 34073687935.01113 avg.stats.iperf.udp.bps: 1048573.3083402428 max.stats.iperf.tcp.receiver.bps: 34017924155.510155 max.stats.iperf.tcp.sender.bps: 34073687935.01113 max.stats.iperf.udp.bps: 1048573.3083402428 min.stats.iperf.tcp.receiver.bps: 34017924155.510155 min.stats.iperf.tcp.sender.bps: 34073687935.01113 min.stats.iperf.udp.bps: 1048573.3083402428 Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 109 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 100 insertions(+), 9 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..31a57a9 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,106 @@ class ESJobs return 0 end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) + result = {} + result['kvcount'] = {} + result['sum.stats'] = {} + result['raw.stats'] = {} + result['avg.stats'] = {} + result['max.stats'] = {} + result['min.stats'] = {} + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + job_metrics = stats.keys + left_metrics = metrics - job_metrics + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = value.to_s + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end + + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @results = {} + compact_hash('', @result) + puts @results.to_yaml + #puts JSON.pretty_generate(@results) end end -- 2.23.0

2 1

[PATCH compass-ci] container/delimiter: auto push image to local docker hub
by Cao Xueliang 15 Jan '21

15 Jan '21

Before: Manual push image to local docker hub. After: Auto push image to local docker hub, others can use the latest image after deploy. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/build | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/container/delimiter/build b/container/delimiter/build index fc1af95..7d9b437 100755 --- a/container/delimiter/build +++ b/container/delimiter/build @@ -2,4 +2,20 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +load_cci_defaults + docker build -t delimiter . + +push_image() +{ + local local_docker_hub="$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT" + local src_tag=delimiter:latest + local dst_tag="$local_docker_hub/$src_tag" + + docker tag "$src_tag" "$dst_tag" + docker push "$dst_tag" +} + +push_image -- 2.23.0

1 1

[PATCH v1 compass-ci] container/delimiter: auto push image to local docker hub
by Cao Xueliang 15 Jan '21

15 Jan '21

Before: Manual push image to local docker hub. After: Auto push image to local docker hub, others can use the latest image after deploy. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/defconfig.sh | 10 ++++++++++ container/delimiter/build | 6 ++++++ 2 files changed, 16 insertions(+) diff --git a/container/defconfig.sh b/container/defconfig.sh index 97d924d..4a5f5fa 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -31,3 +31,13 @@ set_es_indices() { find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; } + +push_image() +{ + local local_docker_hub="$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT" + local src_tag=$1 + local dst_tag="$local_docker_hub/$src_tag" + + docker tag "$src_tag" "$dst_tag" + docker push "$dst_tag" +} diff --git a/container/delimiter/build b/container/delimiter/build index fc1af95..f7f53e7 100755 --- a/container/delimiter/build +++ b/container/delimiter/build @@ -2,4 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. ../defconfig.sh + +load_cci_defaults + docker build -t delimiter . + +push_image delimiter:latest -- 2.23.0

1 0

[PATCH compass-ci 1/3] container - libvirt-helper: Dockerfile
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/libvirt-helper/Dockerfile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 container/libvirt-helper/Dockerfile diff --git a/container/libvirt-helper/Dockerfile b/container/libvirt-helper/Dockerfile new file mode 100644 index 0000000..638ebd4 --- /dev/null +++ b/container/libvirt-helper/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER 15991987063(a)163.com + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of output results [example] es-jobs submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430 [output] {"kvcount.suite=iperf"=>2, "kvcount.os=openeuler"=>2, "kvcount.arch=aarch64"=>2, "kvcount.job_state=finished"=>2, "kvcount.tbox_group=vm-2p16g"=>2, "kvcount.submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430"=>2, "sum.stats.kmsg.timestamp:last"=>2, "sum.stats.dmesg.timestamp:last"=>2, "raw.stats.kmsg.timestamp:last"=>[75.256233, 48.990905], "raw.stats.iperf.tcp.receiver.bps"=>[34017924155.510155, nil], "raw.stats.dmesg.timestamp:last"=>[75.256233, 48.990905], "raw.stats.iperf.tcp.sender.bps"=>[34073687935.01113, nil], "raw.stats.iperf.udp.bps"=>[nil, 1048573.3083402428], "avg.stats.iperf.tcp.receiver.bps"=>34017924155.510155, "avg.stats.iperf.tcp.sender.bps"=>34073687935.01113, "avg.stats.iperf.udp.bps"=>1048573.3083402428, "max.stats.iperf.tcp.receiver.bps"=>34017924155.510155, "max.stats.iperf.tcp.sender.bps"=>34073687935.01113, "max.stats.iperf.udp.bps"=>1048573.3083402428, "min.stats.iperf.tcp.receiver.bps"=>34017924155.510155, "min.stats.iperf.tcp.sender.bps"=>34073687935.01113, "min.stats.iperf.udp.bps"=>1048573.3083402428} Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 115 +++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 102 insertions(+), 13 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..754578d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,6 +2,7 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -18,10 +19,10 @@ class ESJobs @refine_jobs = [] @jobs = {} @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' + 0 => 'success', + 1 => 'unknown', + 2 => 'warning', + 3 => 'has_error' } set_defaults deal_jobs @@ -123,17 +124,105 @@ class ESJobs return 0 end - def output - result = { - 'stats.count' => @stats['stats.count'] - } + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) + result = {} + result['kvcount'] = {} + result['sum.stats'] = {} + result['raw.stats'] = {} + result['avg.stats'] = {} + result['max.stats'] = {} + result['min.stats'] = {} + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + job_metrics = stats.keys + left_metrics = metrics - job_metrics + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 end + end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats + + set_default_value(result, stats, metrics) + kvcount(result, job) + end + + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @results = {} + compact_hash('', @result) + pp @results + #puts JSON.pretty_generate(@results) end end -- 2.23.0

2 1

[PATCH compass-ci] sbin/run_job: kill process after running qemu/docker
by Hu Xuejiao 15 Jan '21

15 Jan '21

[why] 1.host-info is adapted in physical machine, so it replaced to ebizzy.yaml 2.after running multi-docker and multi-qemu, these processes need to be deleted Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sbin/run_job b/sbin/run_job index 325fa9f..24d35cc 100755 --- a/sbin/run_job +++ b/sbin/run_job @@ -24,7 +24,7 @@ submit_one_job() submit_job() { - yaml_args=("$cci_makepkg_file" "host-info.yaml") + yaml_args=("$cci_makepkg_file" "ebizzy.yaml") for yaml in "${yaml_args[@]}" do submit_one_job & @@ -75,3 +75,6 @@ run_docker() run_qemu run_docker wait + +kill $(cat pid) +kill $(cat $CCI_SRC/providers/dc.pid) -- 2.23.0

2 1

[PATCH v4 compass-ci] tool/es-jobs: a new tool for summarizing jobs information
by Wu Zhende 15 Jan '21

15 Jan '21

Use es-jobs to summarizes information about a group of related jobs. [Example] es-jobs submit_id=xxx -f 'job_state,dmesg.timestamp:last' -r '-1' -s 'on_fail' means: 1. query the job whose submit_id is xxx 2. I want to get job_state,dmesg.timestamp:last 3. don't refine jobs 4. -s 'on_fail' to get data in stats which contain 'on_fail' [Output] { "job_state": [ "crystal.605716.failed" ], "dmesg.timestamp:last": [ 657.181408 ], "stats_filter_result": { "crystal.605716.openeuler_docker.\u001b[0mwordpress_build_on_fail": 1 }, "stats.count": { "stats.has_error": 1, "stats.has_error_jobs": [ "crystal.605716" ] } } Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 139 +++++++++++++++++++++++++++++++++++++++++++++++++ sbin/es-jobs | 55 +++++++++++++++++++ 2 files changed, 194 insertions(+) create mode 100644 lib/es_jobs.rb create mode 100755 sbin/es-jobs diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb new file mode 100644 index 0000000..a90b1ba --- /dev/null +++ b/lib/es_jobs.rb @@ -0,0 +1,139 @@ +# SPDX-License-Identifier: GPL-2.0-only +# frozen_string_literal: true + +LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' + +require "#{LKP_SRC}/lib/stats" +require_relative './es_query' + +# deal jobs search from es +class ESJobs + def initialize(es_query, my_refine = [], fields = [], stats_filter = []) + @es_query = es_query + @es = ESQuery.new(ES_HOST, ES_PORT) + @refine = my_refine + @fields = fields + @stats_filter = stats_filter + @stats_filter_result = {} + @refine_jobs = [] + @jobs = {} + @stats_level = { + 0 => 'stats.success', + 1 => 'stats.unknown', + 2 => 'stats.warning', + 3 => 'stats.has_error' + } + set_defaults + deal_jobs + end + + def set_defaults + query_result = @es.multi_field_query(@es_query) + query_result['hits']['hits'].each do |job| + @jobs[job['_id']] = job['_source'] + end + + @stats = { + 'stats.count' => Hash.new(0), + 'stats.sum' => Hash.new(0), + 'stats.avg' => Hash.new(0) + } + @result = {} + @fields.each do |field| + @result[field] = [] + end + end + + def add_result_fields(job, level) + return unless @refine.include?(level) || @refine.include?(-1) + + @refine_jobs << job['id'] + @fields.each do |field| + value = job[field] + if value + value = job['id'] + '.' + value if field == 'job_state' + @result[field] << value + end + + next unless job['stats'] + + @result[field] << job['stats'][field] if job['stats'][field] + end + end + + def deal_jobs + stats_count = Hash.new(0) + stats_jobs = {} + + @jobs.each do |job_id, job| + level = deal_stats(job) + add_result_fields(job, level) + + stat_key = @stats_level[level] + stat_jobs_key = stat_key + '_jobs' + + stats_count[stat_key] += 1 + stats_jobs[stat_jobs_key] ||= [] + stats_jobs[stat_jobs_key] << job_id + end + + @stats['stats.count'].merge!(stats_count) + @stats['stats.count'].merge!(stats_jobs) + end + + def deal_stats(job, level = 0) + return 1 unless job['stats'] + + job['stats'].each do |key, value| + match_stats_filter(key, value, job['id']) + calculate_stat(key, value) + level = get_stat_level(key, level) + end + return level + end + + def match_stats_filter(key, value, job_id) + @stats_filter.each do |filter| + next unless key.include?(filter) + + key = job_id + '.' + key + @stats_filter_result[key] = value + + break + end + end + + def calculate_stat(key, value) + if function_stat?(key) + return unless @fields.include?('stats.sum') + + @stats['stats.sum'][key] += value + else + return unless @fields.include?('stats.avg') + + @stats['stats.avg'][key] = (@stats['stats.avg'][key] + value) / 2 + end + end + + def get_stat_level(stat, level) + return level if level >= 3 + return 3 if stat.match(/error|fail/i) + return 2 if stat.match(/warn/i) + + return 0 + end + + def output + result = { + 'stats.count' => @stats['stats.count'] + } + + @stats.each do |key, value| + result[key] = value if @fields.include?(key) + end + + @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? + @result.merge!(result) + puts JSON.pretty_generate(@result) + end +end diff --git a/sbin/es-jobs b/sbin/es-jobs new file mode 100755 index 0000000..c055fe9 --- /dev/null +++ b/sbin/es-jobs @@ -0,0 +1,55 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'optparse' +require 'json' + +require_relative '../lib/es_jobs' + +def parse_argv + items = {} + ARGV.each do |item| + key, value = item.split('=') + if key && value + value_list = value.split(',') + items[key] = value_list.length > 1 ? value_list : value + end + end + items +end +opt_refine = [-1] +opt_fields = [] +opt_stats_filter = [] +opt_parser = OptionParser.new do |opts| + opts.banner = 'Usage: es-jobs [options] search_key1=val1[,val2..] ..' + + opts.separator 'search_key can be submit_id, group_id' + opts.separator 'How to use -r' + opts.separator 'Like es-jobs submit_id=xxx -r "0,1,2,3"' + opts.separator '-1 means not refine, is the default value' + opts.separator ' 0 means stats.succes, refine the jobs whose test cases are all successfully executed' + opts.separator ' 1 means stats.unknown, refine the jobs without the stats' + opts.separator ' 2 means stats.warning, refine the jobs with warnings in tese cases' + opts.separator ' 3 means stats.has_error, refine the jobs with errors in tese cases' + + opts.on('-r fields', '--refine fields', 'refine jobs') do |fields| + opt_refine = fields.split(',').map(&:to_i) + end + + opts.on('-f fields', '--fields fields', 'fields you want to see') do |fields| + opt_fields = fields.split(',') + end + + opts.on('-s fields', '--stats-filter fields', 'return data contains fields in stats') do |fields| + opt_stats_filter = fields.split(',') + end +end + +opt_parser.parse!(ARGV) +items = parse_argv +raise 'Please enter a query' if items.empty? + +es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter) +es_jobs.output -- 2.23.0

2 7

[PATCH compass-ci] use cd "$xxx" for safety reason
by Liu Yinsi 15 Jan '21

15 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/docker2rootfs/common | 4 ++-- sparrow/4-docker/buildall | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index e908079..c26bbc8 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -96,7 +96,7 @@ load_docker_img() cp_rootfs() { docker cp -a rootfs-docker:/tmp/$1 /tmp/ - cd $2 + cd "$2" zcat /tmp/$1 |cpio -idmv } @@ -148,7 +148,7 @@ create_links_vmlinuz_initrd() create_get_initrd echo "Creating links to initrd.lkp and vmlinuz..." - cd $ROOTFS_DIR + cd "$ROOTFS_DIR" ln -fs $ROOTFS_INITRD_LKP initrd.lkp cd $ROOTFS_DIR/boot ln -fs $ROOTFS_VMLINUZ vmlinuz diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index ca4406b..f24226c 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -30,7 +30,7 @@ do_one() [ -n "$LKP_SRC" ] && log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( - cd $container + cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then -- 2.23.0

2 4

[PATCH compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of output results [example] es-jobs submit_id=a7c2f144-aa64-4a23-a390-cfe5bca3b430 [output before] { "stats.count": { "stats.unknown": 2, "stats.success": 2, "stats.unknown_jobs": [ "crystal.613986", "crystal.613985" ], "stats.success_jobs": [ "crystal.597093", "crystal.597090" ] } } [output after] { "stats.count.unknown": 2, "stats.count.success": 2, "stats.count.unknown_jobs": [ "crystal.613986", "crystal.613985" ], "stats.count.success_jobs": [ "crystal.597093", "crystal.597090" ] } Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..c536fe3 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -18,10 +18,10 @@ class ESJobs @refine_jobs = [] @jobs = {} @stats_level = { - 0 => 'stats.success', - 1 => 'stats.unknown', - 2 => 'stats.warning', - 3 => 'stats.has_error' + 0 => 'success', + 1 => 'unknown', + 2 => 'warning', + 3 => 'has_error' } set_defaults deal_jobs @@ -123,6 +123,22 @@ class ESJobs return 0 end + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + def output result = { 'stats.count' => @stats['stats.count'] @@ -134,6 +150,8 @@ class ESJobs @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? @result.merge!(result) - puts JSON.pretty_generate(@result) + @results = {} + compact_hash('', @result) + puts JSON.pretty_generate(@results) end end -- 2.23.0

2 3

[PATCH compass-ci] scheduler: fix "no job now" when consuming jobs within docker
by Ren Wen 14 Jan '21

14 Jan '21

[Error] "message":"Missing required job key: 'my_token'. [Why] The field 'ipxe_kernel_params' doesn't need to be initialized when consuming the job. [How] Remove 'ipxe_kernel_params' from 'initialized_keys'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8377bab..1d25236 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -427,6 +427,7 @@ class Job "kernel_version", "kernel_uri", "kernel_params", + "ipxe_kernel_params", "linux_vmlinuz_path"] end -- 2.23.0

1 1

[PATCH v2 compass-ci] scheduler: fix "no job now" when consuming jobs within docker
by Ren Wen 14 Jan '21

14 Jan '21

[Error] "message":"Missing required job key: 'my_token'. [Why] The field 'ipxe_kernel_params' doesn't need to be initialized when consuming the job, otherwise, will do the account check. [How] Remove 'ipxe_kernel_params' from 'initialized_keys'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8377bab..1d25236 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -427,6 +427,7 @@ class Job "kernel_version", "kernel_uri", "kernel_params", + "ipxe_kernel_params", "linux_vmlinuz_path"] end -- 2.23.0

1 0

[PATCH v7 lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 14 Jan '21

14 Jan '21

[Error Msg] LKP: HOSTNAME vm-2p8g.2288hv5-2s44p-384g--b5-1573861, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [Root Cause] [ 93.815004] + network_ok [ 93.874456] ++ cat /sys/class/net/eth0//operstate [ 93.879278] + '[' up = up ']' [ 93.882356] + return 0 [ 93.941910] + export_ip_mac [ 94.018548] + has_cmd ip [ 94.022832] + command -v ip [ 94.029742] ++ awk '{print $5; exit}' [ 94.036137] ++ ip route get 1.2.3.4 [ 94.045622] ++ /usr/bin/mawk '{print $5; exit}' [ 94.056504] RTNETLINK answers: Network is unreachable [ 94.061418] + export PUB_NIC= [ 94.064587] + PUB_NIC= [Why] even network interfaces ready, but routing not, get ip by routing failed, so add a check export_ip_mac to ensure network is ok. [how] before: network interfaces "up"|"1" -> network is ok after: network interfaces "up"|"1" && export_ip_mac ok -> network is ok Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index b04ea3dd4..2969544a6 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -91,8 +91,8 @@ network_ok() for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && export_ip_mac && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && export_ip_mac && return 0 done is_clearlinux && { @@ -296,6 +296,8 @@ export_ip_mac() export PUB_IP=$(ifconfig $PUB_NIC | awk '/inet / {print $2}') fi + [ -n "$PUB_NIC" ] || return 1 + export PUB_MAC=$(cat /sys/class/net/$PUB_NIC/address) [ -n "$SCHED_HOST" ] && PUB_MAC=$(echo "$PUB_MAC" | tr : -) } @@ -746,7 +748,6 @@ boot_init() setup_network run_ntpdate - export_ip_mac announce_bootup -- 2.23.0

4 8

help star
by Wu Fengguang 14 Jan '21

14 Jan '21

Xueliang is our help star, thanks Xueliang! caoxl@crystal ~/code/todo/bin% ./help-star ../people/*/credit/2021-01-14.md 24.6 Wu Fengguang 16.2 Cao Xueliang 16.1 Yu Chuan 12.2 Luan Shengde 11.8 Wei Jihui 11.7 Liu Shaofei 10.2 Zhang Dewan 8.9 Xiao Shenwei 8.5 Zhang Yale 8.3 Liu Yinsi 7.6 Wu Zhende 7.2 Ren Wen 6.9 Wang Yong 6.5 Lu Weitao 5.2 Lin Jiaxin 5.0 Li Ping 4.7 Bai Jing 4.6 Xu Xijian 4.3 Cui Yili 2.4 Li Yuanchao 2.4 Wang Chenglong 2.2 Du Kaitian 1.3 Lu Kaiyi 1.1 Hu Xuejiao 0.0 Tong Qunfeng 0.0 Wang Shuai 0.0 Cui Xucui

1 0

[PATCH v5 lkp-tests] stats: add ansible_test stats script
by Zhang Yale 14 Jan '21

14 Jan '21

[Why] Output needs to be displayed in the stats.json file, so this output need to processed as a key-value pair. [input] PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 playbook_run_on_fail: 1 [output] ansible_test.total.nr_ok: 2 ansible_test.total.nr_changed: 0 ansible_test.total.nr_failed: 1 ansible_test.total.nr_skipped: 0 ansible_test.total.nr_rescued: 0 ansible_test.total.nr_ignored: 0 playbook_run_on_fail: 1 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 stats/ansible_test diff --git a/stats/ansible_test b/stats/ansible_test new file mode 100755 index 00000000..4c7d07b3 --- /dev/null +++ b/stats/ansible_test @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + case line.chomp! + when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ + puts "ansible_test.total.nr_ok: #{$1}" + puts "ansible_test.total.nr_changed: #{$2}" + puts "ansible_test.total.nr_failed: #{$4}" + puts "ansible_test.total.nr_skipped: #{$5}" + puts "ansible_test.total.nr_rescued: #{$6}" + puts "ansible_test.total.nr_ignored: #{$7}" + when /playbook_run_on_fail/ + puts line + end +end -- 2.23.0

3 4

review star
by Wu Fengguang 14 Jan '21

14 Jan '21

Jihui is our review star, thanks Jihui! caoxl@crystal ~/code/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 98 32 52 20 Wei Jihui 71 44 125 19 Wu Fengguang 24 13 23 11 Liu Shaofei 59 16 21 9 Cao Xueliang 12 10 31 8 Yu Chuan 41 10 20 7 Lu Kaiyi 33 7 13 5 Ren Wen 16 7 8 5 Cui Yili 16 6 7 4 Bai Jing 17 3 3 4 Wu Zhende 11 6 10 3 Lin Jiaxin 9 4 7 3 Li Yuanchao 6 4 8 3 Xiao Shenwei 18 6 8 3 Wang Yong 19 6 10 2 Wang Chenglong 14 4 8 2 Lu Weitao 20 4 6 2 Li Ping 29 3 4 2 Xu Xijian 55 4 8 1 Luan Shengde 20 2 2 1 Hu Xuejiao 6 0 0 0 Zhang Dewan 51 1 2 0 Liu Yinsi 30 3 2 0 Zhang Yale

1 0

git count
by Wu Fengguang 14 Jan '21

14 Jan '21

Yinsi is our code star, thanks Yinsi! 12 Liu Yinsi 10 Luan Shengde 9 Lu Kaiyi 8 Hu Xuejiao 5 Cao Xueliang 5 Xu Xijian 5 Ren Wen 4 Lu Weitao 4 Wei Jihui 4 Wu Fengguang 4 Wu Zhende 3 Liu Shaofei 3 Wang Chenglong 3 Li Yuanchao 2 Bai Jing 2 Xiao Shenwei 2 Zhang Yale 2 Cui Yili 2 Wang Yong 2 Lin Jiaxin 1 Yu Chuan 0 Tong Qunfeng 0 Du Kaitian 0 Sun Yukui 0 Zhang Yu 0 Zhang Dewan 0 Li Ping 0 Shi Zhichao

1 0

[PATCH compass-ci 1/3] sbin/auto_submit: fix failed to submit job in container
by Liu Yinsi 14 Jan '21

14 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. [how] store auto submit info to ES to pass verification when submit jobs in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..c121ae0 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -6,6 +6,7 @@ require 'bunny' require 'json' require 'yaml' +require_relative '../lib/build_my_info_client' # receive message and auto submit job class AutoSubmit @@ -82,5 +83,15 @@ class AutoSubmit end end +option = { + my_name: 'Auto Submit Robot', + my_email: 'autosubmit@localhost', + lab: 'autosubmit' +} + +# fix verification problems when submit jobs by auto-submit service +build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) +build_my_info.config_my_info + auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

2 2

[PATCH v5 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 14 Jan '21

14 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/sparrow/4-docker# ./buildall 2021-01-14 03:32:28 INFO -- start build /c/compass-ci/container/archlinux/. Unable to find image 'archlinux:testbed' locally docker: Error response from daemon: manifest for archlinux:testbed not found. See 'docker run --help'. [why] before: if docker images not exist, not build images after: if docker images not exist, build images Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index f24226c..882ef02 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" != 0 ] || [ "$action" != "reboot" ];then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 1

[PATCH v6 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 14 Jan '21

14 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/sparrow/4-docker# ./buildall 2021-01-14 03:32:28 INFO -- start build /c/compass-ci/container/archlinux/. Unable to find image 'archlinux:testbed' locally docker: Error response from daemon: manifest for archlinux:testbed not found. See 'docker run --help'. [why] before: if docker images not exist, not build images after: if docker images not exist, build images Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index f24226c..a60067f 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" != 0 ] || [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

1 0

Re: [PATCH v2 lkp-tests 1/2] tests: modify api-avx2neon
by Wu Fengguang 14 Jan '21

14 Jan '21

>>>+ echo "api-avx2neon running" >>>+ ./test >> >>What if ./test exit 1? > > this script captures and counts its own error output What if ./test exit for other (unexpected) reason? Then we miss a failure case. >>>+ echo "api-avx2neon finished"

1 0

[PATCH v3 lkp-tests] stats: add stats script for openeuler_docker test
by Bai Jing 14 Jan '21

14 Jan '21

[input] ^[[0mwordpress.build.fail: 1 [output] wordpress.build.fail: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 stats/openeuler_docker diff --git a/stats/openeuler_docker b/stats/openeuler_docker new file mode 100755 index 000000000..e31934573 --- /dev/null +++ b/stats/openeuler_docker @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + line.chomp! + next unless line =~ /build.fail/ + + puts line +end -- 2.23.0

2 1

[PATCH compass-ci] qeumu/kvm.sh: modify the comment
by Hu Xuejiao 14 Jan '21

14 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index ce3b75f..8b05f83 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -53,7 +53,7 @@ parse_ipxe_script() check_option_value() { [ -s "$kernel" ] || { - echo "The kernel does not exist" + echo "The kernel does not exist: $kernel" exit } -- 2.23.0

2 1

[PATCH v2 lkp-tests] stats: add stats script for openeuler_docker test
by Bai Jing 14 Jan '21

14 Jan '21

[input] ^[[0mwordpress_build_on_fail: 1 [output] wordpress_build_on_fail: 1 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/openeuler_docker | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 stats/openeuler_docker diff --git a/stats/openeuler_docker b/stats/openeuler_docker new file mode 100755 index 000000000..3f8012a8c --- /dev/null +++ b/stats/openeuler_docker @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + line.chomp! + next unless line =~ /build_on_fail/ + + puts line +end -- 2.23.0

2 2

Re: [PATCH v2 lkp-tests 2/2] lib: fix 409 conflict
by Wu Fengguang 14 Jan '21

14 Jan '21

TO ALL. On Thu, Jan 14, 2021 at 10:55:46AM +0800, Li Ping wrote: >On Thu, Jan 14, 2021 at 09:08:50AM +0800, Wu Fengguang wrote: >>On Wed, Jan 13, 2021 at 09:19:47PM +0800, Li Ping wrote: >>>[why] >>>find a -type d >>>a >>>a/b >>>a/c >>>find a -type d -exec curl -sSf -X MKCOL "$http_url/{}" \; >>>==> curl: (22) The requested URL returned error: 409 Conflict >>>reason: upload an empty folder causes this error. >> >>I suspect 409 happens because server side already have that dir. >>It's like you run > > That's what I thought at first, then I make new dir, > upload the new dir also causes this error. It does not happen if you didn't say it. That's the golden rule of mailing list. Please document such important points in the changelog in future. Thanks, Fengguang

1 0

Re: [PATCH v2 lkp-tests 2/2] lib: fix 409 conflict
by Wu Fengguang 14 Jan '21

14 Jan '21

On Thu, Jan 14, 2021 at 10:55:46AM +0800, Li Ping wrote: >On Thu, Jan 14, 2021 at 09:08:50AM +0800, Wu Fengguang wrote: >>On Wed, Jan 13, 2021 at 09:19:47PM +0800, Li Ping wrote: >>>[why] >>>find a -type d >>>a >>>a/b >>>a/c >>>find a -type d -exec curl -sSf -X MKCOL "$http_url/{}" \; >>>==> curl: (22) The requested URL returned error: 409 Conflict >>>reason: upload an empty folder causes this error. >> >>I suspect 409 happens because server side already have that dir. >>It's like you run > > That's what I thought at first, then I make new dir, > upload the new dir also causes this error. > > if I create new file under the dir would not cause this error. > so i think there is no need to upload empty folder at all. No "think" please. You should give clear statement, like "webdav supports auto-and-recursively create dirs when uploading some dir_level1/dir_level2/file". And prove it with commandline examples or webdav API definitions. Whether or not your patch is right, the changelog is not clear enough to prove whether it's "definitely/always right" or "right by chance". >> >> mkdir a => success >> mkdir a => 409 >> >>But will the server side auto create new subdirs if they do not >>already exist? Probably the below find will fail. > >> >>Please experiment with deeper dirs, like a/b/c/file > > Actually, I have experimented with deeper dirs Then please showcase it in changelog. BTW, please try this example https://stackoverflow.com/questions/42941176/how-to-create-directories-recu… If our webdav server can auto create subdirs, then what's the difference in our server comparing to the standard/old one? Thanks, Fengguang

1 0

[PATCH compass-ci] lib/error_messages.rb: uniform naming style
by Lin Jiaxin 14 Jan '21

14 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 14 +++++++------- lib/git_bisect.rb | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 5bbc329..3e1564b 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -8,8 +8,8 @@ require_relative "#{ENV['LKP_SRC']}/lib/common" # usage: # foo = ErrorMessages.new(build-pkg) # build-pkg is a file that includes errors generated by make -# errorid_array = foo.obtain_error_messages_by_errorid(errorid) -# errorid_array includes all error_messages that error same as errorid +# error_id_array = foo.obtain_error_messages_by_error_id(error_id) +# error_id_array includes all error_messages that error same as error_id # error_messages_hash = foo.obtain_error_messages # error_messages_hash.each { |k,v| puts k; puts; v.each {|m| puts m } } # output: @@ -40,15 +40,15 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_errorid(errorid) - error_messages_by_errorid = [] + def obtain_error_messages_by_error_id(error_id) + error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| - if ('build-pkg.' + build_pkg_error_id(k)) == errorid - error_messages_by_errorid += v.to_a + if ('build-pkg.' + build_pkg_error_id(k)) == error_id + error_messages_by_error_id += v.to_a end end - error_messages_by_errorid + error_messages_by_error_id end private diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 2043730..2fee60c 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -97,7 +97,7 @@ class GitBisect return nil unless temp[0].include?('is the first bad commit') || temp[-1].include?('bisect run success') first_bad_commit = Utils.parse_first_bad_commit(result) - bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) + bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_error_id(@error_id) all_errors = Utils.obt_errors(@work_dir, first_bad_commit) first_bad_commit_result_root = Utils.obt_result_root_by_commit(first_bad_commit) FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) -- 2.23.0

1 0

Re: [PATCH lkp-tests] lib/job.sh: modify the generated file suffix
by Cao Xueliang 14 Jan '21

14 Jan '21

On Thu, Jan 14, 2021 at 03:44:10PM +0800, Li Ping wrote: >[why] >user extract $program result actually generating a YAML file. >and compass-ci/sbin/result2stats have already support merge yaml file > >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > lib/job.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > >diff --git a/lib/job.sh b/lib/job.sh >index 03c49e0ad..dc65f68d8 100755 >--- a/lib/job.sh >+++ b/lib/job.sh >@@ -318,5 +318,5 @@ run_test() > run_target_stats() > { > local script_name=$1 >- $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.json >+ $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.yaml I don't think this can change json to yaml format. Thanks, Xueliang > } >-- >2.23.0 >

1 0

[PATCH v9 compass-ci 2/2] lib/es_query.rb: parse es_result to a Hash result
by Lu Kaiyi 14 Jan '21

14 Jan '21

[why] make es result more easy access. [example] input: es_result = [{"key"=>"build-pkg", "doc_count"=>354526, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>5708}, {"key"=>"finished", "doc_count"=>1033}, {"key"=>"incomplete", "doc_count"=>204}, {"key"=>"submit", "doc_count"=>136}, ... output: result_hash = {"build-pkg"=> {"failed"=>5708, "finished"=>1033, "incomplete"=>204, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1785, "failed"=>675, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 91aa6ab..928bd68 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -128,7 +128,7 @@ class ESQuery result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] return nil if result.empty? - result + parse_fields(result) end end @@ -216,3 +216,47 @@ def build_aggs_from_fields(fields) aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs aggs_hash end + +# input: +# es_result = +# [{"key"=>"build-pkg", +# "doc_count"=>354526, +# "all_job_state"=> +# {"doc_count_error_upper_bound"=>0, +# "sum_other_doc_count"=>0, +# "buckets"=> +# [{"key"=>"failed", "doc_count"=>5708}, +# {"key"=>"finished", "doc_count"=>1033}, +# {"key"=>"incomplete", "doc_count"=>204}, +# {"key"=>"submit", "doc_count"=>136}, +# ... +# +# output: +# result_hash = +# {"build-pkg"=> +# {"failed"=>5708, +# "finished"=>1033, +# "incomplete"=>204, +# "submit"=>136, +# "OOM"=>11, +# "post_run"=>2}, +# "cci-depends"=> +# {"finished"=>1785, +# "failed"=>675, +# ... +def parse_fields(es_result) + result_hash = {} + es_result.each do |result| + key = result['key'] + sub_field = result.keys.detect { |field| field.start_with?('all_') } + + if sub_field + all_field = result[sub_field]['buckets'] + result_hash[key] = parse_fields(all_field) + else + result_hash[key] = result['doc_count'] + end + end + + result_hash +end -- 2.23.0

1 0

[PATCH v9 compass-ci 1/2] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 14 Jan '21

14 Jan '21

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 70 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..91aa6ab 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,49 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = build_aggs_from_fields(fields) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +189,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields) + aggs_hash = {} + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] ||= {} + aggs_hash['aggs']["all_#{field}"] = { + 'terms' => { field: field, size: 1000 } + } + sub_aggs = build_aggs_from_fields(fields) + aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs + aggs_hash +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] jobs: add ansible_test.yaml
by Zhang Yale 14 Jan '21

14 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/ansible_test.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/ansible_test.yaml diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml new file mode 100644 index 00000000..18651715 --- /dev/null +++ b/jobs/ansible_test.yaml @@ -0,0 +1,5 @@ +suite: ansible_test +category: functional + +ansible_test: + ansible_repo: -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: reduce possible duplicate judgment
by Li Yuanchao 14 Jan '21

14 Jan '21

The method is called several times, add judgment in the method, so don't need to add it everywhere the method is called. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 2adaaa1..86f02e5 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -187,6 +187,8 @@ class MirrorMain end def do_push(fork_key) + return if @fork_stat[fork_key][:queued] + @fork_stat[fork_key][:queued] = true @git_info[fork_key][:cur_refs] = get_cur_refs(fork_key) if @git_info[fork_key][:cur_refs].nil? @git_queue.push(@git_info[fork_key]) @@ -196,7 +198,7 @@ class MirrorMain return if @git_queue.size >= 1 fork_key = @priority_queue.delete_min_return_key - do_push(fork_key) unless @fork_stat[fork_key][:queued] + do_push(fork_key) @priority_queue.push fork_key, (@priority - @fork_stat[fork_key][:priority]) @priority += 1 end -- 2.23.0

1 0

[PATCH v4 lkp-tests] stats: add ansible_test stats script
by Zhang Yale 14 Jan '21

14 Jan '21

[Why] Output needs to be displayed in the stats.json file, so this output need to processed as a key-value pair. [input] PLAY RECAP ********************************************************************* localhost : ok=2 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 playbook_run_on_fail: 1 [output] ansible_test.total.nr_ok: 2 ansible_test.total.nr_changed: 0 ansible_test.total.nr_failed: 1 ansible_test.total.nr_skipped: 0 ansible_test.total.nr_rescued: 0 ansible_test.total.nr_ignored: 0 playbook_run_on_fail: 1 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/ansible_test | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 stats/ansible_test diff --git a/stats/ansible_test b/stats/ansible_test new file mode 100755 index 00000000..99aeccc1 --- /dev/null +++ b/stats/ansible_test @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby + +while (line = STDIN.gets) + case line.chomp! + when /: ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ + puts "ansible_test.total.nr_ok: #{$1}" + puts "ansible_test.total.nr_changed: #{$2}" + puts "ansible_test.total.nr_failed: #{$4}" + puts "ansible_test.total.nr_skipped: #{$5}" + puts "ansible_test.total.nr_rescued: #{$6}" + puts "ansible_test.total.nr_ignored: #{$7}" + when /playbook_run_on_fail/ + puts line + end +end -- 2.23.0

2 2

[PATCH v2 compass-ci] sbin/es-find support range query
by Lu Weitao 14 Jan '21

14 Jan '21

[Example] es-find -r/--range "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/sbin/es-find b/sbin/es-find index 17eefe6..5305472 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -5,6 +5,7 @@ # Usage: # es-find key=val ... +# es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -29,7 +30,41 @@ def parse_argv items end +# output: +# eg: +# { +# start_time => { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# ... +# } +def prase_range(range) + range_hash = {} + keyword_list = range.split(';') + keyword_list.each do |keyword| + key, value = keyword.split('=') + next unless key && value + + assign_range(range_hash, key, value) + end + + range_hash +end + +def assign_range(range_hash, key, value) + range_hash[key] = {} + value_list = value.split(',') + if value_list.length > 1 + range_hash[key][:gte] = value_list[0] + range_hash[key][:lte] = value_list[1] + else + range_hash[key][:gte] = value + end +end + options = { nr_jobs: 10, desc_keyword: nil } +range = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -49,6 +84,10 @@ opt_parser = OptionParser.new do |opts| options[:desc_keyword] = d end + opts.on('-r', '--range KEYWORD', 'range by keyword: start_time|end_time|...') do |r| + range = r + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -57,6 +96,8 @@ end opt_parser.parse!(ARGV) items = parse_argv +items[:range] = prase_range(range) if range + es = ESQuery.new(ES_HOST, ES_PORT) query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-find support range query
by Lu Weitao 14 Jan '21

14 Jan '21

[Example] es-find -r/--range "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/sbin/es-find b/sbin/es-find index 17eefe6..8348554 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -5,6 +5,7 @@ # Usage: # es-find key=val ... +# es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -29,7 +30,42 @@ def parse_argv items end +# output: +# eg: +# { +# start_time => { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# ... +# } +def prase_range(range) + range_hash = {} + keyword_list = range.split(';') + puts "keyword_list ==> #{keyword_list}" + keyword_list.each do |keyword| + key, value = keyword.split('=') + next unless key && value + + assign_range(range_hash, key, value) + end + + range_hash +end + +def assign_range(range_hash, key, value) + range_hash[key] = {} + value_list = value.split(',') + if value_list.length > 1 + range_hash[key][:gte] = value_list[0] + range_hash[key][:lte] = value_list[1] + else + range_hash[key][:gte] = value + end +end + options = { nr_jobs: 10, desc_keyword: nil } +range = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -49,6 +85,10 @@ opt_parser = OptionParser.new do |opts| options[:desc_keyword] = d end + opts.on('-r', '--range KEYWORD', 'range by keyword: start_time|end_time|...') do |r| + range = r + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -57,6 +97,8 @@ end opt_parser.parse!(ARGV) items = parse_argv +items[:range] = prase_range(range) if range + es = ESQuery.new(ES_HOST, ES_PORT) query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) -- 2.23.0

1 1

[PATCH v2 lkp-tests] jobs: and ansible_test.yaml
by Zhang Yale 14 Jan '21

14 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/ansible_test.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/ansible_test.yaml diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml new file mode 100644 index 00000000..18651715 --- /dev/null +++ b/jobs/ansible_test.yaml @@ -0,0 +1,5 @@ +suite: ansible_test +category: functional + +ansible_test: + ansible_repo: -- 2.23.0

1 1

[PATCH compass-ci] container/add-software: refactor "cat <<-EOF" to make code aligned
by Xu Xijian 14 Jan '21

14 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/add-software/run b/container/add-software/run index 840fb2c..aab659b 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -3,7 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. [ -L "$1/boot/vmlinuz" -a -n "$2" ] || { - cat << 'EOF' + cat <<-'EOF' Example usage: 1> ./run /os/centos/aarch64/7.6.1810 pack1 pack2 ... 2> ./run /os/centos/aarch64/7.6.1810 $(show-depends-packages centos) @@ -11,7 +11,7 @@ the whole path is '$LKP_SRC/distro/adaptation/centos'. [Remind] This tool also works for openEuler. -EOF + EOF exit } -- 2.23.0

1 0