- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/2] qemu/kvm.sh: optimize the prompt when the error occurs
by Yu Chuan 19 Jan '21

19 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/log.sh | 32 ++++++++++++++++++++++++++++++++ providers/qemu/kvm.sh | 12 ++++++++---- 2 files changed, 40 insertions(+), 4 deletions(-) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 000000000000..56b08f77b3d0 --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,32 @@ +#!/bin/bash + +die() +{ + log_error "$@" + exit 1 +} + +log() +{ + log_info "$@" +} + +log_info() +{ + echo "$(date +'%F %T') INFO -- $*" +} + +log_debug() +{ + echo "$(date +'%F %T') DEBUG -- $*" +} + +log_warn() +{ + echo "$(date +'%F %T') WARN -- $*" >&2 +} + +log_error() +{ + echo "$(date +'%F %T') ERROR -- $*" >&2 +} diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 35678fbd44ef..a136c8ff45cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -4,6 +4,8 @@ # - nr_cpu # - memory +source $CCI_SRC/lib/log.sh + : ${nr_cpu:=1} : ${memory:=1G} @@ -52,15 +54,17 @@ parse_ipxe_script() check_option_value() { - [ -s "$kernel" ] || { - echo "The kernel does not exist: $kernel" - exit + [ -n "$kernel" ] || { + log_info "Can not find job for current hostname: $hostname." + exit 0 } + + [ -s "$kernel" ] || die "Can not find kernel file or kernel file is empty: $kernel." # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit + [ -n "$initrds" ] || die "Can not find any initrd for current vm." } set_initrd() -- 2.23.0

2 2

[PATCH v3 lkp-tests] hosts: add new host file for libvirt
by Xiao Shenwei 19 Jan '21

19 Jan '21

[why] libvirt use a xml file to start vm <os> <kernel>xxx</kernel> <initrd>xxx</initrd> <cmdline>xxx</cmdline> </os> so, we can submit job use os_mount=initramfs to get kernel and initrd absolute path. Minimum memory limit: 16G Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/vt-2p16g | 5 +++++ hosts/vt-2p32g | 5 +++++ 2 files changed, 10 insertions(+) create mode 100644 hosts/vt-2p16g create mode 100644 hosts/vt-2p32g diff --git a/hosts/vt-2p16g b/hosts/vt-2p16g new file mode 100644 index 000000000..aecbb7948 --- /dev/null +++ b/hosts/vt-2p16g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 16G diff --git a/hosts/vt-2p32g b/hosts/vt-2p32g new file mode 100644 index 000000000..aa7c8c065 --- /dev/null +++ b/hosts/vt-2p32g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 32G -- 2.23.0

1 0

[PATCH lkp-tests 1/2] stats/crystal/rcuscale.cr: create rcuscale.cr as same content with rcuscale
by Lu Kaiyi 19 Jan '21

19 Jan '21

create rcuscale.cr as same content with rcuscale Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/rcuscale.cr | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 stats/crystal/rcuscale.cr diff --git a/stats/crystal/rcuscale.cr b/stats/crystal/rcuscale.cr new file mode 100644 index 000000000..45c21f8cf --- /dev/null +++ b/stats/crystal/rcuscale.cr @@ -0,0 +1,30 @@ +#!/usr/bin/env ruby + +# Input example: +# [ 20.504916] rcu-scale:--- Start of test: nreaders=2 nwriters=2 verbose=1 shutdown=0 +# [ 20.521488] rcu-scale: rcu_scale_reader task started +# [ 20.526468] rcu-scale: rcu_scale_reader task started +# [ 20.536483] rcu-scale: rcu_scale_writer task started +# [ 20.541467] rcu-scale: rcu_scale_writer task started +# [ 31.728409] rcu-scale: rcu_scale_writer 0 has 100 measurements +# [ 31.730422] rcu-scale: rcu_scale_writer 1 has 100 measurements +# [ 31.744797] rcu-scale: Test complete +# [ 320.627980] rcu-scale: writer 0 gps: 101 +# [ 320.637100] rcu-scale: writer 1 gps: 100 +# [ 320.640206] rcu-scale: start: 30396186972 end: 31136206117 duration: 740019145 gps: 201 batches: 207 +# ... + +result = 'unknown' +type = 'unknown' + +while (line = STDIN.gets) + case line + when /^\[.*\] ([A-Za-z]+)-scale: Test complete/ + type = $1 + result = 'pass' + break + end +end + +stat = type + '.' + result +puts "#{stat}: 1" -- 2.23.0

2 1

[PATCH lkp-tests 2/2] stats/crystal/rcuscale.cr: convert rcuscale.cr to crystal with ruby2crystal tools
by Lu Kaiyi 19 Jan '21

19 Jan '21

/c/crystal/ruby_crystal_codemod/exe/ruby_crystal_codemod rcuscale.cr /c/lkp-tests/tools/ruby2crystal.sh rcuscale.cr Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/rcuscale.cr | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/stats/crystal/rcuscale.cr b/stats/crystal/rcuscale.cr index 45c21f8cf..e5c0a2f28 100644 --- a/stats/crystal/rcuscale.cr +++ b/stats/crystal/rcuscale.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal # Input example: # [ 20.504916] rcu-scale:--- Start of test: nreaders=2 nwriters=2 verbose=1 shutdown=0 @@ -14,17 +14,17 @@ # [ 320.640206] rcu-scale: start: 30396186972 end: 31136206117 duration: 740019145 gps: 201 batches: 207 # ... -result = 'unknown' -type = 'unknown' +result = "unknown" +type = "unknown" while (line = STDIN.gets) case line when /^\[.*\] ([A-Za-z]+)-scale: Test complete/ type = $1 - result = 'pass' + result = "pass" break end end -stat = type + '.' + result +stat = type + "." + result puts "#{stat}: 1" -- 2.23.0

2 1

[PATCH v4 compass-ci] docker/docker.rb: add docker serial log
by Wu Zhende 19 Jan '21

19 Jan '21

Generate serial log file. And add the start and end flags into it. It's for generate docker's dmsg file in result root. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 8e489c6..56c9754 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -18,6 +18,7 @@ names = Set.new %w[ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 +LOG_DIR = '/srv/cci/serial/logs' def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -82,12 +83,35 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, + { 'hostname' => hostname, + 'docker_image' => docker_image, + 'load_path' => load_path, + 'log_dir' => "#{LOG_DIR}/#{hostname}" + }, ENV['CCI_SRC'] + '/providers/docker/run.sh' ) clean_dir(load_path) end +def record_start_log(log_file, hash: {}) + start_time = Time.new + File.open(log_file, 'w') do |f| + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep(2) + f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting DOCKER" + f.puts "\n#{hash['job']}" + end + return start_time +end + +def record_end_log(log_file, start_time) + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal DOCKER duration: #{duration} minutes" + end +end + def main(hostname, queues) set_host2queues(hostname, queues) url = get_url hostname @@ -95,10 +119,15 @@ def main(hostname, queues) hash = parse_response url return del_host2queues(hostname) if hash.nil? + log_file = "#{LOG_DIR}/#{hostname}" + start_time = record_start_log(log_file, hash: hash) + load_path = build_load_path(hostname) load_initrds(load_path, hash) start_container(hostname, load_path, hash) + del_host2queues(hostname) + record_end_log(log_file, start_time) end def loop_main(hostname, queues) -- 2.23.0

1 0

[PATCH v3 compass-ci] docker/docker.rb: add docker serial log
by Wu Zhende 19 Jan '21

19 Jan '21

Generate serial log file. And add the start and end flags into it. It's for generate docker's dmsg file in result root. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 8e489c6..edbc54d 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -18,6 +18,7 @@ names = Set.new %w[ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 +LOG_DIR = '/srv/cci/serial/logs' def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -82,12 +83,35 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, + { 'hostname' => hostname, + 'docker_image' => docker_image, + 'load_path' => load_path, + 'log_dir' => "#{LOG_DIR}/#{hostname}" + }, ENV['CCI_SRC'] + '/providers/docker/run.sh' ) clean_dir(load_path) end +def record_start_log(log_file, hash: {}) + start_time = Time.new + File.open(log_file, 'w') do |f| + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep(2) + f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting DOCKER" + f.puts "\n#{hash['job']}" + end + return start_time +end + +def record_end_log(log_file, start_time) + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal DOCKER duration: #{duration} minutes" + end +end + def main(hostname, queues) set_host2queues(hostname, queues) url = get_url hostname @@ -95,10 +119,15 @@ def main(hostname, queues) hash = parse_response url return del_host2queues(hostname) if hash.nil? + log_file = "#{LOG_DIR}/#{hostname}" + start_time = record_start_log(log_file, hash: hash) + load_path = build_load_path(hostname) load_initrds(load_path, hash) start_container(hostname, load_path, hash) + del_host2queues(hostname) + record_end_log(log_file, start_time) end def loop_main(hostname, queues) -- 2.23.0

2 1

[PATCH lkp-tests] jobs/submit-cci.yaml: automatically submit test case.yaml
by Hu Xuejiao 19 Jan '21

19 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/submit-cci.yaml | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 jobs/submit-cci.yaml diff --git a/jobs/submit-cci.yaml b/jobs/submit-cci.yaml new file mode 100644 index 00000000..129ff44a --- /dev/null +++ b/jobs/submit-cci.yaml @@ -0,0 +1,3 @@ +suite: submit-cci +category: functional +submit-cci: -- 2.23.0

2 1

[PATCH v1 compass-ci] delimiter: use submit script in lkp-tests to submit job
by Cao Xueliang 19 Jan '21

19 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/constants.rb | 1 + src/delimiter/utils.rb | 17 +++++++++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 23b5053..457e33d 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -13,3 +13,4 @@ DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" # The files which are in this dir can be uploaded by lkp-tests TMP_RESULT_ROOT = ENV['TMP_RESULT_ROOT'] || '/tmp/lkp/result' +PROCESS_JOB_YAML = "/tmp/process.yaml" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index aaad394..2c6549c 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -61,15 +61,17 @@ module Utils return monitor.run end + def save_job_to_yaml(job, yaml_file) + File.open(yaml_file, 'w') { |f| YAML.dump(job, f) } + end + def submit_job(job) - sched = SchedClient.new - response = sched.submit_job(job.to_json) + save_job_to_yaml(job, PROCESS_JOB_YAML) + response = %x(#{LKP_SRC}/sbin/submit #{PROCESS_JOB_YAML}) puts "submit job response: #{response}" - res_arr = JSON.parse(response) - return nil if res_arr.empty? || !res_arr[0]['message'].empty? + return $1 if response =~ /job id=(.*)/ - # just consider build-pkg job - return res_arr[0]['job_id'] + return nil end # submit the bad job @@ -84,6 +86,8 @@ module Utils extract_finished = monitor_run_stop(query) return nil unless extract_finished.zero? + raise "the job is incredible for bisect: #{new_job_id}" unless credible?(job['bad_job_id'], new_job_id, error_id) + stats = query_stats(new_job_id, 10) raise "es cant query #{new_job_id} stats field!" unless stats @@ -138,6 +142,7 @@ module Utils job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id + job.delete('tboxgroup') job.delete('subqueue') job.delete('queue') job.delete('id') -- 2.23.0

1 0

[PATCH compass-ci] delimiter: use submit script in lkp-tests to submit job
by Cao Xueliang 19 Jan '21

19 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- providers/docker/run.sh | 2 ++ src/delimiter/constants.rb | 1 + src/delimiter/utils.rb | 17 +++++++++++------ 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 4d53db7..5e692e4 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -26,9 +26,11 @@ cmd=( -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci + -e LKP_SRC=/c/lkp-tests -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/sbin:ro -v $CCI_SRC:/c/compass-ci:ro + -v $LKP_SRC:/c/lkp-tests:ro -v /srv/git:/srv/git:ro -v /srv/result:/srv/result:ro -v /etc/localtime:/etc/localtime:ro diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 23b5053..457e33d 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -13,3 +13,4 @@ DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" # The files which are in this dir can be uploaded by lkp-tests TMP_RESULT_ROOT = ENV['TMP_RESULT_ROOT'] || '/tmp/lkp/result' +PROCESS_JOB_YAML = "/tmp/process.yaml" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index aaad394..2c6549c 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -61,15 +61,17 @@ module Utils return monitor.run end + def save_job_to_yaml(job, yaml_file) + File.open(yaml_file, 'w') { |f| YAML.dump(job, f) } + end + def submit_job(job) - sched = SchedClient.new - response = sched.submit_job(job.to_json) + save_job_to_yaml(job, PROCESS_JOB_YAML) + response = %x(#{LKP_SRC}/sbin/submit #{PROCESS_JOB_YAML}) puts "submit job response: #{response}" - res_arr = JSON.parse(response) - return nil if res_arr.empty? || !res_arr[0]['message'].empty? + return $1 if response =~ /job id=(.*)/ - # just consider build-pkg job - return res_arr[0]['job_id'] + return nil end # submit the bad job @@ -84,6 +86,8 @@ module Utils extract_finished = monitor_run_stop(query) return nil unless extract_finished.zero? + raise "the job is incredible for bisect: #{new_job_id}" unless credible?(job['bad_job_id'], new_job_id, error_id) + stats = query_stats(new_job_id, 10) raise "es cant query #{new_job_id} stats field!" unless stats @@ -138,6 +142,7 @@ module Utils job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id + job.delete('tboxgroup') job.delete('subqueue') job.delete('queue') job.delete('id') -- 2.23.0

2 2

Re: [PATCH lkp-tests] jobs/submit-cci.yaml: automatically submit test case.yaml
by Zhang Yale 19 Jan '21

19 Jan '21

case.yaml? Reply-To: In-Reply-To: <20210119021655.3324160-1-1034502035(a)qq.com> On Tue, Jan 19, 2021 at 10:16:55AM +0800, Hu Xuejiao wrote: >Signed-off-by: Hu XueJiao <1034502035(a)qq.com> >--- > jobs/submit-cci.yaml | 3 +++ > 1 file changed, 3 insertions(+) > create mode 100644 jobs/submit-cci.yaml > >diff --git a/jobs/submit-cci.yaml b/jobs/submit-cci.yaml >new file mode 100644 >index 00000000..129ff44a >--- /dev/null >+++ b/jobs/submit-cci.yaml >@@ -0,0 +1,3 @@ >+suite: submit-cci >+category: functional Here need a blank line Thanks, Yale >+submit-cci: >-- >2.23.0 >

1 0