- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] sparrow/0-package: add packages
by Liu Yinsi 19 Jan '21

19 Jan '21

container os-cifs and os-nfs will use sudo command, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index b25dade..b8ef769 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -24,6 +24,7 @@ pkgs=( hostname wget curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 4d13413..947cbdc 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -24,6 +24,7 @@ pkgs=( qemu-system-aarch64 qemu-system-x86 qemu-system-riscv64 + sudo ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 04996f5..61a37c4 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -23,6 +23,7 @@ pkgs=( hostname libvirt curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

2 2

[PATCH compass-ci] sbin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 19 Jan '21

19 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..c368ad6 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: fix that es-states clean when git-mirror restart
by Li Yuanchao 19 Jan '21

19 Jan '21

When init fork_stat{}, first query the es, get states in es if there is. So historical data can be kept. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 86f02e5..1109cfe 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -109,15 +109,8 @@ class MirrorMain @webhook_queue = channel.queue('web_hook') end - def fork_stat_init(stat_key) - @fork_stat[stat_key] = { - queued: false, - priority: 0, - fetch_time: [], - offset_fetch: 0, - new_refs_time: [], - offset_new_refs: 0 - } + def fork_stat_init(git_repo) + @fork_stat[git_repo] = get_fork_stat(git_repo) end def load_defaults(repodir) @@ -372,4 +365,31 @@ class MirrorMain handle_submodule(submodule) end + + def copy_from_es(es_data) + fork_stat = {} + fork_stat[:queued] = es_data['queued'] + fork_stat[:priority] = es_data['priority'] + fork_stat[:fetch_time] = es_data['fetch_time'] + fork_stat[:offset_fetch] = es_data['offset_fetch'] + fork_stat[:new_refs_time] = es_data['new_refs_time'] + fork_stat[:offset_new_refs] = es_data['offset_new_refs'] + return fork_stat + end + + def get_fork_stat(git_repo) + query = { query: { match: { git_repo: git_repo } } } + result = @es_client.search(index: 'repo', body: query)['hits'] + return copy_from_es(result['hits'][0]['_source']) if result['total'].positive? + + fork_stat = { + queued: false, + priority: 0, + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 + } + return fork_stat + end end -- 2.23.0

2 1

[PATCH v2 compass-ci 1/2] qemu/kvm.sh: optimize the prompt when the error occurs
by Yu Chuan 19 Jan '21

19 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/log.sh | 32 ++++++++++++++++++++++++++++++++ providers/qemu/kvm.sh | 12 ++++++++---- 2 files changed, 40 insertions(+), 4 deletions(-) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 000000000000..56b08f77b3d0 --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,32 @@ +#!/bin/bash + +die() +{ + log_error "$@" + exit 1 +} + +log() +{ + log_info "$@" +} + +log_info() +{ + echo "$(date +'%F %T') INFO -- $*" +} + +log_debug() +{ + echo "$(date +'%F %T') DEBUG -- $*" +} + +log_warn() +{ + echo "$(date +'%F %T') WARN -- $*" >&2 +} + +log_error() +{ + echo "$(date +'%F %T') ERROR -- $*" >&2 +} diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 35678fbd44ef..a136c8ff45cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -4,6 +4,8 @@ # - nr_cpu # - memory +source $CCI_SRC/lib/log.sh + : ${nr_cpu:=1} : ${memory:=1G} @@ -52,15 +54,17 @@ parse_ipxe_script() check_option_value() { - [ -s "$kernel" ] || { - echo "The kernel does not exist: $kernel" - exit + [ -n "$kernel" ] || { + log_info "Can not find job for current hostname: $hostname." + exit 0 } + + [ -s "$kernel" ] || die "Can not find kernel file or kernel file is empty: $kernel." # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit + [ -n "$initrds" ] || die "Can not find any initrd for current vm." } set_initrd() -- 2.23.0

2 2

[PATCH v3 lkp-tests] hosts: add new host file for libvirt
by Xiao Shenwei 19 Jan '21

19 Jan '21

[why] libvirt use a xml file to start vm <os> <kernel>xxx</kernel> <initrd>xxx</initrd> <cmdline>xxx</cmdline> </os> so, we can submit job use os_mount=initramfs to get kernel and initrd absolute path. Minimum memory limit: 16G Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/vt-2p16g | 5 +++++ hosts/vt-2p32g | 5 +++++ 2 files changed, 10 insertions(+) create mode 100644 hosts/vt-2p16g create mode 100644 hosts/vt-2p32g diff --git a/hosts/vt-2p16g b/hosts/vt-2p16g new file mode 100644 index 000000000..aecbb7948 --- /dev/null +++ b/hosts/vt-2p16g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 16G diff --git a/hosts/vt-2p32g b/hosts/vt-2p32g new file mode 100644 index 000000000..aa7c8c065 --- /dev/null +++ b/hosts/vt-2p32g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 32G -- 2.23.0

1 0

[PATCH lkp-tests 1/2] stats/crystal/rcuscale.cr: create rcuscale.cr as same content with rcuscale
by Lu Kaiyi 19 Jan '21

19 Jan '21

create rcuscale.cr as same content with rcuscale Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/rcuscale.cr | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 stats/crystal/rcuscale.cr diff --git a/stats/crystal/rcuscale.cr b/stats/crystal/rcuscale.cr new file mode 100644 index 000000000..45c21f8cf --- /dev/null +++ b/stats/crystal/rcuscale.cr @@ -0,0 +1,30 @@ +#!/usr/bin/env ruby + +# Input example: +# [ 20.504916] rcu-scale:--- Start of test: nreaders=2 nwriters=2 verbose=1 shutdown=0 +# [ 20.521488] rcu-scale: rcu_scale_reader task started +# [ 20.526468] rcu-scale: rcu_scale_reader task started +# [ 20.536483] rcu-scale: rcu_scale_writer task started +# [ 20.541467] rcu-scale: rcu_scale_writer task started +# [ 31.728409] rcu-scale: rcu_scale_writer 0 has 100 measurements +# [ 31.730422] rcu-scale: rcu_scale_writer 1 has 100 measurements +# [ 31.744797] rcu-scale: Test complete +# [ 320.627980] rcu-scale: writer 0 gps: 101 +# [ 320.637100] rcu-scale: writer 1 gps: 100 +# [ 320.640206] rcu-scale: start: 30396186972 end: 31136206117 duration: 740019145 gps: 201 batches: 207 +# ... + +result = 'unknown' +type = 'unknown' + +while (line = STDIN.gets) + case line + when /^\[.*\] ([A-Za-z]+)-scale: Test complete/ + type = $1 + result = 'pass' + break + end +end + +stat = type + '.' + result +puts "#{stat}: 1" -- 2.23.0

2 1

[PATCH lkp-tests 2/2] stats/crystal/rcuscale.cr: convert rcuscale.cr to crystal with ruby2crystal tools
by Lu Kaiyi 19 Jan '21

19 Jan '21

/c/crystal/ruby_crystal_codemod/exe/ruby_crystal_codemod rcuscale.cr /c/lkp-tests/tools/ruby2crystal.sh rcuscale.cr Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/rcuscale.cr | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/stats/crystal/rcuscale.cr b/stats/crystal/rcuscale.cr index 45c21f8cf..e5c0a2f28 100644 --- a/stats/crystal/rcuscale.cr +++ b/stats/crystal/rcuscale.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal # Input example: # [ 20.504916] rcu-scale:--- Start of test: nreaders=2 nwriters=2 verbose=1 shutdown=0 @@ -14,17 +14,17 @@ # [ 320.640206] rcu-scale: start: 30396186972 end: 31136206117 duration: 740019145 gps: 201 batches: 207 # ... -result = 'unknown' -type = 'unknown' +result = "unknown" +type = "unknown" while (line = STDIN.gets) case line when /^\[.*\] ([A-Za-z]+)-scale: Test complete/ type = $1 - result = 'pass' + result = "pass" break end end -stat = type + '.' + result +stat = type + "." + result puts "#{stat}: 1" -- 2.23.0

2 1

[PATCH v4 compass-ci] docker/docker.rb: add docker serial log
by Wu Zhende 19 Jan '21

19 Jan '21

Generate serial log file. And add the start and end flags into it. It's for generate docker's dmsg file in result root. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 8e489c6..56c9754 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -18,6 +18,7 @@ names = Set.new %w[ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 +LOG_DIR = '/srv/cci/serial/logs' def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -82,12 +83,35 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, + { 'hostname' => hostname, + 'docker_image' => docker_image, + 'load_path' => load_path, + 'log_dir' => "#{LOG_DIR}/#{hostname}" + }, ENV['CCI_SRC'] + '/providers/docker/run.sh' ) clean_dir(load_path) end +def record_start_log(log_file, hash: {}) + start_time = Time.new + File.open(log_file, 'w') do |f| + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep(2) + f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting DOCKER" + f.puts "\n#{hash['job']}" + end + return start_time +end + +def record_end_log(log_file, start_time) + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal DOCKER duration: #{duration} minutes" + end +end + def main(hostname, queues) set_host2queues(hostname, queues) url = get_url hostname @@ -95,10 +119,15 @@ def main(hostname, queues) hash = parse_response url return del_host2queues(hostname) if hash.nil? + log_file = "#{LOG_DIR}/#{hostname}" + start_time = record_start_log(log_file, hash: hash) + load_path = build_load_path(hostname) load_initrds(load_path, hash) start_container(hostname, load_path, hash) + del_host2queues(hostname) + record_end_log(log_file, start_time) end def loop_main(hostname, queues) -- 2.23.0

1 0

[PATCH v3 compass-ci] docker/docker.rb: add docker serial log
by Wu Zhende 19 Jan '21

19 Jan '21

Generate serial log file. And add the start and end flags into it. It's for generate docker's dmsg file in result root. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 8e489c6..edbc54d 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -18,6 +18,7 @@ names = Set.new %w[ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 +LOG_DIR = '/srv/cci/serial/logs' def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -82,12 +83,35 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, + { 'hostname' => hostname, + 'docker_image' => docker_image, + 'load_path' => load_path, + 'log_dir' => "#{LOG_DIR}/#{hostname}" + }, ENV['CCI_SRC'] + '/providers/docker/run.sh' ) clean_dir(load_path) end +def record_start_log(log_file, hash: {}) + start_time = Time.new + File.open(log_file, 'w') do |f| + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep(2) + f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting DOCKER" + f.puts "\n#{hash['job']}" + end + return start_time +end + +def record_end_log(log_file, start_time) + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal DOCKER duration: #{duration} minutes" + end +end + def main(hostname, queues) set_host2queues(hostname, queues) url = get_url hostname @@ -95,10 +119,15 @@ def main(hostname, queues) hash = parse_response url return del_host2queues(hostname) if hash.nil? + log_file = "#{LOG_DIR}/#{hostname}" + start_time = record_start_log(log_file, hash: hash) + load_path = build_load_path(hostname) load_initrds(load_path, hash) start_container(hostname, load_path, hash) + del_host2queues(hostname) + record_end_log(log_file, start_time) end def loop_main(hostname, queues) -- 2.23.0

2 1

[PATCH lkp-tests] jobs/submit-cci.yaml: automatically submit test case.yaml
by Hu Xuejiao 19 Jan '21

19 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/submit-cci.yaml | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 jobs/submit-cci.yaml diff --git a/jobs/submit-cci.yaml b/jobs/submit-cci.yaml new file mode 100644 index 00000000..129ff44a --- /dev/null +++ b/jobs/submit-cci.yaml @@ -0,0 +1,3 @@ +suite: submit-cci +category: functional +submit-cci: -- 2.23.0

2 1