- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] dracut-initrd: add support of custom bootstrap-$(arch).cgz
by Yu Chuan 19 Mar '21

19 Mar '21

[Why] When user want run job with their own custom bootstrap instead of lkp-tests framework, they will generate a bootstrap-$(arch).cgz firstly, and scheduler will give the bootstrap-$(arch).cgz path to testbox, then testbox download the bootstrap-$(arch).cgz and use it as part of initrd. So we add a step to handle bootstrap-$(arch).cgz, extract all of it to $NEWROOT and boot from $NEWROOT. [Addition] The root dir in bootstrap-$(arch).cgz is /custom_bootstrap. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/dracut-initrd/bin/overlay-lkp.sh | 5 +++++ container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh | 5 +++++ 2 files changed, 10 insertions(+) diff --git a/container/dracut-initrd/bin/overlay-lkp.sh b/container/dracut-initrd/bin/overlay-lkp.sh index 9a2800e532f9..c2165baa6eea 100644 --- a/container/dracut-initrd/bin/overlay-lkp.sh +++ b/container/dracut-initrd/bin/overlay-lkp.sh @@ -1,6 +1,11 @@ #!/bin/bash # SPDX-License-Identifier: GPL-2.0 +# transfer custom bootstrap +[ -d /custom_bootstrap ] && { + cp -a /custom_bootstrap/* "$NEWROOT"/ +} + # transfer LKP dirs [ -d /lkp ] || return 0 diff --git a/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh b/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh index 44f994e1475e..8597f3086b6a 100755 --- a/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh +++ b/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh @@ -7,6 +7,11 @@ if ! getargbool 0 local; then return fi +# transfer custom bootstrap +[ -d /custom_bootstrap ] && { + cp -a /custom_bootstrap/* "$NEWROOT"/ +} + # transfer LKP dirs [ -d /lkp ] || return 0 -- 2.23.0

1 0

[PATCH compass-ci] scheduler: add custom_bootstrap logic
by Cao Xueliang 19 Mar '21

19 Mar '21

If the custom_bootstrap field in job yaml, need runtime field too, then just return the custom_bootstrap.cgz to the testbox. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 40aaa62..b96cf15 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -514,6 +514,15 @@ class Job private def common_initrds temp_initrds = [] of String + if @hash.has_key?("custom_bootstrap") + raise "need runtime field in the job yaml." unless @hash.has_key?("runtime") + + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + JobHelper.service_path("#{SRV_INITRD}/custom_bootstrap/#{@hash["my_email"]}/bootstrap-#{os_arch}.cgz") + + return temp_initrds + end + temp_initrds << "#{INITRD_HTTP_PREFIX}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz") temp_initrds << "#{SCHED_HTTP_PREFIX}/job_initrd_tmpfs/#{id}/job.cgz" -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/qemu/kvm.sh: added QEMU multiple network card functionality
by Wang Chenglong 19 Mar '21

19 Mar '21

[How] Use the 'nr_nic' to set the number of network cards in yaml. nr_nic: 2 root@vm-2p8g ~# ifconfig enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.42.60 netmask 255.255.0.0 broadcast 172.18.255.255 inet6 fe80::843:24ff:fef2:ec17 prefixlen 64 scopeid 0x20<link> ether 0a:43:24:f2:ec:17 txqueuelen 1000 (Ethernet) RX packets 155492 bytes 1172898602 (1.0 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 75 bytes 11207 (10.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp0s2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.88.228 netmask 255.255.0.0 broadcast 172.18.255.255 inet6 fe80::820:70ff:fe34:d27b prefixlen 64 scopeid 0x20<link> ether 0a:20:70:34:d2:7b txqueuelen 1000 (Ethernet) RX packets 168 bytes 9079 (8.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 199208 bytes 32959935 (31.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- providers/qemu/kvm.sh | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index dcab81a..a1c0046 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -119,9 +119,34 @@ add_disk() done } +set_mac() +{ + job_id=$(awk -F'/' '/job_initrd_tmpfs/{print $(NF-1)}' $ipxe_script) + + if [ $(command -v es-find) ]; then + nr_nic=$(es-find id=$job_id | awk '/nr_nic/{print $2}' | tr -d ,) + else + echo "command not found: es-find. set nr_nic=1" + sleep 1 + fi + + mac_arr[1]=$mac + nr_nic=${nr_nic:-1} + + [ "$nr_nic" -ge "2" ] || return + for i in $(seq 2 $nr_nic) + do + mac=$(echo $hostname$i | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/0a-\1-\2-\3-\4-\5/') + mac_arr[$i]=$mac + done +} + set_nic() { - nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" + for i in $(seq 1 $nr_nic) + do + nic[$i]="-nic tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac_arr[$i]}" + done } set_device() @@ -190,7 +215,7 @@ individual_option() -machine virt-4.0,accel=kvm,gic-version=3 -cpu Kunpeng-920 -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-kvm) @@ -198,17 +223,17 @@ individual_option() -machine virt-4.0,accel=kvm,gic-version=3 -cpu Kunpeng-920 -bios $bios - -nic $nic + ${nic[@]} ) [ "$(arch)" == "x86_64" ] && arch_option=( -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-system-x86_64) arch_option=( -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-system-riscv64) @@ -234,6 +259,7 @@ set_options() { set_bios set_helper + set_mac set_nic set_device set_netdev -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu/kvm.sh: added QEMU multiple network card functionality
by Wang Chenglong 19 Mar '21

19 Mar '21

[How] Use the 'nr_nic' to set the number of network cards in yaml. nr_nic: 2 root@vm-2p8g ~# ifconfig enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.42.60 netmask 255.255.0.0 broadcast 172.18.255.255 inet6 fe80::843:24ff:fef2:ec17 prefixlen 64 scopeid 0x20<link> ether 0a:43:24:f2:ec:17 txqueuelen 1000 (Ethernet) RX packets 155492 bytes 1172898602 (1.0 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 75 bytes 11207 (10.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp0s2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.88.228 netmask 255.255.0.0 broadcast 172.18.255.255 inet6 fe80::820:70ff:fe34:d27b prefixlen 64 scopeid 0x20<link> ether 0a:20:70:34:d2:7b txqueuelen 1000 (Ethernet) RX packets 168 bytes 9079 (8.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 199208 bytes 32959935 (31.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- providers/qemu/kvm.sh | 37 ++++++++++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index dcab81a..bbd5cd5 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -119,9 +119,35 @@ add_disk() done } +set_mac() +{ + job_id=$(cat ipxe_script|awk -F'/' '/job_initrd_tmpfs/{print $(NF-1)}') + + if [ $(command -v es-find) ]; then + nr_nic=$(es-find id=$job_id | awk '/nr_nic/{print $2}' | tr -d ,) + else + echo "command not found: es-find. set nr_nic=1" + sleep 1 + fi + + mac_arr[1]=$mac + nr_nic=${nr_nic:-1} + + if [ "$nr_nic" -ge "2" ]; then + for i in $(seq 2 $nr_nic) + do + mac=$(echo $hostname$i | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/0a-\1-\2-\3-\4-\5/') + mac_arr[$i]=$mac + done + fi +} + set_nic() { - nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" + for i in $(seq 1 $nr_nic) + do + nic[$i]="-nic tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac_arr[$i]}" + done } set_device() @@ -190,7 +216,7 @@ individual_option() -machine virt-4.0,accel=kvm,gic-version=3 -cpu Kunpeng-920 -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-kvm) @@ -198,17 +224,17 @@ individual_option() -machine virt-4.0,accel=kvm,gic-version=3 -cpu Kunpeng-920 -bios $bios - -nic $nic + ${nic[@]} ) [ "$(arch)" == "x86_64" ] && arch_option=( -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-system-x86_64) arch_option=( -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-system-riscv64) @@ -234,6 +260,7 @@ set_options() { set_bios set_helper + set_mac set_nic set_device set_netdev -- 2.23.0

3 4

[PATCH v2 compass-ci] sbin/es-jobs: add -e to get fail result
by Wu Zhende 19 Mar '21

19 Mar '21

es-jobs -e group_id=xxxx error.result.path=/srv/result/openeuler_docker/xxxx: ["Error: unable to find a match"] Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 24 +++++++++++++++++++++++- sbin/es-jobs | 7 ++++++- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index b0d0fb2..e4e9530 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -12,13 +12,14 @@ require_relative './es_query' # deal jobs search from es class ESJobs - def initialize(es_query, my_refine = [], fields = [], stats_filter = []) + def initialize(es_query, my_refine = [], fields = [], stats_filter = [], error_result = false) @es_query = es_query @jobs = query_jobs_from_es @refine = my_refine @fields = fields @stats_filter = stats_filter @refine_jobs = [] + @error_result = error_result set_jobs_summary end @@ -98,6 +99,7 @@ class ESJobs 'max.stats' => {}, 'min.stats' => {} } + result['error.result'] = {} if @error_result metrics.each { |metric| result['raw.stats'][metric] = [] } result end @@ -119,7 +121,27 @@ class ESJobs result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 result['raw.id']["[#{keyword}=#{job[keyword]}]"] ||= [] result['raw.id']["[#{keyword}=#{job[keyword]}]"] << job['id'] + + next unless @error_result && job[keyword] == "failed" + + get_failed_result(result, job) + end + end + + def get_failed_result(result, job) + path = "/srv#{job['result_root']}/#{job['suite']}" + res = find_error_lines(path) + result['error.result']["path=#{path}"] = res + end + + def find_error_lines(path) + res = [] + return res unless File.exist?(path) + + File.open(path).each do |line| + res << line.to_s if line.match(/error:/i) end + res end def stats_count(result) diff --git a/sbin/es-jobs b/sbin/es-jobs index ff0d540..3804dc9 100755 --- a/sbin/es-jobs +++ b/sbin/es-jobs @@ -22,6 +22,7 @@ end opt_refine = [-1] opt_fields = [] opt_stats_filter = [] +opt_error_result = false opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-jobs [options] search_key1=val1[,val2..] ..' @@ -45,11 +46,15 @@ opt_parser = OptionParser.new do |opts| opts.on('-s fields', '--stats-filter fields', 'return data contains fields in stats') do |fields| opt_stats_filter = fields.split(',') end + + opts.on('-e', '--error.result', 'obtain the error information about job failure') do + opt_error_result = true + end end opt_parser.parse!(ARGV) items = parse_argv raise 'Please enter a query' if items.empty? -es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter) +es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter, opt_error_result) es_jobs.output -- 2.23.0

1 0

review star
by Wu Fengguang 18 Mar '21

18 Mar '21

Yu Chuan is our review star again, thanks YuChuan! @crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 29 4 10 12 Yu Chuan 76 7 9 10 Wei Jihui 71 2 2 8 Cao Xueliang 24 10 14 7 Xu Xijian 16 7 16 7 Luan Shengde 13 3 5 3 Wu Fengguang 18 6 10 3 Li Yuanchao 19 2 2 2 Liu Shaofei 21 4 7 2 Lu Weitao 16 5 5 2 Lin Jiaxin 19 3 5 1 Bai Jing 19 1 1 1 Zhang Dewan 3 1 2 0 Wang Yong 4 0 0 0 Cui Yili 21 0 0 0 Wu Zhende 6 0 0 0 Liu Yinsi 26 0 0 0 Li Ping 36 1 1 0 Wang Chenglong

1 0

git count
by Wu Fengguang 18 Mar '21

18 Mar '21

Jihui,Zhende is our code star, thanks Jihui,Zhende! weijihui@crystal ~/todo/bin% ./git-count 8 Wei Jihui 8 Wu Zhende 7 Liu Shaofei 7 Lu Weitao 7 Luan Shengde 6 Bai Jing 6 Xu Xijian 6 Liu Yinsi 6 Li Yuanchao 5 Cao Xueliang 5 Yu Chuan 3 Wang Chenglong 3 Lin Jiaxin 3 Li Ping 1 Cui Yili 1 Wang Yong 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale 0 Zhang Dewan

1 0

help star
by Wu Fengguang 18 Mar '21

18 Mar '21

Zhende is our help star, thanks Zhende! bijing@crystal ~/todo/bin% ./help-star 14.8 Wu Fengguang 13.2 Wu Zhende 13.0 Wei Jihui 12.7 Yu Chuan 10.9 Cao Xueliang 10.5 Wang Yong 8.0 Luan Shengde 5.5 Lu Weitao 5.3 Li Yuanchao 4.8 Liu Yinsi 4.8 Xu Xijian 4.7 Bai Jing 4.6 Cui Yili 3.9 Li Ping 3.7 Lin Jiaxin 3.2 Liu Shaofei 2.6 Zhang Dewan 2.6 Wang Chenglong 1.3 Du Kaitian 0.0 Zhang Yale

1 0

[PATCH compass-ci] src/*.cr: fix ameba warning
by Wang Yong 18 Mar '21

18 Mar '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- src/lib/sched.cr | 2 +- src/scheduler/testbox_env.cr | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index ac677de..b10a16d 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -224,7 +224,7 @@ class Sched @mq.publish_confirm(queue, msg) break rescue e - res = @mq.reconnect + @mq.reconnect sleep 5 end end diff --git a/src/scheduler/testbox_env.cr b/src/scheduler/testbox_env.cr index bf8cd8a..bdc1972 100644 --- a/src/scheduler/testbox_env.cr +++ b/src/scheduler/testbox_env.cr @@ -5,7 +5,7 @@ class Job def get_service_env hash = Hash(String, JSON::Any).new yaml_any = File.open("/etc/compass-ci/service/service-env.yaml") do |content| - YAML.parse(content).as_h? + YAML.parse(content).as_h? end return hash unless yaml_any -- 2.23.0

1 0

[PATCH lkp-tests] tests/deploy-cci: remove -e
by Liu Yinsi 18 Mar '21

18 Mar '21

[why] -e will cause process exit when return code is not 0, only use when debugging, so remove it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/deploy-cci | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 6f97d7151..102cc751c 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -1,4 +1,4 @@ -#!/bin/sh -e +#!/bin/sh # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -- 2.23.0

1 0