January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 2/2] assign-account: add 'lab' to default yaml
by Luan Shengde 08 Jan '21

08 Jan '21

the default yaml file will contain: my_email my_name lab lab is used to read the lab yaml according to its value. example: if the value of lab is 'z9', the sched will read token from file: z9.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index af08d91..7e0a5a7 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -173,6 +173,7 @@ class AccountStorage # they are added along with 'my_ssh_pubkey' when sending assign account request account_yaml['my_email'] = @data['my_email'] account_yaml['my_name'] = @data['my_name'] + account_yaml['lab'] = @data['lab'] File.open(default_yaml, 'w') do |f| f.puts account_yaml.to_yaml -- 2.23.0

2 1

[PATCH lkp-tests 1/2] lib/job.rb: add fault_reproduction
by Wei Jihui 08 Jan '21

08 Jan '21

do fault reproduction when submit a failed job Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 8346db87..631ebfc9 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -255,6 +255,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) + hash.merge!(fault_reproduction) if hash.has_key?("id") @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e @@ -285,6 +286,10 @@ class Job @jobfile = jobfile end + def fault_reproduction + wait = load_yaml("#{lkp_src}/jobs/fault_reproduction.yaml", {}) + end + def multi_args(hash) jobs_array = [] hash.each { |key, value| -- 2.23.0

2 1

[PATCH lkp-tests] bootstrap: don't try to install ntpdate
by Wu Fengguang 08 Jan '21

08 Jan '21

It's not necessary for every boot. Let's not slow down every test run. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- lib/bootstrap.sh | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index d4b41f3b..b04ea3dd 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -237,20 +237,6 @@ run_ntpdate() [ -n "$LKP_SERVER" ] || return is_clearlinux && clearlinux_timesync && return - has_cmd ntpdate || { - detect_system - local DISTRO=$_system_name_lowercase - [ "$DISTRO" == "unknown" ] && { - echo "[WARNING] Current os cannot be recognized by lkp-test, so skip run_ntpdate." - return - } - has_cmd ruby || { - echo "[WARNING] Cannot find ruby in $DISTRO, so skip run_ntpdate." - return - } - . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) - } - has_cmd ntpdate || return local hour="$(date +%H)" -- 2.23.0

1 0

[PATCH v3 lkp-tests] lib/bootstrap.sh: 'run-ipconfig' step should be before the check step
by Yu Chuan 08 Jan '21

08 Jan '21

[Why] The `run-ipconfig` step is the IP configuration step for initramfs; The `[ -z "$net_devices"]` step is check whether the network is normal. Now the `check normal` step is before 'ip configuration' step, so we encounter the following error: + export NO_NETWORK=1 + echo export NO_NETWORK=1 due to no net devices + return [Debug Msg] + [ ! -e /usr/share/initramfs-tools/scripts/functions ] + get_net_devices + local net_devices + local i + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + [ p0s1/ != /sys/class/net/enp0s1/ ] + basename /sys/class/net/enp0s1/ + net_devices= enp0s1 + [ != /sys/class/net/lo/ ] + continue + echo enp0s1 + local net_devices= enp0s1 + [ -z ] + warn_no_eth0 + [ -f /proc/config.gz ] + return + echo ls /sys/class/net + ls /sys/class/net + is_virt + [ -n ] + [ -n ] + has_cmd virt-what + command -v virt-what + grep -q -w hypervisor /proc/cpuinfo + return 0 + export NO_NETWORK=1 + echo export NO_NETWORK=1 due to no net devices + return Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7d15bb370fa9..770d550724f1 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -160,6 +160,9 @@ setup_network() return } + $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig + network_ok && return + local net_devices=$(get_net_devices) if [ -z "$net_devices" ]; then @@ -181,9 +184,6 @@ setup_network() fi fi - $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig - network_ok && return - local err_msg='IP-Config: Auto-configuration of network failed' dmesg | grep -q -F "$err_msg" || { # Include $err_msg in the error message so that it matches -- 2.23.0

2 3

[PATCH v3 lkp-tests] lib/bootstrap.sh: fix error when os is not integrated into lkp-tests
by Yu Chuan 08 Jan '21

08 Jan '21

[Why] If the current os has not been integrated into lkp-tests, run method `detect_system`, then var `$_system_name_lowercase` will be 'unknown'. So we should adapt this. [Error] + local DISTRO=unknown ++ echo ntpdate ++ /home/yuchuan/lkp-tests/sbin/adapt-packages unknown Traceback (most recent call last): 3: from /home/yuchuan/lkp-tests/sbin/adapt-packages:11:in `<main>' 2: from /usr/share/ruby/psych.rb:497:in `load_file' 1: from /usr/share/ruby/psych.rb:497:in `open' /usr/share/ruby/psych.rb:497:in `initialize': No such file or directory @ rb_sysopen - /home/yuchuan/lkp-tests/distro/adaptation/unknown (Errno::ENOENT) [How] If the current os has not been integrated into lkp-tests, then return. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7d15bb370fa9..912a76bb6687 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -240,6 +240,10 @@ run_ntpdate() has_cmd ntpdate || { detect_system local DISTRO=$_system_name_lowercase + [ "$DISTRO" == "unknown" ] && { + echo "[WARNING] Current os cannot be recognized by lkp-test, so skip run_ntpdate." + return + } . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) } -- 2.23.0

2 3

[PATCH v5 compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 08 Jan '21

08 Jan '21

before: support kvm for aarch64 after: support kvm for aarch64, x86_64, riscv64 test: /srv/result/host-info/2021-01-07/vm-8p64g/debian-sid-riscv64/z9.288315 /srv/result/host-info/2021-01-07/vm-8p64g/debian-sid-x86_64/z9.288332 /srv/result/host-info/2021-01-07/vm-8p64g/debian-sid-aarch64/z9.288335 [how] put public options like $kernel, $initrd, $smp together, list different options like $machine, $bios, $nic according to different qemu command, then merge all options. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 110 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 95 insertions(+), 15 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..a8a2b33 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,12 +18,6 @@ check_logfile() } } -qemu_command() -{ - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm -} - write_logfile() { ipxe_script=ipxe_script @@ -66,12 +60,56 @@ check_option_value() [ -n "$initrds" ] || exit } -get_initrd() +set_initrd() { initrd=initrd cat $initrds > $initrd } +set_bios() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd +} + +set_helper() +{ + helper=/usr/libexec/qemu-bridge-helper + [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper +} + +set_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + +set_device() +{ + device="virtio-net-device,netdev=net0,mac=${mac}" +} + +set_netdev() +{ + netdev="bridge,br=br0,id=net0,helper=${helper}" +} + +set_qemu() +{ + qemu=( + qemu-system-aarch64 + qemu-kvm + qemu-system-x86_64 + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} + print_message() { echo $SCHED_PORT @@ -83,36 +121,78 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} - "${kvm[@]}" --append "${append}" +run_qemu() +{ + case "$qemu" in + qemu-system-aarch64) + arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + [ "$(arch)" == "x86_64" ] && arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-x86_64) + arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-riscv64) + arch_option=( + -machine virt + -device virtio-net-device,netdev=net0,mac=${mac} + -netdev bridge,br=br0,id=net0,helper=${helper} + ) + ;; + esac + + "${kvm[@]}" "${arch_option[@]}" --append "${append}" } check_logfile write_logfile -qemu_command parse_ipxe_script check_option_value -get_initrd + +set_initrd +set_bios +set_helper +set_nic +set_device +set_netdev +set_qemu print_message +public_option run_qemu -- 2.23.0

3 4

[PATCH lkp-tests] lib/bootstrap.sh: avoid the problem caused by the lack of ruby pkg
by Yu Chuan 08 Jan '21

08 Jan '21

[Why] We use $LKP_SRC/sbin/adapt-packages to find the ntpdate pkg suitable for different systems. But we are not sure that ruby is pre-installed in each system. So add a check, if ruby is not pre-installed, return. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 30e4af828bcd..d4b41f3bd0d5 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -244,6 +244,10 @@ run_ntpdate() echo "[WARNING] Current os cannot be recognized by lkp-test, so skip run_ntpdate." return } + has_cmd ruby || { + echo "[WARNING] Cannot find ruby in $DISTRO, so skip run_ntpdate." + return + } . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) } -- 2.23.0

2 1

[PATCH lkp-tests] distro/depends: add vim in initramfs-common
by Zhang Yale 08 Jan '21

08 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- distro/depends/initramfs-common | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/initramfs-common b/distro/depends/initramfs-common index 5f766da6..a2ccf411 100644 --- a/distro/depends/initramfs-common +++ b/distro/depends/initramfs-common @@ -2,3 +2,4 @@ time wget perf which +vim -- 2.23.0

1 0

[PATCH lkp-tests] tests: add email
by Wei Jihui 08 Jan '21

08 Jan '21

send email to user when test job failed Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/email | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 tests/email diff --git a/tests/email b/tests/email new file mode 100755 index 00000000..fe870471 --- /dev/null +++ b/tests/email @@ -0,0 +1,41 @@ +#!/bin/sh +# - my_email +# - info + +: ${info:=failure} + +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/http.sh +. $PROJECT_DIR/lib/env.sh + +data_failure() +{ + data="To: $my_email +Subject: [NOTIFY Compass-ci] Test job failed + +Dear $my_username: + + Sorry to inform you that your test job is failed. you can click the following link to view details. + https://compass-ci.openeuler.org/$result_root + +Regards +Compass-Ci +" +} + +compose_email() +{ + [ "$info" == "failure" ] && data_failure +} + +run_email() +{ + [ -n "$my_email" ] || return 0 + my_username=$(echo "$my_email" | awk -F '@' '{print $1}') + compose_email + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" +} + +run_email -- 2.23.0

2 1

[PATCH v3 compass-ci] scheduler: libvirt: a new boot type of libvirt
by Ren Wen 08 Jan '21

08 Jan '21

Requested by: curl -X GET 'http://<host>:<port>/boot.libvirt/mac/<mac_value>' Response :: some job fields in json format: - jod_id - kernel_uri - initrds_uri - kernel_params - result - LKP_SERVER - templates : this field is for libvirt Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b2f0802..50ebe97 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -135,6 +135,28 @@ class Sched return response.to_json end + private def get_boot_libvirt(job : Job) + response = {} of String => String | Nil | Array(String) | JSON::Any + + response["job_id"] = job.id + response["kernel_uri"] = job.kernel_uri.lchop("kernel ") + + initrds_uri = job["initrds_uri"]? + initrds_uri = initrds_uri.as_a.map(&.as_s.lchop("initrd ")) if initrds_uri + response["initrds_uri"] = initrds_uri + + kernel_params = job["kernel_params"]? + kernel_params = kernel_params.as_a.map(&.to_s).reject(&.starts_with?("initrd=")).join(" ") if kernel_params + response["kernel_params"] = kernel_params + + response["result_root"] = job.result_root + response["LKP_SERVER"] = job["LKP_SERVER"]? + # *templates* is used by libvirt + response["templates"] = job["templates"]? + + return response.to_json + end + private def get_boot_grub(job : Job) initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" @@ -176,6 +198,8 @@ class Sched case boot_type when "ipxe" return job ? get_boot_ipxe(job) : ipxe_msg("No job now") + when "libvirt" + return job ? get_boot_libvirt(job) : {"job_id" => ""}.to_json when "grub" return job ? get_boot_grub(job) : grub_msg("No job now") when "container" -- 2.23.0

3 4