January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] lib/error_messages.rb: obtain_error_messages_by_error_id function add keyword argument
by Lin Jiaxin 18 Jan '21

18 Jan '21

Determine whether to return the error line or error messages by the value of return_error_line Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 3e1564b..d83d890 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -40,12 +40,12 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_error_id(error_id) + def obtain_error_messages_by_error_id(error_id, return_error_line = false) error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| if ('build-pkg.' + build_pkg_error_id(k)) == error_id - error_messages_by_error_id += v.to_a + error_messages_by_error_id += return_error_line ? [k] : v.to_a end end error_messages_by_error_id -- 2.23.0

2 1

[PATCH compass-ci 2/3] qemu/kvm.sh: optimize the prompt when the error occurs
by Yu Chuan 18 Jan '21

18 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- providers/qemu/kvm.sh | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 35678fbd44ef..f1d65272dbf9 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -52,15 +52,23 @@ parse_ipxe_script() check_option_value() { + [ -n "$kernel" ] || { + echo "[ERROR] Can not find job for current hostname: $hostname." + exit 1 + } + [ -s "$kernel" ] || { - echo "The kernel does not exist: $kernel" - exit + echo "[ERROR] Can not find kernel file or kernel file is empty: $kernel." + exit 1 } # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit + [ -n "$initrds" ] || { + echo "[ERROR] Can not find any initrd for current vm." + exit 1 + } } set_initrd() -- 2.23.0

3 4

[PATCH compass-ci] src/lib/web_backend.rb: fix wrong number of arguments
by Lu Weitao 18 Jan '21

18 Jan '21

[Why] wrong number of arguments (given 3, expected 4) 2021-01-18 11:49:46.000000000 +0800 web-backend: 172.17.0.1 - - [18/Jan/2021:03:49:44 UTC] "GET /compare?suite=iperf&dimension=os HTTP/1.1" 500 13 Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/lib/web_backend.rb | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index ea67064..61e3334 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -186,15 +186,15 @@ end def do_get_groups_matrices(must, dimension, total, size, from) job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_hash = combine_group_query_data(job_list, dimension) + matrices, suites_hash, latest_jobs_hash = combine_group_query_data(job_list, dimension) while matrices.empty? from += size break if from > total job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_hash = combine_group_query_data(job_list, dimension) + matrices, suites_hash, latest_jobs_hash = combine_group_query_data(job_list, dimension) end - [matrices, suites_hash] + [matrices, suites_hash, latest_jobs_hash] end def get_groups_matrices(conditions, dimension, must, size, from) @@ -209,11 +209,12 @@ end def get_compare_body(params) dimension, conditions = get_dimension_conditions(params) must = get_es_must(params) - groups_matrices, suites_hash = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) + groups_matrices, suites_hash, latest_jobs_hash = + get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) if !groups_matrices || groups_matrices.empty? body = 'No Data.' else - body = compare_group_matrices(groups_matrices, suites_hash, { no_print: true }) + body = compare_group_matrices(groups_matrices, suites_hash, latest_jobs_hash, { no_print: true }) body = 'No Difference.' if !body || body.empty? end return body -- 2.23.0

1 0

[PATCH compass-ci 3/3] my-qemu.sh: add `-h|--help`, `-d|--debug` params for my-qemu.sh
by Yu Chuan 18 Jan '21

18 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- providers/my-qemu.sh | 32 ++++++++++++++++++++++++++++++++ providers/qemu/kvm.sh | 5 +++-- providers/qemu/pxe.sh | 6 ++++-- 3 files changed, 39 insertions(+), 4 deletions(-) diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 8845a346e195..a8f09a610d3f 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,6 +2,38 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +help() +{ + cat <<-'EOF' + Usage: ./my-qemu.sh [OPTION]... + Ask scheduer service whether there is a job, and start a local vm if there is a job. + Scheduler api: http://${SCHED_HOST}:${SCHED_PORT}/boot.ipxe/mac/${mac} + + Mandatory arguments to long options are mandatory for short options too. + -h, --help display this help and exit + -d, --debug open the local vm serial port in the current shell + EOF + + exit 0 +} + +while true +do + [ $# -eq 0 ] && break + case "$1" in + -h|--help) + help + shift;; + -d|--debug) + export DEBUG=true + shift;; + *) + echo "[ERROR] Unknown param: $1" + help + exit 1;; + esac +done + [[ $tbox_group ]] || tbox_group=vm-2p8g export hostname=$tbox_group.$USER-$$ diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index f1d65272dbf9..a3cbe4248a3a 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -123,7 +123,7 @@ print_message() echo kernel: $kernel echo initrds: $initrds echo append: $append - echo less $log_file + [ "$DEBUG" == "true" ] || echo less $log_file sleep 5 } @@ -140,9 +140,10 @@ public_option() -k en-us -no-reboot -nographic - -serial file:${log_file} -monitor null ) + + [ "$DEBUG" == "true" ] || kvm=("${kvm[@]}" -serial file:${log_file}) } individual_option() diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index fa688331964a..2e205405c793 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -18,7 +18,7 @@ fi qemu=qemu-system-aarch64 command -v $qemu >/dev/null || qemu=qemu-kvm -echo less $serial_log +[ "$DEBUG" == "true" ] || echo less $serial_log kvm=( $qemu @@ -33,7 +33,9 @@ kvm=( -k en-us -no-reboot -nographic - -serial file:${serial_log} -monitor null ) + +[ "$DEBUG" == "true" ] || kvm=("${kvm[@]}" -serial file:${serial_log}) + "${kvm[@]}" -- 2.23.0

2 2

[PATCH compass-ci] sparrow/0-package: add packages
by Liu Yinsi 18 Jan '21

18 Jan '21

container os-cifs and os-nfs will use sudo command, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index b25dade..b8ef769 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -24,6 +24,7 @@ pkgs=( hostname wget curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 4d13413..947cbdc 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -24,6 +24,7 @@ pkgs=( qemu-system-aarch64 qemu-system-x86 qemu-system-riscv64 + sudo ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 04996f5..61a37c4 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -23,6 +23,7 @@ pkgs=( hostname libvirt curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http: mount /srv/cci/libvirt-xml to srv-http
by Xiao Shenwei 18 Jan '21

18 Jan '21

[why] libvirt support user define domain.xml file to start kvm the /srv/cci/libvirt-xml will save user's xml file Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index d9f461b..945cde0 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -16,6 +16,7 @@ cmd=( -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/os:/srv/os:ro -v /srv/git/archlinux:/srv/git/archlinux:ro + -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro -d srv-http ) -- 2.23.0

1 0

[PATCH compass-ci 1/3] qemu/kvm.sh: fix script will hang when there is no job
by Yu Chuan 18 Jan '21

18 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd When there is no job corresponding to current vm in the remote redis queue, var $initrds will be equal to "". And from `cat --help`, we can know the follow: cat Copy standard input to standard output. So the process will get stuck waiting for input. [How] Function check_option_value() is used to check whether $initrds or $kernel is empty. And $initrds and $kernel are assigned in the function parse_ipxe_script(). So move check_option_value() before set_initrd(). Signed-off-by: Yu Chuan <13186087857(a)163.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e9138a7c0..35678fbd44ef 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -197,8 +197,8 @@ write_logfile parse_ipxe_script -set_options check_option_value +set_options print_message -- 2.23.0

2 6

[PATCH v4 compass-ci] docker2rootfs: add unzip vmlinux function
by Wang Chenglong 18 Jan '21

18 Jan '21

[Why] Unpacked vmlinux into vmlinuz format. Signeo-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker2rootfs/common | 33 ++++++++++++++++++++++++++++- container/docker2rootfs/run | 1 + container/docker2rootfs/setup-image | 2 +- 3 files changed, 34 insertions(+), 2 deletions(-) diff --git a/container/docker2rootfs/common b/container/docker2rootfs/common index c26bbc8..e8c56ba 100755 --- a/container/docker2rootfs/common +++ b/container/docker2rootfs/common @@ -5,6 +5,8 @@ # This file is called by script run running on host # define functions and set host environment variables +. $CCI_SRC/container/qcow2rootfs/bin/common + get_host_aarch() { aarch=$(arch) @@ -133,7 +135,7 @@ get_rootfs_kernel() local vmlinuz_file local vmlinuz kernel - vmlinuz_file=$(find ./boot -name "vmlinuz-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz_file=$(find ./boot -name "vmlinu[z|x]-*" | grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel @@ -155,3 +157,32 @@ create_links_vmlinuz_initrd() echo "[INFO] Create links to initrd.lkp and vmlinuz success!" } + +unzip_vmlinuz() { + get_rootfs_kernel + + file ${ROOTFS_VMLINUZ_FILE} | grep -q gzip || return + + get_vmlinuz_start_postion ${ROOTFS_VMLINUZ_FILE} + [ $VMLINUZ_START_POSTION -eq -1 ] && return + + local unzip_str + + if [ $VMLINUZ_START_POSTION -eq 0 ]; then + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 | zcat > $ROOTFS_VMLINUZ.tmp" + else + unzip_str="dd if=$ROOTFS_VMLINUZ bs=1 skip=$VMLINUZ_START_POSTION | zcat > $ROOTFS_VMLINUZ.tmp" + fi + + echo "Unzipping vmlinuz..." + cd "$ROOTFS_DIR/boot" + $unzip_str + mv $ROOTFS_VMLINUZ.tmp $ROOTFS_VMLINUZ + chmod o+r $ROOTFS_VMLINUZ + + if [ $? -eq 0 ]; then + echo "[INFO] Unzipping vmlinuz success!" + else + echo "[WARNING] Unzipping vmlinuz failed!" + fi +} diff --git a/container/docker2rootfs/run b/container/docker2rootfs/run index 8a7b315..cdb19bf 100755 --- a/container/docker2rootfs/run +++ b/container/docker2rootfs/run @@ -31,3 +31,4 @@ start=( cp_rootfs $IMAGE_PACK $2 docker rm -f rootfs-docker create_links_vmlinuz_initrd $2 +unzip_vmlinuz diff --git a/container/docker2rootfs/setup-image b/container/docker2rootfs/setup-image index ffe4937..47bfb7e 100755 --- a/container/docker2rootfs/setup-image +++ b/container/docker2rootfs/setup-image @@ -38,7 +38,7 @@ setup_login() pack_cgz() { echo "Packing package. Please wait." - find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + find ./ ! -path "./tmp/$IMAGE_PACK" | cpio -o -Hnewc | gzip -9 > "./tmp/$IMAGE_PACK" chmod 644 /tmp/${IMAGE_PACK} } -- 2.23.0

1 0

[PATCH v3 compass-ci] sparrow/4-docker: fix logical error
by Liu Yinsi 18 Jan '21

18 Jan '21

before: if execute ./buildall and docker images exist, then ./build after: if execute ./buildall or docker images not exist, then ./build Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index f24226c..77fe43a 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -33,7 +33,7 @@ do_one() cd "$container" [ "$container_name" == 'ssh-r' ] && exit docker images | grep -wq "$container_name" - if [ "$?" == 0 ] && [ "$action" != "reboot" ]; then + if [ "$?" -ne 0 ] || [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install fi -- 2.23.0

5 7

[PATCH v2 compass-ci 1/2] lib/error_messages.rb: obtain_error_messages_by_error_id function add keyword argument
by Lin Jiaxin 18 Jan '21

18 Jan '21

Determine whether to return the error line or error messages by the value of eturn_error_line Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 3e1564b..d83d890 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -40,12 +40,12 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_error_id(error_id) + def obtain_error_messages_by_error_id(error_id, return_error_line = false) error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| if ('build-pkg.' + build_pkg_error_id(k)) == error_id - error_messages_by_error_id += v.to_a + error_messages_by_error_id += return_error_line ? [k] : v.to_a end end error_messages_by_error_id -- 2.23.0

2 1