October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] api-avx2neon: delete unnecessary fields
by Li Yuanchao 23 Oct '20

23 Oct '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- jobs/api-avx2neon.yaml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/jobs/api-avx2neon.yaml b/jobs/api-avx2neon.yaml index 24de6913..f3b46bf0 100644 --- a/jobs/api-avx2neon.yaml +++ b/jobs/api-avx2neon.yaml @@ -1,11 +1,4 @@ suite: api-avx2neon category: functional -testbox: vm-pxe-hi1620-1p1g -arch: aarch64 - -os: centos -os_arch: aarch64 -os_version: 7 - api-avx2neon: -- 2.23.0

1 0

[PATCH v5 compass-ci 4/6] container/send-internet-mail: Dockerfile
by Luan Shengde 23 Oct '20

23 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/Dockerfile | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 container/send-internet-mail/Dockerfile diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile new file mode 100644 index 0000000..598064b --- /dev/null +++ b/container/send-internet-mail/Dockerfile @@ -0,0 +1,11 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian +MAINTAINER shdluan(a)163.com +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y ruby-mail && \ + apt-get install -y ruby-json && \ + apt-get install -y ruby-sinatra -- 2.23.0

1 0

[PATCH compass-ci] fix: sched: cluster job needs more fields
by Ren Wen 23 Oct '20

23 Oct '20

[Why] Nodes need to communicate with each other when testing cluster jobs. Ips can bind to on macs after setting 'direct_mac' and 'direct_ips' for cluster jobsI [HowTo] 1) Need a cluster config file in $LKP_SRC/cluster, for example: ```yaml assumption: <assumption01> switch: switch01 ip0: 3 --- vm-2p8g--renwen: roles: [ server ] macs: [ "ec:f4:bb:cb:7b:92" ] vm-2p4g--renwen: roles: [ client ] macs: [ "ec:f4:bb:cb:54:90" ] ``` nr_node=2 cs1 switch=sw1 ip0=0 ($ip_prefix.0-.1) cs2 switch=sw1 ip0=2 ($ip_prefix.2-.3) "switch": only need if 2+ clusters share the same switch "ip0": only need if shared switch, and != 0 "assumption": the info about this cluster config 2) Scheduler get config from config file and set 'direct_macs' and 'direct_ips' for jobs. 3) When running a cluster job, bind direct_ips to direct_ macs in test environment. 4) Test client connect with daemon client to test job. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 070a16b..8922874 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -144,7 +144,7 @@ class Sched def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - return YAML.parse(File.read(cluster_file_path)).as_h + return YAML.parse_all(File.read(cluster_file_path)) end def get_commit_date(job) @@ -198,8 +198,13 @@ class Sched # collect all job ids job_ids = [] of String + basic_config = cluster_config[0] + net_id = basic_config["net_id"]? || "192.168.222" + ip0 = basic_config["ip0"].as_i + # switch = basic_config["switch"].to_s # useless now + # steps for each host - cluster_config.each do |host, config| + cluster_config[-1].as_h.each do |host, config| tbox_group = host.to_s job_id = add_task(tbox_group, lab) @@ -219,7 +224,15 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + direct_macs = config["macs"].as_a + direct_ips = [] of String + direct_macs.size.times do + raise "Host id is greater than 254, host_id: #{ip0}" if ip0 > 254 + direct_ips << "#{net_id}.#{ip0}" + ip0 += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

2 2

[PATCH v4 lkp-tests] feat(iso2rootfs): add step: submit rootfs test job after rootfs is made
by Yu Chuan 23 Oct '20

23 Oct '20

[Why] The produced dailybuild rootfs needs further testing by submit a job. [How] 1. Configure lkp env by {LKP_SRC}/sbin/install-dependencies.sh 2. Generate global compass-ci yaml. - The path of yaml: - /etc/compass-ci/defaults/sparrow.yaml - The yaml have two necessary fields: - SCHED_HOST - SCHED_PORT - The value of these two fields are consistent with the value in job.yaml of iso2rootfs. 3. Submit the job.yaml of rootfs test job with append follow params: - os - os_arch - os_version - os_mount - testbox Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 6 ++++ tests/iso2rootfs | 76 +++++++++++++++++++++++++++++++++++++++----- 2 files changed, 74 insertions(+), 8 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index f413c9cee1f0..644c64be06b4 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -11,3 +11,9 @@ iso2rootfs: rootfs_path: initrd_http_host: initrd_http_port: + test_yaml: + test_os: + test_os_arch: + test_os_version: + test_os_mount: + test_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index bf1f995e78df..e8a027b0dad2 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -53,12 +53,18 @@ check_yaml_vars() "rootfs_path" "initrd_http_host" "initrd_http_port" + "test_yaml" + "test_os" + "test_os_arch" + "test_os_version" + "test_os_mount" + "test_testbox" ) local yaml_t for yaml_t in "${yaml_vars[@]}" do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "var$${yaml_t}$ is none" + [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." done } @@ -133,7 +139,7 @@ mount_rootfs() install_pkgs "cifs-utils" ;; *) - die "rootfs_protocol is none, exit !!!" + die "cannot support the rootfs_protocol: ${rootfs_protocol}." ;; esac @@ -164,7 +170,7 @@ check_sha256sum_update() if [ -n "$SHA256SUM_CACHE" ] then - [ -n "$SHA256SUM_NET" ] || die "get net sha256sum failed !!!" + [ -n "$SHA256SUM_NET" ] || die "cannot get sha256sum of dailybuild iso." if [ "$SHA256SUM_CACHE" == "$SHA256SUM_NET" ] then @@ -185,7 +191,9 @@ check_os() { log_info "starting check os ..." echo "${ALLOW_OS[@]}" | grep -wq "${os}" || - die "current host $${os}$ is not in $${ALLOW_OS[@]}$, exit !!!" + die "current os is not in allow os. + current os: ${os}. + allow os: $${ALLOW_OS[@]}$" } get_cgz() @@ -210,12 +218,12 @@ install_pkgs() local pkg_installer command -v yum && pkg_installer="yum" - [[ -n "$pkg_installer" ]] || die "can not find pkg installer !!!" + [[ -n "$pkg_installer" ]] || die "cannot find pkg installer." local pt for pt do - "$pkg_installer" install -y "$pt" || die "install pkg$$pt$ failed !!!" + "$pkg_installer" install -y "$pt" || die "cannot install pkg: $pt." done } @@ -364,7 +372,7 @@ test_rootfs() root_path="cifs:${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) - die "rootfs_protocol is none, exit !!!" + die "cannot support the rootfs_protocol: ${rootfs_protocol}." ;; esac get_qemu_efi_fd @@ -404,6 +412,57 @@ test_rootfs() EOF } +############ submit test job yaml ############ +generate_global_yaml() +{ + local config_yaml="/etc/compass-ci/defaults/sparrow.yaml" + + mkdir -p "$(dirname "${config_yaml}")" + echo "SCHED_HOST: ${SCHED_HOST}" >> "${config_yaml}" + echo "SCHED_PORT: ${SCHED_PORT}" >> "${config_yaml}" +} + +generate_submit_append_str() +{ + local wait_add_vars=( + "os" + "os_arch" + "os_version" + "os_mount" + "testbox" + ) + for vt in "${wait_add_vars[@]}" + do + SUBMIT_APPEND_STR="${SUBMIT_APPEND_STR} ${vt}=$(eval echo "\$test_${vt}")" + done +} + +submit_test_job_yaml() +{ + log_info "starting submit ${test_yaml} ..." + + [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + + cd "${LKP_SRC}" && { + chmod +x ./sbin/install-dependencies.sh + ./sbin/install-dependencies.sh + + generate_global_yaml + } + + cd "${LKP_SRC}/jobs" && { + [ -f "${test_yaml}" ] || + die "cannot find test yaml in LKP_SRC/jobs. + test yaml: ${test_yaml}." + + generate_submit_append_str + + "${LKP_SRC}/sbin/submit" ${SUBMIT_APPEND_STR} "${test_yaml}" + } + + log_info "submit ${test_yaml} finished" +} + ############ post works ############ post_works() { @@ -414,7 +473,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "$(basename ${ROOTFS_DES_DIR})" "${soft_link}" + ln -s "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" @@ -433,6 +492,7 @@ main() run_qcow2rootfs test_rootfs + submit_test_job_yaml post_works } -- 2.23.0

2 1

[PATCH compass-ci] job.cr: initrd modules and headers with initramfs
by Xu Xijian 23 Oct '20

23 Oct '20

When run job with initramfs, we should initrd modules.cgz and headers.cgz too. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8f5dd36..75d75e5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -303,10 +303,15 @@ class Job temp_initrds = [] of String osimage_dir = "#{SRV_INITRD}/osimage/#{os_dir}" + boot_dir = "#{SRV_OS}/#{os_dir}/boot" temp_initrds << "#{INITRD_HTTP_PREFIX}" + "#{JobHelper.service_path("#{osimage_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{boot_dir}/modules-default.cgz")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{boot_dir}/headers-default.cgz")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

2 1

[PATCH compass-ci] sparrow: install iproute
by Liu Yinsi 23 Oct '20

23 Oct '20

sparrow/2-network/br0 needs ip command. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 261592c..adfdfd5 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -20,6 +20,7 @@ pkgs=( liblockfile lockfile-progs busybox + iproute ) yum install -y "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 020c34a..48bb4c5 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -18,6 +18,7 @@ pkgs=( lockfile-progs bridge-utils busybox + iproute2 ) apt-get install -y "${pkgs[@]}" diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 5c64fe8..bd1f728 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -19,6 +19,7 @@ pkgs=( redis wget busybox + iproute ) yum install -y "${pkgs[@]}" -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor.rb: only one way to ssh
by Wu Zhende 22 Oct '20

22 Oct '20

[Why] Now, all the scenarios use "sshr" to login testbox. So delete the direct way to ssh. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index fcf7e220..9ce1d475 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -75,18 +75,9 @@ class Monitor return unless data['log'] data = JSON.parse(data['log']) - if job['uuid'] - # means Internet user - # using the redirection service to log in - return unless data['ssh_port'] + return unless data['ssh_port'] - ssh_connect(job['SCHED_HOST'], data['ssh_port'], web_socket) - else - # intranet user: direct login internal IP - return unless data['ip'] - - ssh_connect(data['ip'], 22, web_socket) - end + ssh_connect(job['SCHED_HOST'], data['ssh_port'], web_socket) end def ssh_connect(ssh_host, ssh_port, web_socket) -- 2.23.0

1 0

[PATCH compass-ci] fix: sched: add 2 fields into job: direct_macs and direct_ips
by Ren Wen 22 Oct '20

22 Oct '20

[Why] Nodes need to communicate with each other when testing cluster jobs. After adding 2 fileds: 'direct_macs' and 'direct_ips', can bind ips to macs. [HowTo] When you test a cluster job, need to config cluster file correctly. Below is the template: ```yaml <hostname>: roles: [ <"server">, ... ] macs: [ <mac>, ... ] ips: [ <ip>, ... ] <hostname>: roles: [ <"client">, ... ] macs: [ <mac>, ... ] ips: [ <ip>, ... ] ``` For 'ips': 1) should be the intranet IP 2) all ips in cluster file should be in the same network segment 3) subnet mask is 24. [InFuture] Set default ips when there's no ips in cluster file or only one host has ips. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 7215df5..a26091b 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -219,7 +219,8 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + job["direct_macs"] = config["macs"].as_a.join(" ") + job["direct_ips"] = config["ips"].as_a.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

4 4

[PATCH lkp-tests] tests: fix lmbench3 tool adapt to aarch64 framwork
by Bai Jing 22 Oct '20

22 Oct '20

error: cat: results/x84_64-linux-gnu/taishan200-2280-2s48p-256g--a28.0 No such file or directory Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/lmbench3 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/lmbench3 b/tests/lmbench3 index 8570a35a..f09c49e2 100755 --- a/tests/lmbench3 +++ b/tests/lmbench3 @@ -138,7 +138,7 @@ run_lmbench3_development() eval "$test=yes" - sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/x86_64-linux-gnu/lmbench + sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/aarch64-linux-gnu/lmbench log_eval " ( echo $nr_threads @@ -177,7 +177,7 @@ run_lmbench3_development() show_result() { - cat results/x86_64-linux-gnu/$testbox.0 || { + cat results/aarch64-linux-gnu/$testbox.0 || { echo "no test result found" exit -1 } -- 2.23.0

3 3

[PATCH compass-ci] sbin: complier => compiler
by Cao Xueliang 22 Oct '20

22 Oct '20

[why] The container crystal-compiler build a image and named alpine:crystal-compiler, the sbin crystal-format use the image. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/crystal-format | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/crystal-format b/sbin/crystal-format index 14f82b2..99f7a6a 100755 --- a/sbin/crystal-format +++ b/sbin/crystal-format @@ -15,7 +15,7 @@ cmd=( --rm -v $v_dir:$v_dir -w $PWD - alpine:crystal-complier + alpine:crystal-compiler crystal tool format "$@" ) -- 2.23.0

3 2