- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sparrow/0-package: adapt to x86_64 architectures
by Liu Yinsi 23 Oct '20

23 Oct '20

lockfile-create download link compatible with x86_64 and aarch64 architectures. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/openEuler | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index bd1f728..e408d77 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -24,7 +24,8 @@ pkgs=( yum install -y "${pkgs[@]}" -wget http://rpmfind.net/linux/openmandriva/cooker/repository/aarch64/main/releas… -wget http://rpmfind.net/linux/openmandriva/cooker/repository/aarch64/main/releas… +arch=$(uname -p) +wget http://rpmfind.net/linux/openmandriva/cooker/repository/${arch}/main/releas… +wget http://rpmfind.net/linux/openmandriva/cooker/repository/${arch}/main/releas… -rpm -i lib64lockfile1-1.16-1-omv4002.aarch64.rpm lockfile-progs-0.1.18-5-omv4002.aarch64.rpm +rpm -i lib64lockfile1-1.16-1-omv4002.${arch}.rpm lockfile-progs-0.1.18-5-omv4002.${arch}.rpm -- 2.23.0

3 3

[PATCH v2 lkp-tests] api-avx2neon: delete unnecessary fields in api-avx2neon.yaml
by Li Yuanchao 23 Oct '20

23 Oct '20

These fields will be configured in compass-ci/sbin/auto_submit.yaml Both configured in two places may cause unexcepted problems Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- jobs/api-avx2neon.yaml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/jobs/api-avx2neon.yaml b/jobs/api-avx2neon.yaml index 24de6913..f3b46bf0 100644 --- a/jobs/api-avx2neon.yaml +++ b/jobs/api-avx2neon.yaml @@ -1,11 +1,4 @@ suite: api-avx2neon category: functional -testbox: vm-pxe-hi1620-1p1g -arch: aarch64 - -os: centos -os_arch: aarch64 -os_version: 7 - api-avx2neon: -- 2.23.0

1 0

[PATCH v5 compass-ci 3/6] container/send-internet-mail: run.rb
by Luan Shengde 23 Oct '20

23 Oct '20

invoke mail-post/internet-smtp to run the send mail service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/run.rb | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/send-internet-mail/run.rb diff --git a/container/send-internet-mail/run.rb b/container/send-internet-mail/run.rb new file mode 100644 index 0000000..bd98da8 --- /dev/null +++ b/container/send-internet-mail/run.rb @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# mail-post have default smtp setup for sending intranet email +# internet-smtp is smtp config file for sending internet email +# when sending internet email, require internet-smtp, smtp setup in +# internet-smtp will override the default smtp setup in mail-post + +require 'sinatra' + +require "#{ENV['CCI_SRC']}/lib/mail-post" +require "#{ENV['CCI_SRC']}/lib/internet-smtp" -- 2.23.0

2 1

[PATCH lkp-tests] api-avx2neon: delete unnecessary fields
by Li Yuanchao 23 Oct '20

23 Oct '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- jobs/api-avx2neon.yaml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/jobs/api-avx2neon.yaml b/jobs/api-avx2neon.yaml index 24de6913..f3b46bf0 100644 --- a/jobs/api-avx2neon.yaml +++ b/jobs/api-avx2neon.yaml @@ -1,11 +1,4 @@ suite: api-avx2neon category: functional -testbox: vm-pxe-hi1620-1p1g -arch: aarch64 - -os: centos -os_arch: aarch64 -os_version: 7 - api-avx2neon: -- 2.23.0

1 0

[PATCH v5 compass-ci 4/6] container/send-internet-mail: Dockerfile
by Luan Shengde 23 Oct '20

23 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/Dockerfile | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 container/send-internet-mail/Dockerfile diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile new file mode 100644 index 0000000..598064b --- /dev/null +++ b/container/send-internet-mail/Dockerfile @@ -0,0 +1,11 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian +MAINTAINER shdluan(a)163.com +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y ruby-mail && \ + apt-get install -y ruby-json && \ + apt-get install -y ruby-sinatra -- 2.23.0

1 0

[PATCH compass-ci] fix: sched: cluster job needs more fields
by Ren Wen 23 Oct '20

23 Oct '20

[Why] Nodes need to communicate with each other when testing cluster jobs. Ips can bind to on macs after setting 'direct_mac' and 'direct_ips' for cluster jobsI [HowTo] 1) Need a cluster config file in $LKP_SRC/cluster, for example: ```yaml assumption: <assumption01> switch: switch01 ip0: 3 --- vm-2p8g--renwen: roles: [ server ] macs: [ "ec:f4:bb:cb:7b:92" ] vm-2p4g--renwen: roles: [ client ] macs: [ "ec:f4:bb:cb:54:90" ] ``` nr_node=2 cs1 switch=sw1 ip0=0 ($ip_prefix.0-.1) cs2 switch=sw1 ip0=2 ($ip_prefix.2-.3) "switch": only need if 2+ clusters share the same switch "ip0": only need if shared switch, and != 0 "assumption": the info about this cluster config 2) Scheduler get config from config file and set 'direct_macs' and 'direct_ips' for jobs. 3) When running a cluster job, bind direct_ips to direct_ macs in test environment. 4) Test client connect with daemon client to test job. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 070a16b..8922874 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -144,7 +144,7 @@ class Sched def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - return YAML.parse(File.read(cluster_file_path)).as_h + return YAML.parse_all(File.read(cluster_file_path)) end def get_commit_date(job) @@ -198,8 +198,13 @@ class Sched # collect all job ids job_ids = [] of String + basic_config = cluster_config[0] + net_id = basic_config["net_id"]? || "192.168.222" + ip0 = basic_config["ip0"].as_i + # switch = basic_config["switch"].to_s # useless now + # steps for each host - cluster_config.each do |host, config| + cluster_config[-1].as_h.each do |host, config| tbox_group = host.to_s job_id = add_task(tbox_group, lab) @@ -219,7 +224,15 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + direct_macs = config["macs"].as_a + direct_ips = [] of String + direct_macs.size.times do + raise "Host id is greater than 254, host_id: #{ip0}" if ip0 > 254 + direct_ips << "#{net_id}.#{ip0}" + ip0 += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

2 2

[PATCH v4 lkp-tests] feat(iso2rootfs): add step: submit rootfs test job after rootfs is made
by Yu Chuan 23 Oct '20

23 Oct '20

[Why] The produced dailybuild rootfs needs further testing by submit a job. [How] 1. Configure lkp env by {LKP_SRC}/sbin/install-dependencies.sh 2. Generate global compass-ci yaml. - The path of yaml: - /etc/compass-ci/defaults/sparrow.yaml - The yaml have two necessary fields: - SCHED_HOST - SCHED_PORT - The value of these two fields are consistent with the value in job.yaml of iso2rootfs. 3. Submit the job.yaml of rootfs test job with append follow params: - os - os_arch - os_version - os_mount - testbox Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 6 ++++ tests/iso2rootfs | 76 +++++++++++++++++++++++++++++++++++++++----- 2 files changed, 74 insertions(+), 8 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index f413c9cee1f0..644c64be06b4 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -11,3 +11,9 @@ iso2rootfs: rootfs_path: initrd_http_host: initrd_http_port: + test_yaml: + test_os: + test_os_arch: + test_os_version: + test_os_mount: + test_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index bf1f995e78df..e8a027b0dad2 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -53,12 +53,18 @@ check_yaml_vars() "rootfs_path" "initrd_http_host" "initrd_http_port" + "test_yaml" + "test_os" + "test_os_arch" + "test_os_version" + "test_os_mount" + "test_testbox" ) local yaml_t for yaml_t in "${yaml_vars[@]}" do - [ -n "$(eval echo "\$${yaml_t}")" ] || die "var$${yaml_t}$ is none" + [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." done } @@ -133,7 +139,7 @@ mount_rootfs() install_pkgs "cifs-utils" ;; *) - die "rootfs_protocol is none, exit !!!" + die "cannot support the rootfs_protocol: ${rootfs_protocol}." ;; esac @@ -164,7 +170,7 @@ check_sha256sum_update() if [ -n "$SHA256SUM_CACHE" ] then - [ -n "$SHA256SUM_NET" ] || die "get net sha256sum failed !!!" + [ -n "$SHA256SUM_NET" ] || die "cannot get sha256sum of dailybuild iso." if [ "$SHA256SUM_CACHE" == "$SHA256SUM_NET" ] then @@ -185,7 +191,9 @@ check_os() { log_info "starting check os ..." echo "${ALLOW_OS[@]}" | grep -wq "${os}" || - die "current host $${os}$ is not in $${ALLOW_OS[@]}$, exit !!!" + die "current os is not in allow os. + current os: ${os}. + allow os: $${ALLOW_OS[@]}$" } get_cgz() @@ -210,12 +218,12 @@ install_pkgs() local pkg_installer command -v yum && pkg_installer="yum" - [[ -n "$pkg_installer" ]] || die "can not find pkg installer !!!" + [[ -n "$pkg_installer" ]] || die "cannot find pkg installer." local pt for pt do - "$pkg_installer" install -y "$pt" || die "install pkg$$pt$ failed !!!" + "$pkg_installer" install -y "$pt" || die "cannot install pkg: $pt." done } @@ -364,7 +372,7 @@ test_rootfs() root_path="cifs:${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) - die "rootfs_protocol is none, exit !!!" + die "cannot support the rootfs_protocol: ${rootfs_protocol}." ;; esac get_qemu_efi_fd @@ -404,6 +412,57 @@ test_rootfs() EOF } +############ submit test job yaml ############ +generate_global_yaml() +{ + local config_yaml="/etc/compass-ci/defaults/sparrow.yaml" + + mkdir -p "$(dirname "${config_yaml}")" + echo "SCHED_HOST: ${SCHED_HOST}" >> "${config_yaml}" + echo "SCHED_PORT: ${SCHED_PORT}" >> "${config_yaml}" +} + +generate_submit_append_str() +{ + local wait_add_vars=( + "os" + "os_arch" + "os_version" + "os_mount" + "testbox" + ) + for vt in "${wait_add_vars[@]}" + do + SUBMIT_APPEND_STR="${SUBMIT_APPEND_STR} ${vt}=$(eval echo "\$test_${vt}")" + done +} + +submit_test_job_yaml() +{ + log_info "starting submit ${test_yaml} ..." + + [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + + cd "${LKP_SRC}" && { + chmod +x ./sbin/install-dependencies.sh + ./sbin/install-dependencies.sh + + generate_global_yaml + } + + cd "${LKP_SRC}/jobs" && { + [ -f "${test_yaml}" ] || + die "cannot find test yaml in LKP_SRC/jobs. + test yaml: ${test_yaml}." + + generate_submit_append_str + + "${LKP_SRC}/sbin/submit" ${SUBMIT_APPEND_STR} "${test_yaml}" + } + + log_info "submit ${test_yaml} finished" +} + ############ post works ############ post_works() { @@ -414,7 +473,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "$(basename ${ROOTFS_DES_DIR})" "${soft_link}" + ln -s "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" @@ -433,6 +492,7 @@ main() run_qcow2rootfs test_rootfs + submit_test_job_yaml post_works } -- 2.23.0

2 1

[PATCH compass-ci] job.cr: initrd modules and headers with initramfs
by Xu Xijian 23 Oct '20

23 Oct '20

When run job with initramfs, we should initrd modules.cgz and headers.cgz too. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8f5dd36..75d75e5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -303,10 +303,15 @@ class Job temp_initrds = [] of String osimage_dir = "#{SRV_INITRD}/osimage/#{os_dir}" + boot_dir = "#{SRV_OS}/#{os_dir}/boot" temp_initrds << "#{INITRD_HTTP_PREFIX}" + "#{JobHelper.service_path("#{osimage_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{boot_dir}/modules-default.cgz")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{boot_dir}/headers-default.cgz")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

2 1

[PATCH compass-ci] sparrow: install iproute
by Liu Yinsi 23 Oct '20

23 Oct '20

sparrow/2-network/br0 needs ip command. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 261592c..adfdfd5 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -20,6 +20,7 @@ pkgs=( liblockfile lockfile-progs busybox + iproute ) yum install -y "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 020c34a..48bb4c5 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -18,6 +18,7 @@ pkgs=( lockfile-progs bridge-utils busybox + iproute2 ) apt-get install -y "${pkgs[@]}" diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 5c64fe8..bd1f728 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -19,6 +19,7 @@ pkgs=( redis wget busybox + iproute ) yum install -y "${pkgs[@]}" -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor.rb: only one way to ssh
by Wu Zhende 22 Oct '20

22 Oct '20

[Why] Now, all the scenarios use "sshr" to login testbox. So delete the direct way to ssh. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index fcf7e220..9ce1d475 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -75,18 +75,9 @@ class Monitor return unless data['log'] data = JSON.parse(data['log']) - if job['uuid'] - # means Internet user - # using the redirection service to log in - return unless data['ssh_port'] + return unless data['ssh_port'] - ssh_connect(job['SCHED_HOST'], data['ssh_port'], web_socket) - else - # intranet user: direct login internal IP - return unless data['ip'] - - ssh_connect(data['ip'], 22, web_socket) - end + ssh_connect(job['SCHED_HOST'], data['ssh_port'], web_socket) end def ssh_connect(ssh_host, ssh_port, web_socket) -- 2.23.0

1 0