October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests 2/2] jobs: add test-tools.yaml
by Li Ping 20 Oct '20

20 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- jobs/test-tools.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 jobs/test-tools.yaml diff --git a/jobs/test-tools.yaml b/jobs/test-tools.yaml new file mode 100644 index 00000000..1f711dcf --- /dev/null +++ b/jobs/test-tools.yaml @@ -0,0 +1,6 @@ +suite: test-tools +category: functional + +test-tools: + testsuite: + case: -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] delete the field: testcase
by Li Ping 20 Oct '20

20 Oct '20

[why] there is no need to add the field: testcase in the $job.yaml Signed-off-by: Li Ping <15396232681(a)163.com> --- bin/lkp-setup-rootfs | 2 +- lib/bootstrap.sh | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 09f7e7eb..05a3325c 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -51,7 +51,7 @@ job_does_not_need_reboot() } grep -q "^do_not_reboot_for_same_kernel: 1$" $job && - is_same_kernel_and_rootfs && is_same_testcase && is_same_bp_memmap && { + is_same_kernel_and_rootfs && is_same_suite && is_same_bp_memmap && { echo "LKP: [do_not_reboot_for_same_kernel] start a new job without rebooting" return 0 } diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 2506c021..a7776d68 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -631,12 +631,12 @@ is_same_kernel_and_rootfs() return 1 } -is_same_testcase() +is_same_suite() { - local current_testcase=$testcase - local next_testcase=$(awk '/^testcase: /{print $2}' $job | tr -d '"') + local current_suite=$suite + local next_suite=$(awk '/^suite: /{print $2}' $job | tr -d '"') - [ "$current_testcase" = "$next_testcase" ] + [ "$current_suite" = "$next_suite" ] } is_same_bp_memmap() -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: submit with yaml file without directory
by Wei Jihui 20 Oct '20

20 Oct '20

if get string named with *.yaml from ARGV, treat it as job yaml. even it is without directory. and find_jobfile check whether the job yaml exists. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 831bfb5a..979e3dca 100755 --- a/sbin/submit +++ b/sbin/submit @@ -60,7 +60,7 @@ options.parse!(ARGV) find_job_yaml = false ARGV.delete_if do |arg| - find_job_yaml = true if File.file?(arg) + find_job_yaml = true if arg =~ /\.yaml$/ if arg.index '=' if find_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') -- 2.23.0

3 2

[PATCH v2 compass-ci] add exception handing for docker.rb
by Xiao Shenwei 20 Oct '20

20 Oct '20

[why] when run multi-docker job on physical machine, run 50 client to request job, if server down or error like Connection refused. these client willexit. when the server recovers, need to run these client again. [how] add exception handing to ensure all client does not exit Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index c241d3c..3600025 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -88,6 +88,19 @@ def main(hostname) run(hostname, load_path, hash) end +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.backtrace + sleep 25 + ensure + sleep 5 + end + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -99,10 +112,7 @@ def multi_docker(hostname, nr_container) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop do - main("#{hostname}-#{i}") - sleep 5 - end + loop_main("#{hostname}-#{i}") end pids << pid end -- 2.23.0

2 2

[PATCH v3 compass-ci] feat: build centos initramfs image with one command
by Wang Chenglong 20 Oct '20

20 Oct '20

Use the tool to create a new centos-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/packages-to-install. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/centos/README.md | 16 ++++++++ container/osimage/centos/create-image | 43 ++++++++++++++++++++ container/osimage/centos/lib | 41 +++++++++++++++++++ container/osimage/centos/packages-to-install | 9 ++++ container/osimage/centos/packages-to-remove | 11 +++++ container/osimage/centos/run | 27 ++++++++++++ 6 files changed, 147 insertions(+) create mode 100644 container/osimage/centos/README.md create mode 100755 container/osimage/centos/create-image create mode 100755 container/osimage/centos/lib create mode 100644 container/osimage/centos/packages-to-install create mode 100644 container/osimage/centos/packages-to-remove create mode 100755 container/osimage/centos/run diff --git a/container/osimage/centos/README.md b/container/osimage/centos/README.md new file mode 100644 index 0000000..b5c7b07 --- /dev/null +++ b/container/osimage/centos/README.md @@ -0,0 +1,16 @@ ++# Use the tool to create a new centos-${version} initramfs image. ++ ++Usage: ++ cd ${CCI_SRC}/rootfs/initramfs/centos/aarch64/${version}/ ++ ./build ++ ++Some configuration items: ++./bin/packages-to-install ++ If you want to pre-install the software, you can write the package names in ./bin/packages-to-install. ++ ++./bin/packages-to-remove ++ If you want remove some unnecessary files, you can write the names in ./bin/packages-to-remove. ++ ++$HOME/.config/compass-ci/rootfs-passwd ++ Set the password for the image into this file. + diff --git a/container/osimage/centos/create-image b/container/osimage/centos/create-image new file mode 100755 index 0000000..a3495b2 --- /dev/null +++ b/container/osimage/centos/create-image @@ -0,0 +1,43 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Install functions +install_packages() +{ + echo "install" + yum repolist + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +# Setup functions +setup_rootfs() +{ + echo "setup" + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + + +# Config password +setup_login() { + [ -n "$ROOT_NEW_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${image_name}" |grep -vf /tmp/packages-to-remove | cpio -o -Hnewc | gzip -9 > /tmp/${image_name} + chmod 644 /tmp/${image_name} +} + +install_packages +setup_rootfs +setup_login +pack_cgz diff --git a/container/osimage/centos/lib b/container/osimage/centos/lib new file mode 100755 index 0000000..53aae2a --- /dev/null +++ b/container/osimage/centos/lib @@ -0,0 +1,41 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${os_name}:${os_version}" +image_name="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs-passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker_image() +{ + docker pull $image + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[INFO] Pull docker image failed." + exit + fi +} + +# cp image package to host +cp_image() +{ + docker cp -a init_docker:/tmp/${image_name} $HOME/ + echo "result: $(ls $HOME/${image_name})" +} diff --git a/container/osimage/centos/packages-to-install b/container/osimage/centos/packages-to-install new file mode 100644 index 0000000..a09bc28 --- /dev/null +++ b/container/osimage/centos/packages-to-install @@ -0,0 +1,9 @@ +openssh-server +openssl +cifs-utils +hostname +curl +iproute +iputils +which +time diff --git a/container/osimage/centos/packages-to-remove b/container/osimage/centos/packages-to-remove new file mode 100644 index 0000000..07a4a23 --- /dev/null +++ b/container/osimage/centos/packages-to-remove @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/packages-to-remove +/tmp/packages-to-install diff --git a/container/osimage/centos/run b/container/osimage/centos/run new file mode 100755 index 0000000..699bbc6 --- /dev/null +++ b/container/osimage/centos/run @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/packages-to-remove:/tmp/packages-to-remove + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker_image +${cmd[@]} +cp_image +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

2 2

[PATCH v4 compass-ci] feat: build centos initramfs image with one command
by Wang Chenglong 20 Oct '20

20 Oct '20

Use the tool to create a new centos-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/packages-to-install. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/centos/README.md | 16 ++++++++ container/osimage/centos/create-image | 43 ++++++++++++++++++++ container/osimage/centos/lib | 41 +++++++++++++++++++ container/osimage/centos/packages-to-install | 9 ++++ container/osimage/centos/packages-to-remove | 11 +++++ container/osimage/centos/run | 27 ++++++++++++ 6 files changed, 147 insertions(+) create mode 100644 container/osimage/centos/README.md create mode 100755 container/osimage/centos/create-image create mode 100755 container/osimage/centos/lib create mode 100644 container/osimage/centos/packages-to-install create mode 100644 container/osimage/centos/packages-to-remove create mode 100755 container/osimage/centos/run diff --git a/container/osimage/centos/README.md b/container/osimage/centos/README.md new file mode 100644 index 0000000..b5c7b07 --- /dev/null +++ b/container/osimage/centos/README.md @@ -0,0 +1,16 @@ ++# Use the tool to create a new centos-${version} initramfs image. ++ ++Usage: ++ cd ${CCI_SRC}/rootfs/initramfs/centos/aarch64/${version}/ ++ ./build ++ ++Some configuration items: ++./bin/packages-to-install ++ If you want to pre-install the software, you can write the package names in ./bin/packages-to-install. ++ ++./bin/packages-to-remove ++ If you want remove some unnecessary files, you can write the names in ./bin/packages-to-remove. ++ ++$HOME/.config/compass-ci/rootfs-passwd ++ Set the password for the image into this file. + diff --git a/container/osimage/centos/create-image b/container/osimage/centos/create-image new file mode 100755 index 0000000..fac8ddc --- /dev/null +++ b/container/osimage/centos/create-image @@ -0,0 +1,43 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Install functions +install_packages() +{ + echo "install" + yum repolist + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +# Setup functions +setup_rootfs() +{ + echo "setup" + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + + +# Config password +setup_login() { + [ -n "$ROOT_NEW_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${image_name}" | grep -vf /tmp/packages-to-remove | cpio -o -Hnewc | gzip -9 > /tmp/${image_name} + chmod 644 /tmp/${image_name} +} + +install_packages +setup_rootfs +setup_login +pack_cgz diff --git a/container/osimage/centos/lib b/container/osimage/centos/lib new file mode 100755 index 0000000..53aae2a --- /dev/null +++ b/container/osimage/centos/lib @@ -0,0 +1,41 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${os_name}:${os_version}" +image_name="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs-passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker_image() +{ + docker pull $image + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[INFO] Pull docker image failed." + exit + fi +} + +# cp image package to host +cp_image() +{ + docker cp -a init_docker:/tmp/${image_name} $HOME/ + echo "result: $(ls $HOME/${image_name})" +} diff --git a/container/osimage/centos/packages-to-install b/container/osimage/centos/packages-to-install new file mode 100644 index 0000000..a09bc28 --- /dev/null +++ b/container/osimage/centos/packages-to-install @@ -0,0 +1,9 @@ +openssh-server +openssl +cifs-utils +hostname +curl +iproute +iputils +which +time diff --git a/container/osimage/centos/packages-to-remove b/container/osimage/centos/packages-to-remove new file mode 100644 index 0000000..07a4a23 --- /dev/null +++ b/container/osimage/centos/packages-to-remove @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/packages-to-remove +/tmp/packages-to-install diff --git a/container/osimage/centos/run b/container/osimage/centos/run new file mode 100755 index 0000000..699bbc6 --- /dev/null +++ b/container/osimage/centos/run @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/packages-to-remove:/tmp/packages-to-remove + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker_image +${cmd[@]} +cp_image +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

3 6

[PATCH v3 compass-ci] src/lib: find depends_initrd for multitask
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the code need multi-qemu, not multi-qemu-x, and scheduler can not find the depends_initrd multi-qemu-x.cgz. Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..93bc712 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,9 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + if program =~ /(.*)-\d+$/ + program = $1 + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +415,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

1 0

[PATCH compass-ci] src/lib: find depends_initrd for multitask
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the code need multi-qemu, not multi-qemu-x, and scheduler can not find the depends_initrd multi-qemu-x.cgz. Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..a0d7263 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,9 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + if program =~ /(.*)(\-\d{1,}$)/ + program = $1 + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +415,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq!.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq!.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

1 0

[PATCH compass-ci] code-spec: add some rules
by Wu Fengguang 20 Oct '20

20 Oct '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/code-spec.md b/doc/code-spec.md index a859239..d405cdd 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -4,8 +4,11 @@ code注意事项 - 代码应当放进函数里不要在脚本顶层直接写逻辑 - 代码应当放进lib/里脚本本身最小化, 仅解析命令行参数, 调用lib/入口函数 - 如果出现了全局变量一般意味着需要写一个 class/module 来放这个状态量了 -- 一个函数, 不要超过10行代码 +- 一个函数, 不要超过10行代码, 该原则适用于所有ruby/crystal/shell各类程序. - 起准确和有意义的变量/函数名字, 名字要让人看出赋予他们的确切意义 +- 函数要有分层结构. 每个函数须有明确的定位: 业务层逻辑, 还是底层支持性routine. + 一个函数不能同时做两个层次的事. 同一函数内避免混杂不同类型的事务. +- 如果注释仅仅是代码的简单重复, 不提供额外信息量, 不要写. 重构友好编码 ============ -- 2.23.0

1 0

[PATCH v2 compass-ci] src/lib: for multitask like multi-qemu
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the code need multi-qemu, not multi-qemu-x. Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..a48f233 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,9 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + if program =~ /(.*)(\-\d{1,}$)/ + program_num = $1 + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +415,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq!.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq!.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

2 2

[PATCH v3 compass-ci] fix: sched: set 2 fields to job: direct_macs, direct_ips
by Ren Wen 20 Oct '20

20 Oct '20

[Why] nodes need communicate with each other when running cluster jobs. after adding 2 fileds, 'direct_ips' and 'direct_macs', can bind ips to macs. 'macs' get from cluster file ('$LKP_SRC/cluster/*'), 'ips' set by scheduler default: 192.168.<211..254>.<host_id> Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a18bdee..1d5b5c4 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -198,6 +198,10 @@ class Sched # collect all job ids job_ids = [] of String + # used to set ips + ip_prefix = "192.168." + host_id = 1 + # steps for each host cluster_config.each do |host, config| tbox_group = host.to_s @@ -219,7 +223,18 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + + # host macs and host ips + direct_macs = config["macs"].as_a + direct_ips = [] of String + net_id = "#{ip_prefix}#{254 - job_id.hash%44}" # 192.168.<211..254> + direct_macs.size.times do + raise "more than 255 macs" if host_id > 255 + direct_ips << "#{net_id}.#{host_id}" + host_id += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

2 2

[PATCH compass-ci] /src/lib/web_backend.rb: fix "/compare_candidates" tbox_group regex bug
by Zhang Yuhang 20 Oct '20

20 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 48cfadb..fda01ca 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -47,7 +47,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end -- 2.23.0

2 2

[PATCH lab-z9] hosts: add new hostfile
by Zhang Yu 20 Oct '20

20 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a101 | 22 ++++++++++++++++++++++ hosts/taishan200-2280-2s48p-256g--a102 | 24 ++++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a101 create mode 100644 hosts/taishan200-2280-2s48p-256g--a102 diff --git a/hosts/taishan200-2280-2s48p-256g--a101 b/hosts/taishan200-2280-2s48p-256g--a101 new file mode 100644 index 0000000..1226913 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a101 @@ -0,0 +1,22 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b89192e1 + - /dev/disk/by-id/scsi-350000399b88bf66d + - /dev/disk/by-id/scsi-350000399b88bef85 +ssd_partitions: + - /dev/disk/by-id/ata-INTEL_SSDSC2KB480G8_BTYF94730FF4480BGN +mac_addr: + - 44:67:47:d7:6c:9c + - 44:67:47:d7:6c:9d + - 44:67:47:d7:6c:9e + - 44:67:47:d7:6c:9f + - 44:67:47:d7:6c:a0 + - 44:67:47:d7:6c:a1 + - 44:67:47:d7:6c:a2 + - 44:67:47:d7:6c:a3 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.6 diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-256g--a102 new file mode 100644 index 0000000..267a962 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a102 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 512G +nr_hdd_partitions: 2 +nr_ssd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88bfbc9 + - /dev/disk/by-id/scsi-350000399b8919395 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH1T9HMLT-00005_S455NY0MB55506 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH1T9HMLT-00005_S455NY0MB55187 + - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000335 + - nvme-nvme.19e5-30333242454a31304743303030333335-48574533365034333031324d3030304e-00000001-part* +mac_addr: + - 44:67:47:c9:ea:30 + - 44:67:47:c9:ea:31 + - 44:67:47:c9:ea:32 + - 44:67:47:c9:ea:33 + - 44:67:47:c9:ea:34 + - 44:67:47:c9:ea:35 + - 44:67:47:c9:ea:36 + - 44:67:47:c9:ea:37 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.7 -- 2.23.0

1 0

[PATCH compass-ci] src/lib: for multitask like multi-qemu
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the scheduler can not find the program multi-qemu-0 and the program multi-qemu-1 Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..e563ddf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,10 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + program_num = program.match(/(.*)(\-\d{1,}$)/) + if program_num != nil + program = program_num.not_nil![1] + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +416,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq!.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq!.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

3 3

[PATCH v2 compass-ci] fix: sched: set 2 fields to job: direct_macs, direct_ips
by Ren Wen 20 Oct '20

20 Oct '20

[Why] nodes need communicate with each other when running cluster jobs. after adding 2 fileds, 'direct_ips' and 'direct_macs', can bind ips to macs. 'macs' get from cluster file ('$LKP_SRC/cluster/*'), 'ips' set by scheduler default. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a18bdee..82d3610 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -198,6 +198,10 @@ class Sched # collect all job ids job_ids = [] of String + # used to set ips + ip_prefix = "192.169." + host_id = 1 + # steps for each host cluster_config.each do |host, config| tbox_group = host.to_s @@ -219,7 +223,17 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + + # host macs and host ips + direct_macs = config["macs"].as_a + direct_ips = [] of String + net_id = "#{ip_prefix}#{job_id.hash%255}" # 192.169.<0..254> + direct_macs.size.times do + direct_ips << "#{net_id}.#{host_id}" + host_id += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

1 1

[PATCH v7 compass-ci 2/6] container/mail-robot: answerback-email.rb
by Luan Shengde 20 Oct '20

20 Oct '20

AssignUuid: read email content check email available check if base url in upstream-repos check if commit url valid check pub_key assign account generate uuid write account/uuid to es send uuid to user Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/answerback-email.rb | 229 +++++++++++++++++++++++ 1 file changed, 229 insertions(+) create mode 100755 container/mail-robot/answerback-email.rb diff --git a/container/mail-robot/answerback-email.rb b/container/mail-robot/answerback-email.rb new file mode 100755 index 0000000..66d13e4 --- /dev/null +++ b/container/mail-robot/answerback-email.rb @@ -0,0 +1,229 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '/c/compass-ci/lib/es_client.rb' + +# Assign uuid/account +class AssignUuid + def initialize(mail_content) + @mail_content = mail_content + @send_mail_server = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + end + + def build_message(sum_infos) + message = <<~EMAIL_MESSAGE + To: #{sum_infos['my_email']} + Subject: Account Ready + + Dear #{sum_infos['my_name']} + + Thank you for joining us. + You can use the following info to submit jobs: + + First: add the info to ~/.config/compass-ci/defaults/{USER}.yaml + + uuid: #{sum_infos['my_uuid']} + lab: z9 + + Second: download lkp-tests and dependencies + git clone https://gitee.com_/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source {HOME}/.bashrc && source {HOME}/.bash_profile + + Third: submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + how to write job yaml + submit jobs/netperf.yaml + + lastly: add your own information and uuid to ~/.config/compass-ci/default/{USER}.yml + my_name: {your name} + my_email: {your email address} + my_uuid: {applied uuid} + + Repalce the fields {...} with your local environment variable. + + regards + compass-ci + EMAIL_MESSAGE + + return message + end + + def build_err_message(email, message) + message = <<~EMAIL_MESSAGE + To: #{email} + Subject: apply account failed + + Dear #{email.split('@')[0]} + + Your application for account failed with following errors + #{message} + + regards + compass-ci + EMAIL_MESSAGE + end + + def send_mail(email_msg) + %x(curl -XPOST "#{@send_mail_server}:#{ENV['S_M_PORT']}/send_mail_text" -d "#{email_msg}") + end + + def url_addr(email) + mail_body = @mail_content.part[0].body.decoded + mail_content = mail_body.gsub(/=/, '').split(/\r|\n|=/).join + url = %x(echo '#{mail_content}' |grep -o -E "https?://[^/]*/[^/]*/[^/]*/[^/]*/[0-9a-zA-Z]{40}").split(/\n/)[0] + + if url.nil? + message = 'No commit url found' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise message + end + + return url + end + + def check_email_available(email) + url = url_addr(email).chomp + + base_url = url.gsub(%r{/[^/]*/[0-9a-zA-Z]{40}$}, '') + + upstream_dir = '/c/upstream-repos' + check_url_in_upstream_repos(upstream_dir, base_url, email) + check_url_commit(url, base_url, email) + + return url + end + + def check_url_in_upstream_repos(upstream_dir, base_url, email) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + if match_out.empty? + message = 'The url is not in upstream-repo list' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise StandardError, message, base_url + end + end + + def check_url_commit(url, base_url, email) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + check_gitee_commit(url, base_url, email) if hub_name.eql? 'gitee.com' + check_non_gitee_commit(url, email) unless hub_name.eql? 'gitee.com' + end + + def check_non_gitee_commit(url, email) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index email + + if email_index.nil? + message = 'Your commit url is not a valid commit url.' + email_msg = build_err_message(email, message) + send_mail(email_msg) + raise StandardError, message, url + end + end + + def check_gitee_commit(url, base_url, email) + repo_info = { + 'repos_dir' => '/repos_dir', + 'repo_dir' => url.split('/')[-3], + 'repo_url' => [base_url, 'git'].join('.'), + 'repo_path' => File.join('/repos_dir', url.split('/')[-3]), + 'commit_id' => url.split('/')[-1] + } + check_gitee_repo_commit(url, email, repo_info) + end + + def check_gitee_repo_commit(url, email, repo_info) + Dir.chdir repo_info['repos_dir'] + %x(/usr/bin/git clone #{repo_info['repo_url']}) + email_index = %x(/usr/bin/git -C #{repo_info['repo_dir']} show #{repo_info['commit_id']}).index email + + FileUtils.rm_rf repo_info['repo_dir'] + + if email_index.nil? + message = 'Your commit url is not a valid commit url.' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise StandardError, message, url + end + end + + def pub_key_value(email) + pub_key = @mail_content.part[1].body.decoded.split(/\r|\n/).join if @mail_content.part[1].filename == 'id_rsa.pub' + + unless pub_key + message = 'No pub_key found' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise message + end + + return pub_key + end + + def account_info(pub_key, email) + account_info_str = %x(curl -XGET "#{ENV['J_HOST']}:#{ENV['J_PORT']}/assign_account" -d "pub_key: #{pub_key}") + account_info = JSON.parse account_info_str + + if account_info['account'].empty? + message = 'No more available jumper account.' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise StandardError, message, account_info if account_info['account'].empty? + end + + return account_info + end + + def build_sum_infos + email = @mail_content.from[0] + my_name = @mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + commit_url = check_email_available(email) + pub_key = pub_key_value(email) + acct_infos = account_info(pub_key, email) + account_uuid = %x(uuidgen).chomp + sum_infos = { + 'my_email' => email, + 'my_name' => my_name, + 'my_commit_url' => commit_url, + 'my_login_name' => acct_infos['account'], + 'my_uuid' => account_uuid, + } + store_account_info(sum_infos) + sum_infos['acct_infos'] = acct_infos + sum_infos + end + + def send_uuid(sum_infos) + email_msg = build_message(sum_infos) + + send_mail(email_msg) + end + + def send_account + sum_infos = build_sum_infos + + send_uuid(sum_infos) + end + + def store_account_info(sum_infos) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(sum_infos['my_email'], sum_infos) + end +end -- 2.23.0

2 4

[PATCH lkp-tests 1/2] jobs: add test-tools.yaml
by Li Ping 20 Oct '20

20 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- jobs/test-tools.yaml | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 jobs/test-tools.yaml diff --git a/jobs/test-tools.yaml b/jobs/test-tools.yaml new file mode 100644 index 00000000..153fefd0 --- /dev/null +++ b/jobs/test-tools.yaml @@ -0,0 +1,10 @@ +suite: test-tools +testcase: test-tools +category: functional + +os_mount: cifs +os_version: 20.03-dailybuild + +test-tools: + testsuite: + case: -- 2.23.0

2 1

[PATCH lkp-tests] bin: mkdir /etc/pki/ca-trust/source/anchors
by Hu Xuejiao 20 Oct '20

20 Oct '20

When run job-task, appeat warning: cannot create /etc/pki/ca-trust/source/anchors/docker_registry_proxy.crt: Directory nonexistent Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- bin/lkp-setup-rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index d1b17a9c..09930990 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -139,6 +139,7 @@ install_docker_proxy() Environment="HTTPS_PROXY=http://$DOCKER_PROXY_HOST:$DOCKER_PROXY_PORT/" EOF + mkdir -p /etc/pki/ca-trust/source/anchors curl http://$DOCKER_PROXY_HOST:$DOCKER_PROXY_PORT/ca.crt > \ /etc/pki/ca-trust/source/anchors/docker_registry_proxy.crt || return update-ca-trust extract -- 2.23.0

1 0

[PATCH lkp-tests] bin: update ~ to /root
by Hu Xuejiao 20 Oct '20

20 Oct '20

When run job-task, appear warning: /lkp/lkp/src/bin/lkp-setup-rootfs: 154: cannot create ~/.gemrc: Directory nonexistent Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- bin/lkp-setup-rootfs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 09f7e7eb..d1b17a9c 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -151,7 +151,7 @@ EOF modify_gem_source() { - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc + echo ':sources: ["http://rubygems.org"]' >> /root/.gemrc return 0 } -- 2.23.0

1 0

[PATCH compass-ci] fix: sched: set 2 fields to job: direct_macs, direct_ips
by Ren Wen 20 Oct '20

20 Oct '20

cluster nodes need communicate with each other, so set 2 fields: 1) direct_macs: get from cluster file ($LKP_SRC/cluster/*) 2) direct_ips: set by scheduler default (they're local ips) It will add ips to corresponding devices before nodes executing jobs. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a18bdee..62b7b9d 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -198,6 +198,11 @@ class Sched # collect all job ids job_ids = [] of String + # set net id, and host id + # ip = net_id + set_id + net_id = "192.168.211." + host_id = 1 + # steps for each host cluster_config.each do |host, config| tbox_group = host.to_s @@ -219,7 +224,16 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + + # host macs and host ips + direct_macs = config["macs"].as_a + direct_ips = [] of String + direct_macs.size.times do + direct_ips << "#{net_id}#{host_id}" + host_id += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

2 1

[PATCH lkp-tests 2/2] delete the field: testcase
by Li Ping 20 Oct '20

20 Oct '20

[why] there is no need to add the field: testcase in the $job.yaml Signed-off-by: Li Ping <15396232681(a)163.com> --- bin/lkp-setup-rootfs | 2 +- lib/bootstrap.sh | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 09f7e7eb..05a3325c 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -51,7 +51,7 @@ job_does_not_need_reboot() } grep -q "^do_not_reboot_for_same_kernel: 1$" $job && - is_same_kernel_and_rootfs && is_same_testcase && is_same_bp_memmap && { + is_same_kernel_and_rootfs && is_same_suite && is_same_bp_memmap && { echo "LKP: [do_not_reboot_for_same_kernel] start a new job without rebooting" return 0 } diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 2506c021..a7776d68 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -631,12 +631,12 @@ is_same_kernel_and_rootfs() return 1 } -is_same_testcase() +is_same_suite() { - local current_testcase=$testcase - local next_testcase=$(awk '/^testcase: /{print $2}' $job | tr -d '"') + local current_suite=$suite + local next_suite=$(awk '/^suite: /{print $2}' $job | tr -d '"') - [ "$current_testcase" = "$next_testcase" ] + [ "$current_suite" = "$next_suite" ] } is_same_bp_memmap() -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests: add get_ipmi_ip function to host-info
by Zhang Yu 20 Oct '20

20 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tests/host-info b/tests/host-info index a57855dc..9ca06235 100755 --- a/tests/host-info +++ b/tests/host-info @@ -80,6 +80,12 @@ get_partitions() fi } +get_ipmi_ip() +{ + imp_ip=$(ipmitool lan print |awk 'NR==3{print $4}') + [ $? -eq 0 ] && echo "ipmi_ip:" $imp_ip +} + main() { get_node @@ -87,6 +93,7 @@ main() get_partitions get_mac get_model_name + get_ipmi_ip } main -- 2.23.0

4 3

[PATCH lkp-tests] sshd: send emails to login services for users
by Zhang Yale 20 Oct '20

20 Oct '20

[Why] Provide a login port for user by email, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 99 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 74 insertions(+), 25 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..cd57be4f 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,16 @@ #!/bin/sh # pub_key -# email +# my_email +# my_uuid +# ssh_port_len +# ssh_port_base + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 +[ -n "$ssh_port_len" ] || ssh_port_len=2000 +[ -n "$ssh_port_base" ] || ssh_port_base=50000 run_ssh() { @@ -11,40 +21,79 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") - data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh - -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: - - Login: - ssh root@$PUB_IP - Due time: - $deadline - - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch + data="To: $my_email +Subject: [NOTIFY Compass-ci] $testbox ready to use + +Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-ci +" +} + +data_failure() +{ + data="To: $my_email +Subject: [NOTIFY Compass-ci] Applying $testbox failed + +Dear $my_username: + Sorry to inform you that your application failed. + You may need to wait for a while and then try again. Regards -compass-ci +Compass-ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + +set_port() +{ + [ -n "$my_uuid" ] || return + + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%"$ssh_port_len"+"$ssh_port_base")) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ + -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] && return + done + port="" +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$my_email" ] && [ -n "$my_uuid" ] || return 0 + my_username=$(echo "$my_email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +set_port +report_ssh_port $port run_email -- 2.23.0

3 5

[PATCH v9 lkp-tests] sshd: send emails to login services for users
by Zhang Yale 19 Oct '20

19 Oct '20

Provide a login port for user by email, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 89 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 20 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..55015b40 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,16 @@ #!/bin/sh # pub_key # email +# my_uuid +# ssh_port_len +# ssh_port_base + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 +[ -n "$sshr_port_len" ] || sshr_port_len=2000 +[ -n "$sshr_port_base" ] || sshr_port_base=30000 run_ssh() { @@ -11,40 +21,79 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh +Subject: [NOTIFY Compass-ci] $testbox ready to use + +Dear $my_name: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch - Login: - ssh root@$PUB_IP - Due time: - $deadline +Regards +Compass-ci +" +} + +data_failure() +{ + data="To: $email +Subject: [NOTIFY Compass-ci] Applying $testbox failed - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch +Dear $my_name: + Sorry to inform you that your application failed. + You may need to wait for a while and then try again. Regards -compass-ci +Compass-ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + +set_port() +{ + [ -n "$my_uuid" ] || return + + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ + -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] && return + done + port="" +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$email" ] && [ -n "$my_uuid" ] || return 0 + my_name=$(echo "$email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +set_port +report_ssh_port $port run_email -- 2.23.0

3 3

[PATCH compass-ci] job.cr: initramfs and nfs use the same kernel file
by Xu Xijian 19 Oct '20

19 Oct '20

Actually the vmlinuz file in SRV_INITRD is copy from SRV_OS, so they are the same file, just use the same path is enough. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..fc15441 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -272,18 +272,9 @@ class Job return true end - private def vmlinuz - if os_mount == "initramfs" - return "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/vmlinuz")}" - else - return "#{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" - end - end - private def set_kernel_uri - self["kernel_uri"] = "kernel #{vmlinuz()}" + self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" end private def kernel_common_params -- 2.23.0

1 0

[PATCH lab-z9] allot/idle: remove the field -hi1620 from tbox_group
by Li Ping 19 Oct '20

19 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml index 4525fcd..513b631 100644 --- a/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml +++ b/allot/idle/taishan200-2280-2s48p-256g--a60/multi-qemu.yaml @@ -8,11 +8,11 @@ os_mount: initramfs simplify-ci: multi-qemu-0: nr_vm: 20 - tbox_group: vm-hi1620-2p8g + tbox_group: vm-2p8g multi-qemu-1: nr_vm: 2 - tbox_group: vm-hi1620-2p4g + tbox_group: vm-2p4g multi-qemu-2: nr_vm: 2 - tbox_group: vm-hi1620-1p1g + tbox_group: vm-1p1g sleep: 100d -- 2.23.0

1 0

[PATCH v7 compass-ci 1/6] container/mail-robot: mail-robot.rb
by Luan Shengde 19 Oct '20

19 Oct '20

mail robot for email application monitor mailbox check new added email file in new check new mail's subject return if unmatched subject call answerback-email check to apply uuid/account Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/mail-robot.rb | 43 ++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..70b8b25 --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'answerback-email.rb' + +mail_inbox = '/srv/cci/Maildir/.compass-ci/new/' +mail_drafts = '/srv/cci/Maildir/.compass-ci/cur/' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + unless added.empty? + added.each do |mail_file| + begin + check_to_send_account(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + puts e.backtrace + end + end + end + end + listener.start + sleep +end + +def check_to_send_account(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + subject = mail_content.subject + return unless subject =~ /apply account/i + + assign_uuid = AssignUuid.new(mail_content) + assign_uuid.send_account + FileUtils.mv(mail_file, mail_drafts) +end + +monitor_dir(mail_inbox, mail_drafts) -- 2.23.0

3 4

[PATCH lkp-tests] lib/job.rb: load hosts file for atomic job
by Wei Jihui 19 Oct '20

19 Oct '20

[why] for commit c725e0f022b7: def each_jobs(&block) each_job_init - load_hosts_config job = deepcopy(@job) @job2 = {} load_defaults + load_hosts_config each_job_init each_job(&block) @jobs.each do |hash| job = deepcopy(@job) this @job has no load hosts config, so it should do it under "@jobs.each do |hash|" for atomic job. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..6359254e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -696,9 +696,11 @@ class Job each_job_init each_job(&block) @jobs.each do |hash| + @load_hosts_done = false @job = deepcopy(job) @job2 = hash load_defaults + load_hosts_config each_job_init each_job(&block) end -- 2.23.0

2 2

[PATCH v7 compass-ci 5/6] container/mail-robot: start file
by Luan Shengde 19 Oct '20

19 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/start | 39 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..108ed83 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,39 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_INTERNET_MAIL_PORT +] + +defaults = relevant_defaults(names) + +JUMPER_HOST = defaults['JUMPER_HOST'] +JUMPER_PORT = defaults['JUMPER_PORT'] +SEND_MAIL_PORT = defaults['SEND_INTERNET_MAIL_PORT'] + +cmd = %W[ + docker run + --restart=always + --name=mail-robot + -d + -e J_HOST=#{JUMPER_HOST} + -e J_PORT=#{JUMPER_PORT} + -e S_M_PORT=#{SEND_MAIL_PORT} + -v /c/upstream-repos:/c/upstream-repos:ro + -v /srv/cci/Maildir:/srv/cci/Maildir:rw + -v /c/compass-ci:/c/compass-ci:ro + mail-robot +] + +cmd += ['bash', '-c', 'umask 002 && ruby ./mail-robot.rb'] + +system(*cmd) -- 2.23.0

2 2

[PATCH v2 compass-ci] sched: throw an exception when deps/pkg does not exist
by Cao Xueliang 19 Oct '20

19 Oct '20

When submitting a job, an exception is thrown when deps/pkg does not exist on the server. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..7785c2a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -389,10 +389,12 @@ class Job program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") && File.exists?(deps_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") && File.exists?(pkg_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) end end -- 2.23.0

2 5

[PATCH v2 lkp-tests] sbin/submit: submit with yaml file without directory
by Wei Jihui 19 Oct '20

19 Oct '20

if get string named with *.yaml from ARGV, treat it as job yaml. even it is without directory. and find_jobfile check whether the job yaml exists. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 831bfb5a..ce8accf0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -60,7 +60,7 @@ options.parse!(ARGV) find_job_yaml = false ARGV.delete_if do |arg| - find_job_yaml = true if File.file?(arg) + find_job_yaml = true if arg =~ /^.+\.yaml$/ if arg.index '=' if find_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') -- 2.23.0

2 1

[PATCH v2 lkp-tests 1/2] feat(iso2rootfs): modify the rootfs dir layout and rootfs dir name
by Yu Chuan 19 Oct '20

19 Oct '20

[Why] 1. Modify the dir structure of dailybuild rootfs dir. 2. Modify the dir name of dailybuild rootfs dir. [How] before: "{os}/{os_arch}/debug-versions/dailybuild/{iso_version}-%Y%m%d%H%M%S" after: "{os}/{os_arch}/{iso_version}-%Y-%m-%d-%H-%M-%S" Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 524ab6695f93..89328f2f4a9b 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,6 @@ get_daily_iso_checksum() mount_rootfs() { - DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +102,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-latest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +254,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +309,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -366,8 +365,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ - "${soft_link}" + ln -s "$(basename ${ROOTFS_DES_DIR})" "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 1

[PATCH compass-ci 1/2] /lib/es_query.rb: add web-backend common es methods
by Zhang Yuhang 19 Oct '20

19 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- lib/es_query.rb | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index aaaadc2..9a4e45a 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -9,12 +9,20 @@ require_relative 'constants.rb' class ESQuery HOST = (ENV.key?('ES_HOST') ? ENV['ES_HOST'] : ES_HOST) PORT = (ENV.key?('ES_PORT') ? ENV['ES_PORT'] : ES_PORT).to_i - def initialize(host = HOST, port = PORT, index: 'jobs') + def initialize(host = HOST, port = PORT, index: 'jobs*') @index = index @client = Elasticsearch::Client.new url: "http://#{host}:#{port}" raise 'Connect Elasticsearch error!' unless @client.ping end + def es_query(query) + @client.search index: @index, body: query + end + + def es_count(query) + @client.count(index: @index, body: query)['count'] + end + # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) def multi_field_query(items, size: 10_000) @@ -26,7 +34,23 @@ class ESQuery } }, size: size } - @client.search index: 'jobs*', body: query + es_query(query) + end + + def all_values(field, size: 1000) + query = { + aggs: { + all_field: { + terms: { field: field, size: size } + } + }, + size: 0 + } + es_result = es_query(query)['aggregations']['all_field']['buckets'] + es_result.sort_by! { |h| h['doc_count'] } + es_result.reverse!.map! { |x| x['key'] } + + es_result end def query_by_id(id) -- 2.23.0

3 4

[PATCH lkp-tests] sbin/submit: submit with yaml file without directory
by Wei Jihui 19 Oct '20

19 Oct '20

if get string named with *.yaml from ARGV, task it as job yaml. even it is without directory. and find_jobfile check whether the job yaml exists. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 831bfb5a..ce8accf0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -60,7 +60,7 @@ options.parse!(ARGV) find_job_yaml = false ARGV.delete_if do |arg| - find_job_yaml = true if File.file?(arg) + find_job_yaml = true if arg =~ /^.+\.yaml$/ if arg.index '=' if find_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') -- 2.23.0

2 1

[PATCH compass-ci] fix: sched: alter conditions when submitting cluster and normal jobs
by Ren Wen 19 Oct '20

19 Oct '20

1) when a job without 'cluster' (a job field), or has 'cluster' but starts with 'cs-localhost', the job is a normal job; otherwise, it's a cluster job. 2) alter the numbers of return values of 'get_cluster_config' function, to only return 'cluster config'. Because the conditions how to judge a job is whether a cluster job are changed. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 33 ++++++++++++--------------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a4ddcec..a18bdee 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -139,22 +139,12 @@ class Sched return node_state end - # EXAMPLE: - # cluster_file: "cs-lkp-hsw-ep5" - # return: Hash(YAML::Any, YAML::Any) | Nil, 0 | <hosts_size> - # {"lkp-hsw-ep5" => {"roles" => ["server"], "macs" => ["ec:f4:bb:cb:7b:92"]}, - # "lkp-hsw-ep2" => {"roles" => ["client"], "macs" => ["ec:f4:bb:cb:54:92"]}}, 2 + # get cluster config using own lkp_src cluster file, + # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - - if File.file?(cluster_file_path) - cluster_config = YAML.parse(File.read(cluster_file_path)).as_h - hosts_size = cluster_config.values.size - return cluster_config, hosts_size - end - - return nil, 0 + return YAML.parse(File.read(cluster_file_path)).as_h end def get_commit_date(job) @@ -177,16 +167,17 @@ class Sched job = Job.new(job_content, job_content["id"]?) job["commit_date"] = get_commit_date(job) + # it is not a cluster job if cluster field is empty or + # field's prefix is 'cs-localhost' cluster_file = job["cluster"] - if cluster_file != "" - cluster_config, hosts_size = get_cluster_config( - cluster_file, job.lkp_initrd_user, job.os_arch) - - return submit_cluster_job( - job, cluster_config.not_nil!) if hosts_size >= 2 + if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") + return submit_single_job(job) + else + cluster_config = get_cluster_config(cluster_file, + job.lkp_initrd_user, + job.os_arch) + return submit_cluster_job(job, cluster_config) end - - return submit_single_job(job) rescue ex puts ex.inspect_with_backtrace return [{ -- 2.23.0

2 3

[PATCH v2 compass-ci] container/assign-account: answerback-email.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add parameters for answerback-email -e,--email email_addr: add email address -s,--ssh-pubkey pub_key: add ssh pub_key -f,--raw-email mail_file: add email_adress[,pub_key] via email file why: easier for administrator to manully assign jumper account for user with: - user's email - user's email with pub_key - user's email_file Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 165 ++++++++----------- 1 file changed, 70 insertions(+), 95 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index b5283ba..0789c38 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -3,46 +3,18 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -=begin - -repo_list: - all repos url list from upstream-repos.git - -API: -call graph: -read_mail_content -send_account_request - email_addr - get email address - email_message_id - get email message_id - check_email_available - check email available - pub_key_value - get pub key - account - send apply account request and return account info - build_message - build email message - send_mail - call send_mail to send mail whit build message - -the returned data for account_info like: -{ - "account" => "guest", - "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", - "jumper_port" => "10000" -} -=end +# xx --email xxx --login --ssh-pubkey xxx --raw-email email-file +# samba mount +# ssh logshn (huawei, ) (install pubkey / send password) require 'json' require 'mail' require 'set' -require_relative '../defconfig.rb' +require 'optparse' +require_relative '../defconfig' names = Set.new %w[ - JUMPER_IP + JUMPER_HOST JUMPER_PORT CRYSTAL_INTRANET SEND_MAIL_PORT @@ -50,27 +22,71 @@ names = Set.new %w[ defaults = relevant_defaults(names) -JUMPER_IP = defaults['JUMPER_IP'] +JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] CRYSTAL_INTRANET = defaults['CRYSTAL_INTRANET'] SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] -def build_message(email, message_id, infos) +apply_info = { + 'email_addr' => nil, + 'pub_key' => nil +} + +def format_info(email_file, apply_info) + mail_content = Mail.read(email_file) + + email_addr = mail_content.from[0] + pub_key = mail_content.part[1].body.decoded.split(/\r|\n/).join if mail_content.part[1].filename == 'id_rsa.pub' + + apply_info['email_addr'] = email_addr + apply_info['pub_key'] = pub_key + + apply_info +end + +options = OptionParser.new do |opts| + opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key] [--raw-email email_file]\n" + opts.banner += " -e or -f is required\n" + opts.banner += ' -s is optional when use -e' + + opts.separator '' + opts.separator 'options:' + + opts.on('-e email_addr', '--email email_addr', 'appoint email address') do |email_addr| + apply_info['email_addr'] = email_addr + end + + opts.on('-s pub_key', '--ssh-pubkey pub_key', 'ssh pub_key to enable keyless login') do |pub_key| + apply_info['pub_key'] = pub_key + end + + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + format_info(email_file, apply_info) + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +options.parse!(ARGV) + +def build_message(email, acct_infos) message = <<~EMAIL_MESSAGE To: #{email} - Message-ID: #{message_id} Subject: jumper account is ready - Dear #{email} + Dear user: Thank you for joining us. You can use the following command to login the jumper server: login command: - ssh -p #{infos['jumper_port']} #{infos['account']}@#{infos['jumper_ip']} + ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} account passwd: - account_password: #{infos['passwd']} + account_password: #{acct_infos['passwd']} regards compass-ci @@ -79,67 +95,26 @@ def build_message(email, message_id, infos) return message end -def email_addr(mail_content) - msg = 'not an applying account email' - - raise msg unless mail_content.subject =~ /apply ssh account/i - - email = mail_content.from.join(',') - - return email -end - -# def check_email_available(mail_content, email) -# oos_list = File.read('/c/upstream-repos/repo_list').split(/\n/) -# url = mail_content.body.decoded.split(/\n/).find { |line| line =~ /https?:\/\// } -# base_url = url.split('/')[0,5].join('/') -# message = 'The url is not in upstream repo_list' -# -# raise message unless oos_list.include? base_url -# -# url_fdback = %x(curl #{url}) -# email_index = url_fdback.index email -# -# message = 'No commit info found from the url for the email' -# raise message unless email_index -# end - -def email_message_id(mail_content) - message_id = mail_content.message_id - return message_id -end - -def pub_key_value(mail_content) - pub_key = mail_content.body.decoded.split(/\n/).find { |line| line =~ /ssh-rsa/ } - return pub_key -end - def account_info(pub_key) - account_info_str = %x(curl -XGET '#{JUMPER_IP}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - account_info = JSON.parse account_info_str - - return account_info + account_info_str = if pub_key.nil? + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') + else + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + end + JSON.parse account_info_str end -def send_account(mail_content) - email = email_addr(mail_content) - message_id = email_message_id(mail_content) - # check_email_available(mail_content, email) +def send_account(apply_info) + message = "No email address specified\n" + message += "use -e email_addr add a email address\n" + message += 'or use -f to add a mail file' + raise message if apply_info['email_addr'].nil? - pub_key = pub_key_value(mail_content) - acct_info = account_info(pub_key) + acct_info = account_info(apply_info['pub_key']) - message = build_message(email, message_id, acct_info) + message = build_message(apply_info['email_addr'], acct_info) %x(curl -XPOST '#{CRYSTAL_INTRANET}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -def read_mail_content(mail_file) - mail_content = Mail.read(mail_file) - - return mail_content -end - -mail_file = ARGV[0] -mail_content = read_mail_content(mail_file) -send_account(mail_content) +send_account(apply_info) -- 2.23.0

2 1

[PATCH compass-ci] job.cr: set_pp_initrd() => set_depends_initrd()
by Xu Xijian 19 Oct '20

19 Oct '20

Function set_pp_initrd() is confusing, since what it does is to set depends, so change function name to set_depends_initrd. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 20afa65..1e44d15 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -133,7 +133,7 @@ class Job set_result_root() set_result_service() set_os_mount() - set_pp_initrd() + set_depends_initrd() set_initrds_uri() set_kernel_uri() set_kernel_append_root() @@ -391,7 +391,7 @@ class Job self["user_lkp_src"] = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) end - private def set_pp_initrd + private def set_depends_initrd initrd_deps_arr = Array(String).new initrd_pkg_arr = Array(String).new initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" -- 2.23.0

3 5

[PATCH lkp-tests] /tests/build-pkg: modify save location for *.cgz file
by Liu Shaofei 19 Oct '20

19 Oct '20

1. modify *.cgz file save path /srv/initrd/build-pkg/nasm-git => /srv/initrd/build-pkg/initramfs/openeuler/aarch64/20.03/nasm-git 2. create softlink name that named by tag ../nasm-git/f71820d6055373c9b9381615e28a28f307bb2df4.CGZ ../nasm-git/1.0.4.cgz -> f71820d6055373c9b9381615e28a28f307bb2df4.cgz Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..65cb6dc7 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,10 +23,14 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" @@ -75,12 +81,22 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 } +create_softlink() +{ + [ -n "$upstream_tag" ] || return + [ -e "$PKG_MNT/${pack_to}/${upstream_tag}.cgz" ] && return + + ln -sf "$(basename $(realpath ${cgz_name}))" "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" || return + echo "create soft link: $PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz -> $(basename $(realpath ${cgz_name}))" +} + check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

2 1

[PATCH compass-ci 2/2] user-client/maintain: remove job yaml redundant field
by Liu Yinsi 19 Oct '20

19 Oct '20

use submit -s 'key: value' in walk-os-iperf-test, instead of write in job yaml, remove os related parameters. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/iperf-walk-os.yaml | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/user-client/maintain/walk-os-test/iperf-walk-os.yaml b/user-client/maintain/walk-os-test/iperf-walk-os.yaml index 647220c..b70b5e7 100644 --- a/user-client/maintain/walk-os-test/iperf-walk-os.yaml +++ b/user-client/maintain/walk-os-test/iperf-walk-os.yaml @@ -7,9 +7,6 @@ if role server: if role client: iperf: protocol: tcp -testbox: vm-2p8g--USER -os: OS -os_arch: OS_ARCH -os_version: OS_VER +testbox: vm-2p8g node_roles: server client LKP_DEBUG_PREFIX: bash -x -- 2.23.0

2 1

[PATCH v2 compass-ci] fix: taskqueue: simplify logs
by Tong Qunfeng 19 Oct '20

19 Oct '20

[why] current taskqueue show 4 lines for 1 debug message - empty line for split: - end user request ip:port, request body: 172.17.0.1:48956 {"domain":"..."} - response body: <-- - kemal frame log: 2020-10... 200 POST /add?queue=... 710.15µs cut it 2 line, and duplicate information - kemal frame log has time in {2020-10...} and span {710.15us} examples: before <<< 2020-10-19 02:40:06 UTC --> 172.17.0.1:48956 {"domain":"compass-ci","lab":"z9"} 2020-10-19 02:40:06 UTC <-- {"id":"z9.134780"} 2020-10-19 02:40:06 UTC 200 POST /add?queue=sched%2Ftaishan200-2280-2s64p-256g--a9 710.15µs >>> after <<< from: {172.17.0.1:37444} <-- ack: {"domain":"compass-ci","lab":"crystal","id":"crystal.104640"} 2020-10-19 08:12:20 UTC 200 PUT /consume?queue=extract_stats 1.01ms >>> Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/taskqueue/taskqueue.cr | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/src/taskqueue/taskqueue.cr b/src/taskqueue/taskqueue.cr index c5c202d..98c5623 100644 --- a/src/taskqueue/taskqueue.cr +++ b/src/taskqueue/taskqueue.cr @@ -18,16 +18,22 @@ class TaskQueue @@rate_limiter.bucket(:l1pms, 1_u32, 0.001.seconds) @@rate_limiter.bucket(:l100pms, 1_u32, 0.00001.seconds) + # logs example: + # from: {172.17.0.1:6952} body {"domain":"compass-ci"} <-- ack: {"id":"z9.134780"} + # | from: {172.17.0.1:6952} <-- ack: {} + # 2020-10-19 03:25:01 UTC 201 PUT /consume?queue=sched%2Fdc-2g%2Fidle 356.51µs + # + # "from" is puts by this debug_message + # "2020-10..." auto puts by kemal frame, time in {2020-10-19 03:25:01 UTC} and span {356.51us} def debug_message(env, response, time_in) - puts("\n") + logs = "from: {#{env.request.remote_address}}" - from_message = "#{time_in} --> #{env.request.remote_address}" - if env.request.body != nil - from_message += " #{env.request.body}" - end - puts(from_message) + logs += " body: #{env.request.body}" if env.request.body + + logs += " <-- ack: " + logs += response ? "#{response}" : "{}" - puts("#{Time.utc} <-- #{response}") + puts(logs) end def run -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container/send-mail: mail-post.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add setup smpt for intranet/internet smtp enable send mail to both curystal.ci and WAN email Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/mail-post.rb | 47 ++++++++++++++++++++------------ 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/container/send-mail/mail-post.rb b/container/send-mail/mail-post.rb index 4eecf6e..ffbf31d 100755 --- a/container/send-mail/mail-post.rb +++ b/container/send-mail/mail-post.rb @@ -7,23 +7,25 @@ require 'sinatra' require 'json' require 'yaml' require 'open3' -require_relative 'send-mail.rb' +require_relative 'send-mail' set :bind, '0.0.0.0' -set :port, 11311 +set :port, ENV['SEND_MAIL_PORT'] + +mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp post '/send_mail_yaml' do data = YAML.safe_load request.body.read - raise TypeError, data, 'request data type error' unless data.class.eql? Hash + raise TypeError, data, 'request data type error' unless data.instance_of?(Hash) mail_info = { 'references' => data['references'] || '', - 'from' => data['from'] || 'team(a)crystal.ci', - 'subject' => data['subject'], - 'to' => data['to'], - 'body' => data['body'] + 'from' => data['from'] || (ENV['DEFAULT_EMAIL']).to_s, + 'subject' => data['subject'] || '', + 'to' => data['to'] || '', + 'body' => data['body'] || '' } - check_send_mail(mail_info) + check_send_mail(mail_info, mail_server) end post '/send_mail_text' do @@ -31,18 +33,29 @@ post '/send_mail_text' do mail_info = { 'references' => data.references || '', - 'from' => data.from || 'team(a)crystal.ci', - 'subject' => data.subject, - 'to' => data.to, - 'body' => data.body.decoded + 'from' => data.from || (ENV['DEFAULT_EMAIL']).to_s, + 'subject' => data.subject || '', + 'to' => data.to[0] || '', + 'body' => data.body.decoded || '' } - check_send_mail(mail_info) + check_send_mail(mail_info, mail_server) end -def check_send_mail(mail_info) - raise TypeError, data, 'empty subject.' if mail_info['subject'].empty? - raise TypeError, data, 'empty email address.' if mail_info['to'].empty? - raise TypeError, data, 'empty email content.' if mail_info['body'].empty? +def check_send_mail(mail_info, mail_server) + raise 'empty subject.' if mail_info['subject'].empty? + raise 'empty email address.' if mail_info['to'].empty? + raise 'empty email content.' if mail_info['body'].empty? + + check_smtp(mail_info, mail_server) send_mail(mail_info) end + +def check_smtp(mail_info, mail_server) + if [nil, 'crystal.ci'].include? mail_info['to'].split('@')[1] + intranet_smtp(mail_server) + else + internet_smtp + mail_info['from'] = (ENV['EMAIL_ADDR']).to_s + end +end -- 2.23.0

3 7

[PATCH compass-ci] service/scheduler: add ssh port to job content
by Wu Zhende 19 Oct '20

19 Oct '20

[Why] Network users need to use sshd to login testbox. So testbox should set sshd. And testbox get sshd info from job content. Scheduler needs to add ssh info to job content Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/start | 4 ++++ src/lib/job.cr | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 81e7139..1183cde 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -24,10 +24,14 @@ names = Set.new %w[ GIT_SERVER FLUENTD_SERVER_HOST FLUENTD_SERVER_PORT + SSH_PORT_BASE + SSH_PORT_LEN lab ] defaults = relevant_defaults(names) +defaults["SSH_PORT_BASE"] = 50000 unless defaults["SSH_PORT_BASE"] +defaults["SSH_PORT_LEN"]= 2000 unless defaults["SSH_PORT_LEN"] env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/lib/job.cr b/src/lib/job.cr index 20afa65..fcc1711 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -139,6 +139,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() + set_ssh_port() end private def append_init_field @@ -168,6 +169,11 @@ class Job end end + private def set_ssh_port + self["ssh_port_base"] = ENV["SSH_PORT_BASE"] + self["ssh_port_len"] = ENV["SSH_PORT_LEN"] + end + private def set_os_dir self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end -- 2.23.0

2 1

[PATCH v7 compass-ci 4/6] container/mail-robot: Dockerfile
by Luan Shengde 19 Oct '20

19 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..57307bf --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER shdluan(a)163.com + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport && \ + ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ + echo 'Asia/Shanghai' > /etc/timezone + +COPY mail-robot.rb /usr/local/bin/ + +COPY answerback-email.rb /usr/local/bin/ + +WORKDIR /usr/local/bin/ -- 2.23.0

2 1

[PATCH compass-ci] container/assign-account: answerback-email.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add parameters for answerback-email -e,--email email_addr: add email address -s,--ssh-pubkey pub_key: add ssh pub_key -f,--raw-email mail_file: add email_adress[,pub_key] via email file Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 162 ++++++++----------- 1 file changed, 70 insertions(+), 92 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index b5283ba..4a86a87 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -3,46 +3,18 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -=begin - -repo_list: - all repos url list from upstream-repos.git - -API: -call graph: -read_mail_content -send_account_request - email_addr - get email address - email_message_id - get email message_id - check_email_available - check email available - pub_key_value - get pub key - account - send apply account request and return account info - build_message - build email message - send_mail - call send_mail to send mail whit build message - -the returned data for account_info like: -{ - "account" => "guest", - "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", - "jumper_port" => "10000" -} -=end +# xx --email xxx --login --ssh-pubkey xxx --raw-email email-file +# samba mount +# ssh logshn (huawei, ) (install pubkey / send password) require 'json' require 'mail' require 'set' -require_relative '../defconfig.rb' +require 'optparse' +require_relative '../defconfig' names = Set.new %w[ - JUMPER_IP + JUMPER_HOST JUMPER_PORT CRYSTAL_INTRANET SEND_MAIL_PORT @@ -50,18 +22,64 @@ names = Set.new %w[ defaults = relevant_defaults(names) -JUMPER_IP = defaults['JUMPER_IP'] +JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] CRYSTAL_INTRANET = defaults['CRYSTAL_INTRANET'] SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] -def build_message(email, message_id, infos) +email_info = { + 'email_addr' => '', + 'pub_key' => '', +} + +def format_info(email_file) + mail_content = Mail.read(email_file) + + email_addr = mail_content.from[0] + pub_key = mail_content.part[1].body.decoded.split(/\r|\n/).join if mail_content.part[1].filename == 'id_rsa.pub' + + infos = { + 'email_addr' => email_addr, + 'pub_key' => pub_key, + } + + infos +end + +options = OptionParser.new do |opts| + opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key] [--raw-email email_file]\n" + opts.banner += ' -e or -f is required' + opts.banner += ' -s is optional when use -e' + + opts.separator '' + opts.separator 'options:' + + opts.on('-e email_addr', '--email email_addr', 'appoint email address') do |email_addr| + email_info['email_addr'] = email_addr + end + + opts.on('-s pub_key', '--ssh-pubkey pub_key', 'ssh pub_key to enable keyless login') do |pub_key| + email_info['pub_key'] = pub_key + end + + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + email_info = format_info(email_file) + end + + opts.on_tail('', '-h', '--help', 'show this message') do + puts opts + exit + end +end + +options.parse!(ARGV) + +def build_message(email, infos) message = <<~EMAIL_MESSAGE To: #{email} - Message-ID: #{message_id} Subject: jumper account is ready - Dear #{email} + Dear #{email.split('@')[0]} Thank you for joining us. You can use the following command to login the jumper server: @@ -79,67 +97,27 @@ def build_message(email, message_id, infos) return message end -def email_addr(mail_content) - msg = 'not an applying account email' - - raise msg unless mail_content.subject =~ /apply ssh account/i - - email = mail_content.from.join(',') - - return email -end - -# def check_email_available(mail_content, email) -# oos_list = File.read('/c/upstream-repos/repo_list').split(/\n/) -# url = mail_content.body.decoded.split(/\n/).find { |line| line =~ /https?:\/\// } -# base_url = url.split('/')[0,5].join('/') -# message = 'The url is not in upstream repo_list' -# -# raise message unless oos_list.include? base_url -# -# url_fdback = %x(curl #{url}) -# email_index = url_fdback.index email -# -# message = 'No commit info found from the url for the email' -# raise message unless email_index -# end - -def email_message_id(mail_content) - message_id = mail_content.message_id - return message_id -end - -def pub_key_value(mail_content) - pub_key = mail_content.body.decoded.split(/\n/).find { |line| line =~ /ssh-rsa/ } - return pub_key -end - def account_info(pub_key) - account_info_str = %x(curl -XGET '#{JUMPER_IP}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + account_info_str = if pub_key.empty? + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') + else + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + end account_info = JSON.parse account_info_str - return account_info + account_info end -def send_account(mail_content) - email = email_addr(mail_content) - message_id = email_message_id(mail_content) - # check_email_available(mail_content, email) - - pub_key = pub_key_value(mail_content) - acct_info = account_info(pub_key) +def send_account(email_info) + message = "No email address specified\n" + message += "use -e email_addr add a email address\n" + message += 'or use -f to add a mail file' + raise message if email_info['email_addr'].empty? + acct_info = account_info(email_info['pub_key']) - message = build_message(email, message_id, acct_info) + message = build_message(email_info['email_addr'], acct_info) %x(curl -XPOST '#{CRYSTAL_INTRANET}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -def read_mail_content(mail_file) - mail_content = Mail.read(mail_file) - - return mail_content -end - -mail_file = ARGV[0] -mail_content = read_mail_content(mail_file) -send_account(mail_content) +send_account(email_info) -- 2.23.0

2 2

[PATCH compass-ci 2/2] /src/lib/web_backend.rb: remove common es methods
by Zhang Yuhang 19 Oct '20

19 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 59 +++++++++--------------------------------- 1 file changed, 12 insertions(+), 47 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 48cfadb..721ad77 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -30,16 +30,9 @@ NOT_SHOW_FIELDS = %w[result_root].freeze ALL_FIELDS = FIELDS + NOT_SHOW_FIELDS NOT_NEED_EXIST_FIELDS = %w[error_ids upstream_repo].freeze PREFIX_SEARCH_FIELDS = ['tbox_group'].freeze -ES_CLIENT = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") COMPARE_RECORDS_NUMBER = 50 - -def es_query(query) - ES_CLIENT.search index: 'jobs*', body: query -end - -def es_count(query) - ES_CLIENT.count(index: 'jobs*', body: query)['count'] -end +JOBS = ESQuery.new(index: 'jobs*') +REPO = ESQuery.new(index: 'repo') # "vm-2p8g--212" remove "--212" # "vm-2p8g--zzz" remove "--zzz" @@ -47,7 +40,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end @@ -56,42 +49,15 @@ def filter_tbox_group(es_result) result.to_a end -def all_suite - query = { - aggs: { - all_suite: { - terms: { field: 'suite', size: 1000 } - } - }, - size: 0 - } - es_result = es_query(query)['aggregations']['all_suite']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } - - es_result -end - def all_tbox_group - query = { - aggs: { - all_tbox_group: { - terms: { field: 'tbox_group', size: 1000 } - } - }, - size: 0 - } - es_result = es_query(query)['aggregations']['all_tbox_group']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } - + es_result = JOBS.all_values('tbox_group') filter_tbox_group(es_result) end def compare_candidates_body body = { query_conditions: { - suite: all_suite, + suite: JOBS.all_values('suite'), OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, @@ -148,7 +114,7 @@ end def get_groups_matrices(conditions, dimension, must, size, from) must += build_mutli_field_subquery_body(conditions) count_query = { query: { bool: { must: must } } } - total = es_count(count_query) + total = JOBS.es_count(count_query) return {} if total < 1 query = { @@ -164,14 +130,14 @@ def get_groups_matrices(conditions, dimension, must, size, from) }] } - result = es_query(query) + result = JOBS.es_query(query) matrices = combine_group_query_data(result, dimension) while matrices.empty? from += size break if from > total query[:from] = from - result = es_query(query) + result = JOBS.es_query(query) matrices = combine_group_query_data(result, dimension) end matrices @@ -229,7 +195,7 @@ def es_search(must, size, from) must << { exists: { field: f } } end count_query = { query: { bool: { must: must } } } - total = es_count(count_query) + total = JOBS.es_count(count_query) unless size size = total from = 0 @@ -245,7 +211,7 @@ def es_search(must, size, from) } return {}, total if wrong_size?(size, from) - return es_query(query)['hits']['hits'], total + return JOBS.es_query(query)['hits']['hits'], total end def get_job(result) @@ -327,7 +293,6 @@ def get_repo_url(urls) end def get_repo(git_repo) - repo = nil if git_repo must = [{ regexp: { git_repo: ".*#{git_repo}.*" } }] repo = query_repos(must, from: 0, size: 1)[0] @@ -344,7 +309,7 @@ def query_repos(must, from: 0, size: 1) git_repo: { order: 'asc' } }] } - result = ES_CLIENT.search index: 'repo', body: query + result = REPO.es_query(query) repos = [] result['hits']['hits'].each do |r| r = r['_source'] @@ -361,7 +326,7 @@ def search_repos(git_repo, page_size, page_num) from = size * page_num must = git_repo ? [{ regexp: { git_repo: ".*#{git_repo}.*" } }] : [] count_query = { query: { bool: { must: must } } } - total = ES_CLIENT.count(index: 'repo', body: count_query)['count'] + total = REPO.es_count(count_query) return [], total if wrong_size?(size, from) return query_repos(must, from: from, size: size), total -- 2.23.0

2 2

[PATCH lkp-tests 2/2] fix(iso2rootfs): support skipping iso that not used for dailybuild
by Yu Chuan 19 Oct '20

19 Oct '20

[Why] Openeuler release two kinds of iso type now, we need to skip the released iso type that not used for dailybuild. [How] 1. Confirm whether the purpose is dailybuild by iso name. 2. Exit 0 while the released iso is not used for dailybuild. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 4782378424b5..2ce4d9659f7e 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -22,6 +22,12 @@ log_info() echo "[INFO] $*" } +exit_info() +{ + log_info "$@" + exit 0 +} + die() { echo "[ERROR] $*" >&2 @@ -56,6 +62,28 @@ check_yaml_vars() done } +check_iso_name() +{ + local skipped_iso_prefixes + case ${iso_os} in + "openeuler") + skipped_iso_prefixes=( + "openEuler-2.0-SP8" + ) + ;; + *) + return + ;; + esac + + local prefix + for prefix in "${skipped_iso_prefixes[@]}" + do + [[ ${ISO_NAME} != ${prefix}* ]] || + exit_info "${iso_os} haven't release new iso for openEuler, no need to generate rootfs" + done +} + get_daily_iso_checksum() { ISO_URL="$(curl "${iso_url_file}")" @@ -63,8 +91,10 @@ get_daily_iso_checksum() local pub_ip=$(echo "${iso_url_file}" | grep -oEw "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}") ISO_URL="$(curl "${iso_url_file}" |sed -r "s/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/${pub_ip}/g")" } + ISO_NAME=$(basename "$ISO_URL") ISO_CHECKSUM_URL="${ISO_URL}.sha256sum" + check_iso_name curl -o "${CHECKSUM_FILE_CACHE}.tmp" "${ISO_CHECKSUM_URL}" SHA256SUM_NET=$(awk '{print $1}' "${CHECKSUM_FILE_CACHE}.tmp") @@ -118,9 +148,8 @@ check_sha256sum_update() if [ "$SHA256SUM_CACHE" == "$SHA256SUM_NET" ] then - log_info "${iso_os} haven't release new iso, no need to generate rootfs" rm -f "${CHECKSUM_FILE_CACHE}.tmp" - exit 0 + exit_info "${iso_os} haven't release new iso, no need to generate rootfs" else log_info "${iso_os} release a new iso, start to generate rootfs ..." return -- 2.23.0

3 4

[PATCH lkp-tests 1/2] feat(iso2rootfs): modify the rootfs dir layout and rootfs dir name
by Yu Chuan 19 Oct '20

19 Oct '20

[Why] 1. Modify the dir structure of dailybuild rootfs dir. 2. Modify the dir name of dailybuild rootfs dir. [How] before: "{os}/{os_arch}/debug-versions/dailybuild/{iso_version}-%Y%m%d%H%M%S" after: "{os}/{os_arch}/{iso_version}-%Y-%m-%d-%H-%M-%S" Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 524ab6695f93..4782378424b5 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,6 @@ get_daily_iso_checksum() mount_rootfs() { - DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +102,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-newest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +254,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +309,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -366,7 +365,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ + ln -s "$(basename ${ROOTFS_DES_DIR})" \ "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 2

[PATCH lkp-tests] tests: add test-tools
by Li Ping 19 Oct '20

19 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- tests/test-tools | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 tests/test-tools diff --git a/tests/test-tools b/tests/test-tools new file mode 100755 index 00000000..cee882aa --- /dev/null +++ b/tests/test-tools @@ -0,0 +1,37 @@ +#!/bin/bash +# - testsuite +# - case + +cd $BENCHMARK_ROOT/test-tools/mugen || exit +: "${testsuite:=smoke-testing}" +: "${case:=oe_test_dd_001}" + +set_env() +{ + bash runoet.sh -c + source ~/.bash_profile +} + +test_one_case() +{ + + bash runoet.sh -f $testsuite -r $case -x +} + +test_all_cases() +{ + bash runoet.sh -a +} + +run() +{ + set_env + if [ "${testsuite}" == "all" ] + then + test_all_cases + else + test_one_case + fi +} + +run -- 2.23.0

4 3

[PATCH v7 compass-ci 6/6] container/mail-robot: mail-robot.md
by Luan Shengde 19 Oct '20

19 Oct '20

guidance file for mail-robot Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..d5eeaa9 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,45 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +monitor received mails + check if the mail is for applying uuid/account + +## steps overview + +1. monitor mailbox dir + - check for new email file + +2. check email + - check email subject + - return if unmatched subject + - call answerback-email + - check email available + - check base_url for the oss_url in upstream-repos list + if not, it will raise "url not in upstream-repos list" error + - check if the email has commit for the url + - if gitee, git clone the repo and check + if non-gitee, curl url to check + - if not, raise "no commit found" error + - check pub_key + - check if there is a pub_key in the email + if not, it will raise 'no pub_key' error + - apply ssh account + - apply account uuid + - write account info to es + - send uuid to user + - mv mail file to drafts + +3. continue the monitor + - after 1 and 2, continue to monitor the mailbox + +4. service log + - use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot + +5. usage: + - mail subject: apply account + - mail body: commit url + - mail attachment: id_rsa.pub -- 2.23.0

1 0

[PATCH v7 compass-ci 3/6] container/mail-robot: build file
by Luan Shengde 19 Oct '20

19 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0