- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] generate conserver configuration file
by Cui Yili 16 Oct '20

16 Oct '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/conserver/conserver-head.cf | 16 ++++++++++++ container/conserver/generate_conserver.rb | 32 +++++++++++++++++++++++ 2 files changed, 48 insertions(+) create mode 100644 container/conserver/conserver-head.cf create mode 100755 container/conserver/generate_conserver.rb diff --git a/container/conserver/conserver-head.cf b/container/conserver/conserver-head.cf new file mode 100644 index 0000000..6648f88 --- /dev/null +++ b/container/conserver/conserver-head.cf @@ -0,0 +1,16 @@ +# This is a fairly basic configuration file that interacts with one + +default * { + logfile /srv/cci/serial/logs/&; # '&' is replaced with console name + timestamp 1hab; # write timestamps + rw *; # allow all users + type exec; + master localhost; + portbase 2000; # port numbers start at 2001 and + portinc 100; # go up by 1 (port #1 == 2001, etc) +} + +access * { + trusted 127.0.0.1; +} + diff --git a/container/conserver/generate_conserver.rb b/container/conserver/generate_conserver.rb new file mode 100755 index 0000000..285c393 --- /dev/null +++ b/container/conserver/generate_conserver.rb @@ -0,0 +1,32 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +require 'fileutils' +require 'yaml' + +lab = ENV['lab'] +system 'git clone file:///$CCI_REPOS/lab-$lab.git' + +def generate_conserver(lab) + return unless File.exist?('conserver-head.cf') + + FileUtils.cp('conserver-head.cf', 'conserver.cf') + + host_dir = "lab-#{lab}/hosts/" + return unless Dir.exist?(host_dir) + + Dir.each_child(host_dir) do |host| + ipmi_ip = YAML.load_file("#{host_dir}#{host}")['ipmi_ip'] + next if ipmi_ip.nil? + + console = <<~HEREDOC + console #{host} { + exec /usr/local/bin/ipmi.sh #{ipmi_ip}; + } + HEREDOC + File.open('conserver.cf', 'a') { |f| f << console } + end +end + +generate_conserver(lab) -- 2.23.0

1 0

[PATCH compass-ci] generate conserver configuration file
by Cui Yili 16 Oct '20

16 Oct '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/conserver/conserver-head.cf | 16 ++++++++++++ container/conserver/generate_conserver.rb | 30 +++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 container/conserver/conserver-head.cf create mode 100755 container/conserver/generate_conserver.rb diff --git a/container/conserver/conserver-head.cf b/container/conserver/conserver-head.cf new file mode 100644 index 0000000..6648f88 --- /dev/null +++ b/container/conserver/conserver-head.cf @@ -0,0 +1,16 @@ +# This is a fairly basic configuration file that interacts with one + +default * { + logfile /srv/cci/serial/logs/&; # '&' is replaced with console name + timestamp 1hab; # write timestamps + rw *; # allow all users + type exec; + master localhost; + portbase 2000; # port numbers start at 2001 and + portinc 100; # go up by 1 (port #1 == 2001, etc) +} + +access * { + trusted 127.0.0.1; +} + diff --git a/container/conserver/generate_conserver.rb b/container/conserver/generate_conserver.rb new file mode 100755 index 0000000..1c120b1 --- /dev/null +++ b/container/conserver/generate_conserver.rb @@ -0,0 +1,30 @@ +#!/usr/bin/env ruby + +# frozen_string_literal: true + +require 'fileutils' +require 'yaml' + +lab = ENV['lab'] +system 'git clone file:///$CCI_REPOS/lab-$lab.git' + +FileUtils.cp('conserver-head.cf', 'conserver.cf') + +def generate_conserver(lab) + hostdir = "lab-#{lab}/hosts/" + return unless Dir.exist?(hostdir) + + puts hostdir + Dir.each_child(hostdir) do |host| + ipmi_ip = YAML.load_file("#{hostdir}#{host}")['ipmi_ip'] + next if ipmi_ip.nil? + + File.open('conserver.cf', 'a') do |f| + f << "console #{host} {\n" + f << " exec /usr/local/bin/ipmi.sh #{ipmi_ip}\n" + f << "}\n" + end + end +end + +generate_conserver(lab) -- 2.23.0

1 0

[PATCH v2 compass-ci] feat: build centos initramfs image with one command
by Wang Chenglong 16 Oct '20

16 Oct '20

Use the tool to create a new centos-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/packages-to-install. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/centos/README.md | 16 ++++++++ container/osimage/centos/create-image | 43 ++++++++++++++++++++ container/osimage/centos/lib | 41 +++++++++++++++++++ container/osimage/centos/packages-to-install | 9 ++++ container/osimage/centos/packages-to-remove | 11 +++++ container/osimage/centos/run | 27 ++++++++++++ 6 files changed, 147 insertions(+) create mode 100644 container/osimage/centos/README.md create mode 100755 container/osimage/centos/create-image create mode 100755 container/osimage/centos/lib create mode 100644 container/osimage/centos/packages-to-install create mode 100644 container/osimage/centos/packages-to-remove create mode 100755 container/osimage/centos/run diff --git a/container/osimage/centos/README.md b/container/osimage/centos/README.md new file mode 100644 index 0000000..b5c7b07 --- /dev/null +++ b/container/osimage/centos/README.md @@ -0,0 +1,16 @@ ++# Use the tool to create a new centos-${version} initramfs image. ++ ++Usage: ++ cd ${CCI_SRC}/rootfs/initramfs/centos/aarch64/${version}/ ++ ./build ++ ++Some configuration items: ++./bin/packages-to-install ++ If you want to pre-install the software, you can write the package names in ./bin/packages-to-install. ++ ++./bin/packages-to-remove ++ If you want remove some unnecessary files, you can write the names in ./bin/packages-to-remove. ++ ++$HOME/.config/compass-ci/rootfs-passwd ++ Set the password for the image into this file. + diff --git a/container/osimage/centos/create-image b/container/osimage/centos/create-image new file mode 100755 index 0000000..a3495b2 --- /dev/null +++ b/container/osimage/centos/create-image @@ -0,0 +1,43 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Install functions +install_packages() +{ + echo "install" + yum repolist + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +# Setup functions +setup_rootfs() +{ + echo "setup" + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + + +# Config password +setup_login() { + [ -n "$ROOT_NEW_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${image_name}" |grep -vf /tmp/packages-to-remove | cpio -o -Hnewc | gzip -9 > /tmp/${image_name} + chmod 644 /tmp/${image_name} +} + +install_packages +setup_rootfs +setup_login +pack_cgz diff --git a/container/osimage/centos/lib b/container/osimage/centos/lib new file mode 100755 index 0000000..53aae2a --- /dev/null +++ b/container/osimage/centos/lib @@ -0,0 +1,41 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${os_name}:${os_version}" +image_name="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs-passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker_image() +{ + docker pull $image + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[INFO] Pull docker image failed." + exit + fi +} + +# cp image package to host +cp_image() +{ + docker cp -a init_docker:/tmp/${image_name} $HOME/ + echo "result: $(ls $HOME/${image_name})" +} diff --git a/container/osimage/centos/packages-to-install b/container/osimage/centos/packages-to-install new file mode 100644 index 0000000..a09bc28 --- /dev/null +++ b/container/osimage/centos/packages-to-install @@ -0,0 +1,9 @@ +openssh-server +openssl +cifs-utils +hostname +curl +iproute +iputils +which +time diff --git a/container/osimage/centos/packages-to-remove b/container/osimage/centos/packages-to-remove new file mode 100644 index 0000000..07a4a23 --- /dev/null +++ b/container/osimage/centos/packages-to-remove @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/packages-to-remove +/tmp/packages-to-install diff --git a/container/osimage/centos/run b/container/osimage/centos/run new file mode 100755 index 0000000..699bbc6 --- /dev/null +++ b/container/osimage/centos/run @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/packages-to-remove:/tmp/packages-to-remove + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker_image +${cmd[@]} +cp_image +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

1 1

help star 2020-10-16
by Wu Fengguang 16 Oct '20

16 Oct '20

Yuchuan is our help star, thanks to kind and helpful Yuchuan! $ bin/help-star people/*/credit/2020-10-16.md 31.2 Wu Fengguang 29.2 Yu Chuan 20.8 Wei Jihui 18.8 Cao Xueliang 14.8 Wang Chenglong 12.2 Liu Shaofei 11.0 Shi Zhichao 10.3 Wang Yong 10.1 Lu Weitao 10.1 Xu Xijian 9.9 Bai Jing 9.7 Tong Qunfeng 9.5 Zhang Dewan 9.4 Liu Yinsi 8.3 Wu Zhende 8.0 Luan Shengde 7.5 Cui Yili 7.5 Lin Jiaxin 6.3 Xiao Shenwei 6.1 Zhang Yale 6.1 Du Kaitian 5.3 Zhang Yu 4.9 Zhang Yuhang 4.7 Li Yuanchao 4.0 Li Ping 3.9 Ren Wen 2.3 Zhou Shengrui 2.3 Sun Yukui 2.2 Liu Saofei 1.5 Wang yong 1.3 Hu Xuejiao 0.0 Ren Zhiwen 0.0 Sun Lijun 0.0 Liang Tao 0.0 Wang Shuai 0.0 Lu Kaiyi 0.0 Cui Xucui 0.0 Xie Zheng

1 0

[PATCH compass-ci 1/2] feat: build centos initramfs image with one command
by Wang Chenglong 16 Oct '20

16 Oct '20

Use the tool to create a new centos-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/packages-to-install. --- container/osimage/centos/README.md | 15 +++++++ container/osimage/centos/bin/create-image | 36 ++++++++++++++++ container/osimage/centos/bin/lib | 41 +++++++++++++++++++ .../osimage/centos/bin/packages-to-install | 2 + .../osimage/centos/bin/packages-to-remove | 8 ++++ container/osimage/centos/run | 26 ++++++++++++ 6 files changed, 128 insertions(+) create mode 100644 container/osimage/centos/README.md create mode 100755 container/osimage/centos/bin/create-image create mode 100755 container/osimage/centos/bin/lib create mode 100644 container/osimage/centos/bin/packages-to-install create mode 100644 container/osimage/centos/bin/packages-to-remove create mode 100755 container/osimage/centos/run diff --git a/container/osimage/centos/README.md b/container/osimage/centos/README.md new file mode 100644 index 0000000..000ca05 --- /dev/null +++ b/container/osimage/centos/README.md @@ -0,0 +1,15 @@ +# Use the tool to create a new centos-${version} initramfs image. + +Usage: + cd ${CCI_SRC}/rootfs/initramfs/centos/aarch64/${version}/ + ./build + +Some configuration items: +./bin/packages-to-install + If you want to pre-install the software, you can write the package names in ./bin/packages-to-install. + +./bin/packages-to-remove + If you want remove some unnecessary files, you can write the names in ./bin/packages-to-remove. + +$HOME/.config/compass-ci/rootfs.passwd + Set the password for the image into this file. diff --git a/container/osimage/centos/bin/create-image b/container/osimage/centos/bin/create-image new file mode 100755 index 0000000..39aba7a --- /dev/null +++ b/container/osimage/centos/bin/create-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Configure rootfs +make_rootfs() +{ + yum repolist + yum -y install --skip-broken $(</root/bin/packages-to-install) + yum clean all + rm -rf $(</root/bin/packages-to-remove) + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + +# Config password +pre_config_rootfs() { + [ -n "$ROOT_NEW_PASSWD" ] && { + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" # Change the password in shadow + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" # Configure ssh service + } +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + cd / + find ./ ! -path "./${image_name}" | cpio -o -Hnewc | gzip -9 > /${image_name} + chmod 644 /${image_name} +} + +make_rootfs +pre_config_rootfs +pack_cgz diff --git a/container/osimage/centos/bin/lib b/container/osimage/centos/bin/lib new file mode 100755 index 0000000..955717d --- /dev/null +++ b/container/osimage/centos/bin/lib @@ -0,0 +1,41 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${os_name}:${os_version}" +image_name="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs.passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs.passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker_image() +{ + docker pull $image + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[INFO] Run command in false path ." + exit + fi +} + +# cp image package to host +cp_image() +{ + docker cp -a init_docker:/${image_name} $HOME/ + echo "result: $(ls $HOME/${image_name})" +} diff --git a/container/osimage/centos/bin/packages-to-install b/container/osimage/centos/bin/packages-to-install new file mode 100644 index 0000000..5d31c40 --- /dev/null +++ b/container/osimage/centos/bin/packages-to-install @@ -0,0 +1,2 @@ +openssh-server +openssl diff --git a/container/osimage/centos/bin/packages-to-remove b/container/osimage/centos/bin/packages-to-remove new file mode 100644 index 0000000..ce079b9 --- /dev/null +++ b/container/osimage/centos/bin/packages-to-remove @@ -0,0 +1,8 @@ +/.dockerenv +/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo diff --git a/container/osimage/centos/run b/container/osimage/centos/run new file mode 100755 index 0000000..6710872 --- /dev/null +++ b/container/osimage/centos/run @@ -0,0 +1,26 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/bin/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file "$root_pwd_file" + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/bin/:/root/bin + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker_image +${cmd[@]} +cp_image +docker_rm init_docker &> /dev/null +echo "build finished" + -- 2.23.0

2 4

git count 2020-10-16
by Wu Fengguang 16 Oct '20

16 Oct '20

Yuchuan is code star of the week, thanks Yuchuan! wfg@crystal ~/todo% bin/git-count 17 13 Yu Chuan 10 Wu Fengguang 8 Cao Xueliang 8 Xiao Shenwei 7 Lu Weitao 7 Wei Jihui 6 Li Yuanchao 5 Hu Xuejiao 4 Liu Yinsi 4 Zhang Yu 4 Wang Chenglong 4 Luan Shengde 4 Ren Wen 3 Xu Xijian 3 Liu Shaofei 3 Lu Kaiyi 2 Bai Jing 2 Zhang Yuhang 2 Zhang Dewan 1 Wang Yong 1 Lin Jiaxin 1 Li Ping 1 Zhou Shengrui 0 chief 0 Du Kaitian 0 Ren Zhiwen 0 Sun Lijun 0 Sun Yukui 0 Shi Zhichao 0 Wu Zhende 0 Zhang Yale 0 Cui Yili 0 Liang Tao 0 Xie Zheng

1 0

review star 2020-10-16
by Wu Fengguang 16 Oct '20

16 Oct '20

Shenwei is our review star, thanks Shenwei! wfg@crystal ~/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 266 148 293 63 Wu Fengguang 43 14 35 3 Yu Chuan 55 12 35 15 Tong Qunfeng 49 17 31 19 Xiao Shenwei 31 19 29 8 Liu Yinsi 30 7 21 3 Lu Weitao 34 12 18 3 Ren Wen 16 10 18 11 Xu Xijian 26 9 13 3 Wei Jihui 26 8 9 4 Cao Xueliang 5 3 7 0 Du Kaitian 3 3 6 0 Sun Yukui 27 2 5 0 Hu Xuejiao 60 2 4 3 Luan Shengde 44 2 4 0 Wang Chenglong 14 4 4 0 Cui Yili 2 2 4 0 Xie Zheng 19 1 3 0 Liu Shaofei 28 3 3 2 Zhang Yu 15 2 3 1 Li Yuanchao 16 3 2 1 Bai Jing 2 1 2 1 Lin Jiaxin 36 2 2 0 Zhang Yale 5 2 2 0 Zhou Shengrui 2 2 2 0 Wu Zhende 8 5 1 0 Lu Kaiyi 4 0 0 0 Wang Yong 4 0 0 0 Shi Zhichao 14 1 0 0 Zhang Dewan 15 0 0 0 Li Ping 26 0 0 0 Zhang Yuhang

1 0

[PATCH compass-ci] sched: use user lkp src in set_pp_initrd
by Cao Xueliang 16 Oct '20

16 Oct '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index effba9b..dc9d042 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -6,6 +6,7 @@ require "yaml" require "any_merge" require "scheduler/constants.cr" +require "scheduler/jobfile_operate.cr" struct JSON::Any def []=(key : String, value : String) @@ -80,6 +81,7 @@ class Job access_key access_key_file lkp_initrd_user + user_lkp_src kernel_uri kernel_append_root kernel_params @@ -126,6 +128,7 @@ class Job private def set_defaults append_init_field() + set_user_lkp_src() set_os_dir() set_result_root() set_result_service() @@ -381,6 +384,15 @@ class Job self["kernel_params"] = " #{kernel_common_params()} #{kernel_append_root} #{kernel_console()}" end + private def set_user_lkp_src + lkp_arch_cgz = "#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz" + if File.exists?(lkp_arch_cgz) + self["user_lkp_src"] = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) + else + raise "The #{lkp_arch_cgz} does not exist." + end + end + private def set_pp_initrd initrd_deps_arr = Array(String).new initrd_pkg_arr = Array(String).new @@ -392,11 +404,11 @@ class Job deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?("#{user_lkp_src}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") + if File.exists?("#{user_lkp_src}/pkg/#{program}") initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) end end -- 2.23.0

2 2

[PATCH v2 compass-ci] sched: use user lkp src in set_pp_initrd
by Cao Xueliang 16 Oct '20

16 Oct '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index effba9b..20afa65 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -6,6 +6,7 @@ require "yaml" require "any_merge" require "scheduler/constants.cr" +require "scheduler/jobfile_operate.cr" struct JSON::Any def []=(key : String, value : String) @@ -80,6 +81,7 @@ class Job access_key access_key_file lkp_initrd_user + user_lkp_src kernel_uri kernel_append_root kernel_params @@ -126,6 +128,7 @@ class Job private def set_defaults append_init_field() + set_user_lkp_src() set_os_dir() set_result_root() set_result_service() @@ -381,6 +384,13 @@ class Job self["kernel_params"] = " #{kernel_common_params()} #{kernel_append_root} #{kernel_console()}" end + private def set_user_lkp_src + lkp_arch_cgz = "#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz" + raise "The #{lkp_arch_cgz} does not exist." unless File.exists?(lkp_arch_cgz) + + self["user_lkp_src"] = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) + end + private def set_pp_initrd initrd_deps_arr = Array(String).new initrd_pkg_arr = Array(String).new @@ -392,11 +402,11 @@ class Job deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?("#{user_lkp_src}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") + if File.exists?("#{user_lkp_src}/pkg/#{program}") initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) end end -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-account.sh update es-mapping
by Lu Weitao 16 Oct '20

16 Oct '20

modify some fields to make them more specific Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-account.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sbin/es-account.sh b/sbin/es-account.sh index 2b739d5..635f4fc 100755 --- a/sbin/es-account.sh +++ b/sbin/es-account.sh @@ -18,16 +18,16 @@ else "my_commit_url": { "type": "keyword" }, - "uuid": { + "my_uuid": { "type": "keyword" }, - "email": { + "my_email": { "type": "keyword" }, - "name": { + "my_name": { "type": "keyword" }, - "user": { + "my_login_name": { "type": "keyword" } } -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/send-mail: update Dockerfile
by Luan Shengde 16 Oct '20

16 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/Dockerfile | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index 36b145c..0cf3eeb 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -11,9 +11,7 @@ RUN apt-get update && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone -EXPOSE 25 - COPY mail-post.rb /usr/local/bin/ COPY send-mail.rb /usr/local/bin/ -CMD ["/usr/local/bin/mail-post.rb"] +WORKDIR /usr/local/bin/ -- 2.23.0

1 0

[PATCH compass-ci 2/2] web-backend: add search feature to "/get_repos" api
by Zhang Yuhang 16 Oct '20

16 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 6 ++- src/lib/web_backend.rb | 82 ++++++++++++++++--------------- 2 files changed, 48 insertions(+), 40 deletions(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 60c7827..fcc765f 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -9,6 +9,7 @@ require "#{CCI_SRC}/src/lib/web_backend.rb" set :bind, '0.0.0.0' set :port, 32767 set :show_exceptions, false +set :environment, :production # GET /compare_candidates # return: {"query_conditions": {...}, "dimension": [...]} @@ -48,7 +49,10 @@ get '/get_jobs' do get_jobs(params) end -# GET /get_repos?page_size=12&page_num=3 +# GET /get_repos?git_repo=a&page_size=12&page_num=3 +# git_repo: +# > optional +# # page_size: # > optional # - default: 20 diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 31aa0c3..48cfadb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -95,7 +95,7 @@ def compare_candidates_body OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, - { os: 'debian', os_version: ['10', 'sid'] }, + { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], os_arch: %w[aarch64 x86_64], @@ -304,7 +304,7 @@ def get_jobs_body(params) banner: get_banner(git_repo, branches), jobs: jobs, fields: FIELDS - } + }.to_json end def get_jobs(params) @@ -314,66 +314,70 @@ def get_jobs(params) warn e.message return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get jobs error'] end - [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end -def get_repo_url(repo_file) - return unless File.file? repo_file +def get_repo_url(urls) + return unless urls.is_a?(Array) - urls = YAML.load_file(repo_file)['url'] urls.each do |url| return url if url[0, 4] == 'http' end urls[0] end -def get_repo(git_repo, repo_file = nil) - repo = { - git_repo: git_repo, - git_url: nil - } - return repo if !git_repo && !repo_file - - if !git_repo - git_repo = repo_file[UPSTREAM_REPOS_PATH.size + 1, repo_file.size - 1] - repo[:git_repo] = git_repo - elsif !repo_file - repo_file = File.join(UPSTREAM_REPOS_PATH, git_repo) +def get_repo(git_repo) + repo = nil + if git_repo + must = [{ regexp: { git_repo: ".*#{git_repo}.*" } }] + repo = query_repos(must, from: 0, size: 1)[0] end - - repo[:git_url] = get_repo_url(repo_file) - repo + repo || {} end -def repo_files_list - Dir["#{UPSTREAM_REPOS_PATH}/*/*/*"].sort +def query_repos(must, from: 0, size: 1) + query = { + query: { bool: { must: must } }, + size: size, + from: from, + sort: [{ + git_repo: { order: 'asc' } + }] + } + result = ES_CLIENT.search index: 'repo', body: query + repos = [] + result['hits']['hits'].each do |r| + r = r['_source'] + repos << { + git_url: get_repo_url(r['url']), + git_repo: r['git_repo'] + } + end + repos end -def get_repos_list(repo_files, from, finish, total) - repos_list = [] - - total.times do |index| - next if index < from - break if index >= finish +def search_repos(git_repo, page_size, page_num) + size = page_size + from = size * page_num + must = git_repo ? [{ regexp: { git_repo: ".*#{git_repo}.*" } }] : [] + count_query = { query: { bool: { must: must } } } + total = ES_CLIENT.count(index: 'repo', body: count_query)['count'] + return [], total if wrong_size?(size, from) - repos_list << get_repo(nil, repo_files[index]) - end - repos_list + return query_repos(must, from: from, size: size), total end def get_repos_body(params) - repo_files = repo_files_list page_size = get_positive_number(params[:page_size], 20) page_num = get_positive_number(params[:page_num], 1) - 1 + git_repo = params[:git_repo] - from = page_num * page_size - finish = from + page_size - total = repo_files.size + repos, total = search_repos(git_repo, page_size, page_num) { total: total, - repos: get_repos_list(repo_files, from, finish, total) - } + repos: repos + }.to_json end def get_repos(params) @@ -383,5 +387,5 @@ def get_repos(params) warn e.message return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] end - [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end -- 2.23.0

2 2

[PATCH v3 compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7785c2a..984b9a1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + unless hash_dup[key] == @hash[key] + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,7 +389,7 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) -- 2.23.0

1 0

[PATCH v2 compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7785c2a..a0cae68 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + unless hash_dup[key] == @hash[key] + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,9 +389,9 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}")) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end -- 2.23.0

1 0

[PATCH compass-ci] job.cr: def var osimage_dir for used multi times
by Xu Xijian 16 Oct '20

16 Oct '20

osimage_dir is used twice here, define a var fo it. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..fd6e79b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -299,10 +299,11 @@ class Job private def initramfs_initrds temp_initrds = [] of String + osimage_dir = "#{SRV_INITRD}/osimage/#{os_dir}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + "#{JobHelper.service_path("#{osimage_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

1 0

[PATCH compass-ci] job.cr: def var initrd_os_dir for used multi times
by Xu Xijian 16 Oct '20

16 Oct '20

initrd_os_dir is used twice here, define a var fo it. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..fc712ba 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -299,10 +299,11 @@ class Job private def initramfs_initrds temp_initrds = [] of String + initrd_os_dir = "#{SRV_INITRD}/osimage/#{os_dir}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + "#{JobHelper.service_path("#{initrd_os_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + "#{JobHelper.service_path("#{initrd_os_dir}/run-ipconfig.cgz")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

2 2

[PATCH lkp-tests] fix(iso2rootfs): modify the soft link name
by Yu Chuan 16 Oct '20

16 Oct '20

'dailybuild' is closer to logic than 'debug'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 7ab4a08b0d60..524ab6695f93 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,7 @@ get_daily_iso_checksum() mount_rootfs() { - DEBUG_ROOTFS_DIR="debug-versions/dailybuild" + DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +103,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DEBUG_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +255,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DEBUG_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +310,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DEBUG_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DEBUG_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -363,10 +363,11 @@ post_works() mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" + local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && - rm -f "${iso_version}-debug" && - ln -s "${DEBUG_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ - "${iso_version}-debug" + rm -f "${soft_link}" && + ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ + "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 2

[PATCH compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..ffe3749 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + if "#{hash_dup[key]}" != "#{@hash[key]}" + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,7 +389,7 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") && File.exists?(deps_dest_file) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end -- 2.23.0

2 2

[PATCH v8 lkp-tests] sshd: send emails to login services for users
by Zhang Yale 16 Oct '20

16 Oct '20

Provide a login port for user by email, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 89 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 20 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..1f7c4126 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,16 @@ #!/bin/sh # pub_key # email +# uuid +# ssh_port_len +# ssh_port_base + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 +[ -n "$sshr_port_len" ] || sshr_port_len=2000 +[ -n "$sshr_port_base" ] || sshr_port_base=30000 run_ssh() { @@ -11,40 +21,79 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh +Subject: [NOTIFY Compass-ci] $testbox ready to use + +Dear $user_name: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: + login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch - Login: - ssh root@$PUB_IP - Due time: - $deadline +Regards +Compass-ci +" +} - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch +data_failure() +{ + data="To: $email +Subject: [NOTIFY Compass-ci] Applying $testbox failed + +Dear $user_name: + Sorry to inform you that your application failed. + You may need to wait for a while and then try again. Regards -compass-ci +Compass-ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + +get_port() +{ + [ -n "$uuid" ] || return + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ + -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] || continue + return + done + port="" +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$email" ] && [ -n "$uuid" ] || return 0 + user_name=$(echo "$email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +get_port +report_ssh_port $port run_email -- 2.23.0

2 2

[PATCH compass-ci 4/4] container/send-mail: update start
by Luan Shengde 16 Oct '20

16 Oct '20

add setup default variables update heredoc Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/start | 74 +++++++++++++++++++++++++-------------- 1 file changed, 47 insertions(+), 27 deletions(-) diff --git a/container/send-mail/start b/container/send-mail/start index 80be86a..e287322 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -1,29 +1,53 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -. $CCI_SRC/container/defconfig.sh +require 'set' +require_relative '../defconfig' -docker_rm send-mail +docker_rm 'send-mail' -cmd=( - docker run - --restart=always - --name=send-mail - -u nobody - -d - -p 11311:11311 - -v /etc/localtime:/etc/localtime:ro - send-mail -) +names = Set.new %w[ + EMAIL_PSWD + SEND_MAIL_PORT +] -"${cmd[@]}" +defaults = relevant_defaults(names) -cat<<EOF +EMAIL_PSWD = defaults['EMAIL_PSWD'] +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] + +cmd = %W[ + docker run + --restart=always + --name=send-mail + -u nobody + -d + -e EMAIL_PSWD=#{EMAIL_PSWD} + -e SEND_MAIL_PORT=#{SEND_MAIL_PORT} + -p #{SEND_MAIL_PORT}:#{SEND_MAIL_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-driver json-file + send-mail +] + +cmd += ['bash', '-c', '/usr/local/bin/mail-post.rb'] + +system(*cmd) + +puts <<HEREDOC port: 11311 + +default from email: + when sending crystal.ci email. + team(a)crystal.ci if no from email appiontted + when sending to WAN email + compass-ci(a)qq.com in all cases + --- data='{ -"from": "email_from_addr" # default: team(a)crystal.ci +"from": "email_from_addr" "subject": "email subject", "to": "email_to_addr", "body": "email message" @@ -32,19 +56,17 @@ data='{ or data=" -from: email_from_addr # default: team(a)crystal.ci +from: email_from_addr subject: email subject to: email_to_addr body: email message" -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_yaml' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_yaml' -d "$data" +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_yaml' -d "$data" --- data=" -From: email_from_addr # default: team(a)crystal.ci +From: email_from_addr To: email_to_addr Subject: email_subject @@ -53,8 +75,6 @@ mail_msg_line2 ... " -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_text' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_text' -d "$data" -EOF +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_text' -d "$data" +HEREDOC -- 2.23.0

1 0

[PATCH compass-ci 3/4] container/send-mail: update Dockerfile
by Luan Shengde 16 Oct '20

16 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/Dockerfile | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index 36b145c..0cf3eeb 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -11,9 +11,7 @@ RUN apt-get update && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone -EXPOSE 25 - COPY mail-post.rb /usr/local/bin/ COPY send-mail.rb /usr/local/bin/ -CMD ["/usr/local/bin/mail-post.rb"] +WORKDIR /usr/local/bin/ -- 2.23.0

1 0

[PATCH compass-ci 2/4] container/send-mail: send-mail.rb
by Luan Shengde 16 Oct '20

16 Oct '20

add setup smtp for qq.com enable send mail via smtp.qq.com Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/send-mail.rb | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index f459e78..4fb2b7e 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -5,15 +5,29 @@ require 'mail' -mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp +# mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp # setup smtp config -smtp = { - address: mail_server, - enable_starttls_auto: false -} +def intranet_smtp(mail_server) + smtp = { + address: mail_server, + enable_starttls_auto: false + } + Mail.defaults { delivery_method :smtp, smtp } +end -Mail.defaults { delivery_method :smtp, smtp } +def internet_smtp + smtp = { + address: 'smtp.qq.com', + port: 25, + domain: 'qq.com', + user_name: 'compass-ci(a)qq.com', + password: (ENV['EMAIL_PSWD']).to_s, + openssl_verify_mode: 'none', + enable_starttls_auto: true + } + Mail.defaults { delivery_method :smtp, smtp } +end # send mail def send_mail(mail_info) -- 2.23.0

1 0

[PATCH compass-ci 1/4] container/send-mail:mail-post.rb
by Luan Shengde 16 Oct '20

16 Oct '20

add setup smpt for intranet/internet smtp enable send mail to both curystal.ci and WAN email Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/mail-post.rb | 36 +++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/container/send-mail/mail-post.rb b/container/send-mail/mail-post.rb index 4eecf6e..c7c31d5 100755 --- a/container/send-mail/mail-post.rb +++ b/container/send-mail/mail-post.rb @@ -7,21 +7,21 @@ require 'sinatra' require 'json' require 'yaml' require 'open3' -require_relative 'send-mail.rb' +require_relative 'send-mail' set :bind, '0.0.0.0' -set :port, 11311 +set :port, ENV['SEND_MAIL_PORT'] post '/send_mail_yaml' do data = YAML.safe_load request.body.read - raise TypeError, data, 'request data type error' unless data.class.eql? Hash + raise TypeError, data, 'request data type error' unless data.instance_of?(Hash) mail_info = { 'references' => data['references'] || '', 'from' => data['from'] || 'team(a)crystal.ci', - 'subject' => data['subject'], - 'to' => data['to'], - 'body' => data['body'] + 'subject' => data['subject'] || '', + 'to' => data['to'] || '', + 'body' => data['body'] || '' } check_send_mail(mail_info) end @@ -32,17 +32,29 @@ post '/send_mail_text' do mail_info = { 'references' => data.references || '', 'from' => data.from || 'team(a)crystal.ci', - 'subject' => data.subject, - 'to' => data.to, - 'body' => data.body.decoded + 'subject' => data.subject || '', + 'to' => data.to[0] || '', + 'body' => data.body.decoded || '' } check_send_mail(mail_info) end def check_send_mail(mail_info) - raise TypeError, data, 'empty subject.' if mail_info['subject'].empty? - raise TypeError, data, 'empty email address.' if mail_info['to'].empty? - raise TypeError, data, 'empty email content.' if mail_info['body'].empty? + raise 'empty subject.' if mail_info['subject'].empty? + raise 'empty email address.' if mail_info['to'].empty? + raise 'empty email content.' if mail_info['body'].empty? + + check_smtp(mail_info) send_mail(mail_info) end + +def check_smtp(mail_info) + mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp + if [nil, 'crystal.ci'].include? mail_info['to'].split('@')[1] + intranet_smtp(mail_server) + else + internet_smtp + mail_info['from'] = 'compass-ci(a)qq.com' + end +end -- 2.23.0

1 0

[PATCH lkp-tests] tests: add get_ipmi_ip function to host-info
by Zhang Yu 16 Oct '20

16 Oct '20

If ipmitool returns error or empty, the return of "$(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" is null, and only add the field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/host-info b/tests/host-info index a57855dc..6ee8df21 100755 --- a/tests/host-info +++ b/tests/host-info @@ -80,6 +80,11 @@ get_partitions() fi } +get_ipmi_ip() +{ + echo "ipmi_ip: $(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" +} + main() { get_node @@ -87,6 +92,7 @@ main() get_partitions get_mac get_model_name + get_ipmi_ip } main -- 2.23.0

4 3

[PATCH compass-ci] feat(sparrow): add kernel param: net.ipv6.bindv6only=1
by Yu Chuan 16 Oct '20

16 Oct '20

Turn on the bindv6only function to avoid the use of tcp6 port for services that do not actually need tcp6. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2c7de3754fad..31588f40b602 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -10,6 +10,7 @@ grep -q '^committer:' /etc/group || groupadd -g 1999 committer cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 +net.ipv6.bindv6only=1 vm.max_map_count=262144 EOF -- 2.23.0

1 0