October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests 2/2] jobs: add test-tools.yaml
by Li Ping 20 Oct '20

20 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- jobs/test-tools.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 jobs/test-tools.yaml diff --git a/jobs/test-tools.yaml b/jobs/test-tools.yaml new file mode 100644 index 00000000..1f711dcf --- /dev/null +++ b/jobs/test-tools.yaml @@ -0,0 +1,6 @@ +suite: test-tools +category: functional + +test-tools: + testsuite: + case: -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] delete the field: testcase
by Li Ping 20 Oct '20

20 Oct '20

[why] there is no need to add the field: testcase in the $job.yaml Signed-off-by: Li Ping <15396232681(a)163.com> --- bin/lkp-setup-rootfs | 2 +- lib/bootstrap.sh | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 09f7e7eb..05a3325c 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -51,7 +51,7 @@ job_does_not_need_reboot() } grep -q "^do_not_reboot_for_same_kernel: 1$" $job && - is_same_kernel_and_rootfs && is_same_testcase && is_same_bp_memmap && { + is_same_kernel_and_rootfs && is_same_suite && is_same_bp_memmap && { echo "LKP: [do_not_reboot_for_same_kernel] start a new job without rebooting" return 0 } diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 2506c021..a7776d68 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -631,12 +631,12 @@ is_same_kernel_and_rootfs() return 1 } -is_same_testcase() +is_same_suite() { - local current_testcase=$testcase - local next_testcase=$(awk '/^testcase: /{print $2}' $job | tr -d '"') + local current_suite=$suite + local next_suite=$(awk '/^suite: /{print $2}' $job | tr -d '"') - [ "$current_testcase" = "$next_testcase" ] + [ "$current_suite" = "$next_suite" ] } is_same_bp_memmap() -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: submit with yaml file without directory
by Wei Jihui 20 Oct '20

20 Oct '20

if get string named with *.yaml from ARGV, treat it as job yaml. even it is without directory. and find_jobfile check whether the job yaml exists. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 831bfb5a..979e3dca 100755 --- a/sbin/submit +++ b/sbin/submit @@ -60,7 +60,7 @@ options.parse!(ARGV) find_job_yaml = false ARGV.delete_if do |arg| - find_job_yaml = true if File.file?(arg) + find_job_yaml = true if arg =~ /\.yaml$/ if arg.index '=' if find_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') -- 2.23.0

3 2

[PATCH v2 compass-ci] add exception handing for docker.rb
by Xiao Shenwei 20 Oct '20

20 Oct '20

[why] when run multi-docker job on physical machine, run 50 client to request job, if server down or error like Connection refused. these client willexit. when the server recovers, need to run these client again. [how] add exception handing to ensure all client does not exit Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index c241d3c..3600025 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -88,6 +88,19 @@ def main(hostname) run(hostname, load_path, hash) end +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.backtrace + sleep 25 + ensure + sleep 5 + end + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -99,10 +112,7 @@ def multi_docker(hostname, nr_container) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop do - main("#{hostname}-#{i}") - sleep 5 - end + loop_main("#{hostname}-#{i}") end pids << pid end -- 2.23.0

2 2

[PATCH v3 compass-ci] feat: build centos initramfs image with one command
by Wang Chenglong 20 Oct '20

20 Oct '20

Use the tool to create a new centos-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/packages-to-install. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/centos/README.md | 16 ++++++++ container/osimage/centos/create-image | 43 ++++++++++++++++++++ container/osimage/centos/lib | 41 +++++++++++++++++++ container/osimage/centos/packages-to-install | 9 ++++ container/osimage/centos/packages-to-remove | 11 +++++ container/osimage/centos/run | 27 ++++++++++++ 6 files changed, 147 insertions(+) create mode 100644 container/osimage/centos/README.md create mode 100755 container/osimage/centos/create-image create mode 100755 container/osimage/centos/lib create mode 100644 container/osimage/centos/packages-to-install create mode 100644 container/osimage/centos/packages-to-remove create mode 100755 container/osimage/centos/run diff --git a/container/osimage/centos/README.md b/container/osimage/centos/README.md new file mode 100644 index 0000000..b5c7b07 --- /dev/null +++ b/container/osimage/centos/README.md @@ -0,0 +1,16 @@ ++# Use the tool to create a new centos-${version} initramfs image. ++ ++Usage: ++ cd ${CCI_SRC}/rootfs/initramfs/centos/aarch64/${version}/ ++ ./build ++ ++Some configuration items: ++./bin/packages-to-install ++ If you want to pre-install the software, you can write the package names in ./bin/packages-to-install. ++ ++./bin/packages-to-remove ++ If you want remove some unnecessary files, you can write the names in ./bin/packages-to-remove. ++ ++$HOME/.config/compass-ci/rootfs-passwd ++ Set the password for the image into this file. + diff --git a/container/osimage/centos/create-image b/container/osimage/centos/create-image new file mode 100755 index 0000000..a3495b2 --- /dev/null +++ b/container/osimage/centos/create-image @@ -0,0 +1,43 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Install functions +install_packages() +{ + echo "install" + yum repolist + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +# Setup functions +setup_rootfs() +{ + echo "setup" + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + + +# Config password +setup_login() { + [ -n "$ROOT_NEW_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${image_name}" |grep -vf /tmp/packages-to-remove | cpio -o -Hnewc | gzip -9 > /tmp/${image_name} + chmod 644 /tmp/${image_name} +} + +install_packages +setup_rootfs +setup_login +pack_cgz diff --git a/container/osimage/centos/lib b/container/osimage/centos/lib new file mode 100755 index 0000000..53aae2a --- /dev/null +++ b/container/osimage/centos/lib @@ -0,0 +1,41 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${os_name}:${os_version}" +image_name="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs-passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker_image() +{ + docker pull $image + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[INFO] Pull docker image failed." + exit + fi +} + +# cp image package to host +cp_image() +{ + docker cp -a init_docker:/tmp/${image_name} $HOME/ + echo "result: $(ls $HOME/${image_name})" +} diff --git a/container/osimage/centos/packages-to-install b/container/osimage/centos/packages-to-install new file mode 100644 index 0000000..a09bc28 --- /dev/null +++ b/container/osimage/centos/packages-to-install @@ -0,0 +1,9 @@ +openssh-server +openssl +cifs-utils +hostname +curl +iproute +iputils +which +time diff --git a/container/osimage/centos/packages-to-remove b/container/osimage/centos/packages-to-remove new file mode 100644 index 0000000..07a4a23 --- /dev/null +++ b/container/osimage/centos/packages-to-remove @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/packages-to-remove +/tmp/packages-to-install diff --git a/container/osimage/centos/run b/container/osimage/centos/run new file mode 100755 index 0000000..699bbc6 --- /dev/null +++ b/container/osimage/centos/run @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/packages-to-remove:/tmp/packages-to-remove + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker_image +${cmd[@]} +cp_image +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

2 2

[PATCH v4 compass-ci] feat: build centos initramfs image with one command
by Wang Chenglong 20 Oct '20

20 Oct '20

Use the tool to create a new centos-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/packages-to-install. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/centos/README.md | 16 ++++++++ container/osimage/centos/create-image | 43 ++++++++++++++++++++ container/osimage/centos/lib | 41 +++++++++++++++++++ container/osimage/centos/packages-to-install | 9 ++++ container/osimage/centos/packages-to-remove | 11 +++++ container/osimage/centos/run | 27 ++++++++++++ 6 files changed, 147 insertions(+) create mode 100644 container/osimage/centos/README.md create mode 100755 container/osimage/centos/create-image create mode 100755 container/osimage/centos/lib create mode 100644 container/osimage/centos/packages-to-install create mode 100644 container/osimage/centos/packages-to-remove create mode 100755 container/osimage/centos/run diff --git a/container/osimage/centos/README.md b/container/osimage/centos/README.md new file mode 100644 index 0000000..b5c7b07 --- /dev/null +++ b/container/osimage/centos/README.md @@ -0,0 +1,16 @@ ++# Use the tool to create a new centos-${version} initramfs image. ++ ++Usage: ++ cd ${CCI_SRC}/rootfs/initramfs/centos/aarch64/${version}/ ++ ./build ++ ++Some configuration items: ++./bin/packages-to-install ++ If you want to pre-install the software, you can write the package names in ./bin/packages-to-install. ++ ++./bin/packages-to-remove ++ If you want remove some unnecessary files, you can write the names in ./bin/packages-to-remove. ++ ++$HOME/.config/compass-ci/rootfs-passwd ++ Set the password for the image into this file. + diff --git a/container/osimage/centos/create-image b/container/osimage/centos/create-image new file mode 100755 index 0000000..fac8ddc --- /dev/null +++ b/container/osimage/centos/create-image @@ -0,0 +1,43 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Install functions +install_packages() +{ + echo "install" + yum repolist + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +# Setup functions +setup_rootfs() +{ + echo "setup" + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + + +# Config password +setup_login() { + [ -n "$ROOT_NEW_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${image_name}" | grep -vf /tmp/packages-to-remove | cpio -o -Hnewc | gzip -9 > /tmp/${image_name} + chmod 644 /tmp/${image_name} +} + +install_packages +setup_rootfs +setup_login +pack_cgz diff --git a/container/osimage/centos/lib b/container/osimage/centos/lib new file mode 100755 index 0000000..53aae2a --- /dev/null +++ b/container/osimage/centos/lib @@ -0,0 +1,41 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${os_name}:${os_version}" +image_name="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs-passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker_image() +{ + docker pull $image + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[INFO] Pull docker image failed." + exit + fi +} + +# cp image package to host +cp_image() +{ + docker cp -a init_docker:/tmp/${image_name} $HOME/ + echo "result: $(ls $HOME/${image_name})" +} diff --git a/container/osimage/centos/packages-to-install b/container/osimage/centos/packages-to-install new file mode 100644 index 0000000..a09bc28 --- /dev/null +++ b/container/osimage/centos/packages-to-install @@ -0,0 +1,9 @@ +openssh-server +openssl +cifs-utils +hostname +curl +iproute +iputils +which +time diff --git a/container/osimage/centos/packages-to-remove b/container/osimage/centos/packages-to-remove new file mode 100644 index 0000000..07a4a23 --- /dev/null +++ b/container/osimage/centos/packages-to-remove @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/packages-to-remove +/tmp/packages-to-install diff --git a/container/osimage/centos/run b/container/osimage/centos/run new file mode 100755 index 0000000..699bbc6 --- /dev/null +++ b/container/osimage/centos/run @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/packages-to-remove:/tmp/packages-to-remove + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker_image +${cmd[@]} +cp_image +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

3 6

[PATCH v3 compass-ci] src/lib: find depends_initrd for multitask
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the code need multi-qemu, not multi-qemu-x, and scheduler can not find the depends_initrd multi-qemu-x.cgz. Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..93bc712 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,9 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + if program =~ /(.*)-\d+$/ + program = $1 + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +415,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

1 0

[PATCH compass-ci] src/lib: find depends_initrd for multitask
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the code need multi-qemu, not multi-qemu-x, and scheduler can not find the depends_initrd multi-qemu-x.cgz. Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..a0d7263 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,9 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + if program =~ /(.*)(\-\d{1,}$)/ + program = $1 + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +415,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq!.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq!.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

1 0

[PATCH compass-ci] code-spec: add some rules
by Wu Fengguang 20 Oct '20

20 Oct '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/code-spec.md b/doc/code-spec.md index a859239..d405cdd 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -4,8 +4,11 @@ code注意事项 - 代码应当放进函数里不要在脚本顶层直接写逻辑 - 代码应当放进lib/里脚本本身最小化, 仅解析命令行参数, 调用lib/入口函数 - 如果出现了全局变量一般意味着需要写一个 class/module 来放这个状态量了 -- 一个函数, 不要超过10行代码 +- 一个函数, 不要超过10行代码, 该原则适用于所有ruby/crystal/shell各类程序. - 起准确和有意义的变量/函数名字, 名字要让人看出赋予他们的确切意义 +- 函数要有分层结构. 每个函数须有明确的定位: 业务层逻辑, 还是底层支持性routine. + 一个函数不能同时做两个层次的事. 同一函数内避免混杂不同类型的事务. +- 如果注释仅仅是代码的简单重复, 不提供额外信息量, 不要写. 重构友好编码 ============ -- 2.23.0

1 0

[PATCH v2 compass-ci] src/lib: for multitask like multi-qemu
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the code need multi-qemu, not multi-qemu-x. Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..a48f233 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,9 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + if program =~ /(.*)(\-\d{1,}$)/ + program_num = $1 + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +415,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq!.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq!.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

2 2