October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] fix: scheduler: update_id need set_initrds_uri
by Tong Qunfeng 12 Oct '20

12 Oct '20

Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index bf959c2..504bc19 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -382,9 +382,13 @@ class Job def update_id(id) @hash["id"] = JSON::Any.new(id) - # "result_root" is based on "id" + # "result_root" => "/result/#{suite}/#{tbox_group}/#{date}/#{id}" + # set_initrds_uri -> get_initrds -> common_initrds => ".../#{id}/job.cgz" + # + # "result_root, common_initrds" is associate with "id" # so when update id, we need redo set_ set_result_root() + set_initrds_uri() end def get_uuid_tag -- 2.23.0

1 0

[PATCH v4 compass-ci] feat: build centos-7 initramfs image with one command
by Wang Chenglong 12 Oct '20

12 Oct '20

Use the tool to create a new centos-7 initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/install-list. --- container/osimage/centos-7/README.md | 13 +++++++ container/osimage/centos-7/bin/create-image | 38 +++++++++++++++++++++ container/osimage/centos-7/bin/delete-list | 8 +++++ container/osimage/centos-7/bin/install-list | 2 ++ container/osimage/centos-7/bin/lib | 30 ++++++++++++++++ container/osimage/centos-7/build | 29 ++++++++++++++++ 6 files changed, 120 insertions(+) create mode 100644 container/osimage/centos-7/README.md create mode 100755 container/osimage/centos-7/bin/create-image create mode 100644 container/osimage/centos-7/bin/delete-list create mode 100644 container/osimage/centos-7/bin/install-list create mode 100755 container/osimage/centos-7/bin/lib create mode 100755 container/osimage/centos-7/build diff --git a/container/osimage/centos-7/README.md b/container/osimage/centos-7/README.md new file mode 100644 index 0000000..2c2f956 --- /dev/null +++ b/container/osimage/centos-7/README.md @@ -0,0 +1,13 @@ +# Use the tool to create a new centos-7 initramfs image. + +Usage: + cd ${CCI_SRC/rootfs/initramfs/centos/aarch64/7} + ./build + +Some configuration items: +./bin/install-list + If you want to pre-install the software, you can write the package names in ./bin/install-list. + +./bin/delete-list + If you want remove some unnecessary files, you can write the names in ./bin/delete-list + diff --git a/container/osimage/centos-7/bin/create-image b/container/osimage/centos-7/bin/create-image new file mode 100755 index 0000000..ad51986 --- /dev/null +++ b/container/osimage/centos-7/bin/create-image @@ -0,0 +1,38 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Configure rootfs +make_rootfs() +{ + yum repolist + yum -y install --skip-broken $(</root/bin/install-list) + yum clean all + rm -rf $(</root/bin/delete-list) + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime + cd /root/modules + zcat modules-5.8.0.1.cgz | cpio -idm +} + +# Config password +pre_config_rootfs() { + [ -n "$ROOT_NEW_PASSWD" ] && { + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" # Change the password in shadow + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" # Configure ssh service + } +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + cd / + find ./ ! -path "./${image_name}" ! -path "./root/modules/modules-5.8.0.1.cgz" | cpio -o -Hnewc | gzip -9 > /${image_name} + chmod 644 /${image_name} +} + +make_rootfs +pre_config_rootfs +pack_cgz diff --git a/container/osimage/centos-7/bin/delete-list b/container/osimage/centos-7/bin/delete-list new file mode 100644 index 0000000..ce079b9 --- /dev/null +++ b/container/osimage/centos-7/bin/delete-list @@ -0,0 +1,8 @@ +/.dockerenv +/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo diff --git a/container/osimage/centos-7/bin/install-list b/container/osimage/centos-7/bin/install-list new file mode 100644 index 0000000..5d31c40 --- /dev/null +++ b/container/osimage/centos-7/bin/install-list @@ -0,0 +1,2 @@ +openssh-server +openssl diff --git a/container/osimage/centos-7/bin/lib b/container/osimage/centos-7/bin/lib new file mode 100755 index 0000000..c3c470f --- /dev/null +++ b/container/osimage/centos-7/bin/lib @@ -0,0 +1,30 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(basename $(dirname $(realpath $0)))" +name="$(echo $DIR |awk -F "-" '{print $1}')" +version="$(echo $DIR |awk -F "-" '{print $2}')" +image="${name}:${version}" +image_name="${name}-${version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs.passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs.passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker() +{ + docker pull $image || return + echo "finish downloading image" +} + diff --git a/container/osimage/centos-7/build b/container/osimage/centos-7/build new file mode 100755 index 0000000..5e46306 --- /dev/null +++ b/container/osimage/centos-7/build @@ -0,0 +1,29 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos-7/bin/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file "$root_pwd_file" + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/bin/:/root/bin + -v /srv/initrd/modules/:/root/modules:ro + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker + +docker_rm init_docker +${cmd[@]} + +docker cp -a init_docker:/${image_name} ./ +echo "build finished!!!" + -- 2.23.0

2 4

[PATCH lkp-tests] /lkp-tests/jobs: add test-tools.yaml
by Li Ping 12 Oct '20

12 Oct '20

--- jobs/test-tools.yaml | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 jobs/test-tools.yaml diff --git a/jobs/test-tools.yaml b/jobs/test-tools.yaml new file mode 100644 index 00000000..43ca1343 --- /dev/null +++ b/jobs/test-tools.yaml @@ -0,0 +1,8 @@ +suite: test-tools +category: functional +os_mount: initramfs + +test-tools: + add: 1 + testsuite: smoke-testing + testcase: oe_test_dd_001 -- 2.23.0

1 0

[PATCH compass-ci] fix: scheduler: output exception backtrace
by Tong Qunfeng 12 Oct '20

12 Oct '20

we catch exception at submit_job and prepare_job, and just output the exception message. it is not easy to find out detail reason, so add backtrace. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/sched.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 2f432a9..23332df 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -188,6 +188,7 @@ class Sched return submit_single_job(job) rescue ex + puts ex.inspect_with_backtrace return [{ "job_id" => "0", "message" => ex.to_s, @@ -444,6 +445,7 @@ class Sched job = @es.get_job(job_id.to_s) rescue ex puts "Invalid job (id=#{job_id}) in es. Info: #{ex}" + puts ex.inspect_with_backtrace end end -- 2.23.0

1 0

[PATCH compass-ci] mount busybox into container
by Xiao Shenwei 12 Oct '20

12 Oct '20

some container may missing basic commands, such as curl wget ... use busybox to replace. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 6798f48..3c3b0ec 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -18,6 +18,7 @@ host=${tbox_group%%--*} create_yaml_variables "$LKP_SRC/hosts/${host}" DIR=$(dirname $(realpath $0)) +busybox_path=$(command -v busybox) cmd=( docker run --rm @@ -28,6 +29,7 @@ cmd=( -v ${DIR}/bin:/root/bin:ro -v $CCI_SRC:/c/compass-ci -v /srv/git:/srv/git:ro + -v ${busybox_path}:/bin/busybox --oom-score-adj="-1000" ${docker_image} /root/bin/entrypoint.sh -- 2.23.0

2 1

[PATCH v3 compass-ci] feat: build centos-7 initramfs image with on click
by Wang Chenglong 12 Oct '20

12 Oct '20

Use the tool to create a new centos-7 initramfs image with on click. and we also can customize the pre-installed software by configuring ./bin/repolist. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- rootfs/initramfs/centos/aarch64/7/README.md | 12 ++++++ rootfs/initramfs/centos/aarch64/7/bin/creat | 38 +++++++++++++++++++ rootfs/initramfs/centos/aarch64/7/bin/dellist | 8 ++++ rootfs/initramfs/centos/aarch64/7/bin/lib | 30 +++++++++++++++ .../initramfs/centos/aarch64/7/bin/repolist | 2 + rootfs/initramfs/centos/aarch64/7/build | 29 ++++++++++++++ 6 files changed, 119 insertions(+) create mode 100644 rootfs/initramfs/centos/aarch64/7/README.md create mode 100755 rootfs/initramfs/centos/aarch64/7/bin/creat create mode 100644 rootfs/initramfs/centos/aarch64/7/bin/dellist create mode 100755 rootfs/initramfs/centos/aarch64/7/bin/lib create mode 100644 rootfs/initramfs/centos/aarch64/7/bin/repolist create mode 100755 rootfs/initramfs/centos/aarch64/7/build diff --git a/rootfs/initramfs/centos/aarch64/7/README.md b/rootfs/initramfs/centos/aarch64/7/README.md new file mode 100644 index 0000000..1e47e11 --- /dev/null +++ b/rootfs/initramfs/centos/aarch64/7/README.md @@ -0,0 +1,12 @@ +# Use the tool to create a new centos-7 initramfs image. + +Usage: + ./build + +Some configuration items: +./bin/repolist + If you want to pre-install the software, you can write the package names in ./bin/repolist. + +./bin/dellist + If you want remove some unnecessary files, you can write the names in ./bin/dellist + diff --git a/rootfs/initramfs/centos/aarch64/7/bin/creat b/rootfs/initramfs/centos/aarch64/7/bin/creat new file mode 100755 index 0000000..4dacbdf --- /dev/null +++ b/rootfs/initramfs/centos/aarch64/7/bin/creat @@ -0,0 +1,38 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Configure rootfs +make_rootfs() +{ + yum repolist + yum -y install --skip-broken $(</root/bin/repolist) + yum clean all + rm -rf $(</root/bin/dellist) + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime + cd /root/modules + zcat modules-5.8.0.1.cgz | cpio -idm +} + +# Config password +pre_config_rootfs() { + [ -n "$ROOT_NEW_PASSWD" ] && { + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" # Change the password in shadow + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" # Configure ssh service + } +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + cd / + find ./ ! -path "./${image_name}" ! -path "./root/modules/modules-5.8.0.1.cgz" | cpio -o -Hnewc | gzip -9 > /${image_name} + chmod 644 /${image_name} +} + +make_rootfs +pre_config_rootfs +pack_cgz diff --git a/rootfs/initramfs/centos/aarch64/7/bin/dellist b/rootfs/initramfs/centos/aarch64/7/bin/dellist new file mode 100644 index 0000000..ce079b9 --- /dev/null +++ b/rootfs/initramfs/centos/aarch64/7/bin/dellist @@ -0,0 +1,8 @@ +/.dockerenv +/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo diff --git a/rootfs/initramfs/centos/aarch64/7/bin/lib b/rootfs/initramfs/centos/aarch64/7/bin/lib new file mode 100755 index 0000000..1bf0b5d --- /dev/null +++ b/rootfs/initramfs/centos/aarch64/7/bin/lib @@ -0,0 +1,30 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(dirname $(realpath $0))" +name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${name}:${version}" +image_name="${name}-${version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs.passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs.passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker() +{ + docker pull $image || return + echo "finish downloading image" +} + diff --git a/rootfs/initramfs/centos/aarch64/7/bin/repolist b/rootfs/initramfs/centos/aarch64/7/bin/repolist new file mode 100644 index 0000000..5d31c40 --- /dev/null +++ b/rootfs/initramfs/centos/aarch64/7/bin/repolist @@ -0,0 +1,2 @@ +openssh-server +openssl diff --git a/rootfs/initramfs/centos/aarch64/7/build b/rootfs/initramfs/centos/aarch64/7/build new file mode 100755 index 0000000..5e1a0a0 --- /dev/null +++ b/rootfs/initramfs/centos/aarch64/7/build @@ -0,0 +1,29 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ./bin/lib +. $CCI_SRC/container/defconfig.sh + +check_passwd_file "$root_pwd_file" + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/bin/:/root/bin + -v /srv/initrd/modules/:/root/modules:ro + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/creat +) + +pull_docker + +docker_rm init_docker +${cmd[@]} + +docker cp -a init_docker:/${image_name} ./ +echo "build finished!!!" + -- 2.23.0

2 4

[PATCH compass-ci] feat: a link for one command to build centos-7 initramfs image
by Wang Chenglong 12 Oct '20

12 Oct '20

This link is used to one command to build centos-7 initramfs image Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- rootfs/initramfs/centos/aarch64/7/build | 1 + 1 file changed, 1 insertion(+) create mode 120000 rootfs/initramfs/centos/aarch64/7/build diff --git a/rootfs/initramfs/centos/aarch64/7/build b/rootfs/initramfs/centos/aarch64/7/build new file mode 120000 index 0000000..e3adb9c --- /dev/null +++ b/rootfs/initramfs/centos/aarch64/7/build @@ -0,0 +1 @@ +../../../../../container/osimage/centos-7/build \ No newline at end of file -- 2.23.0

1 0

[PATCH compass-ci] fix rubocop warnings on *.cr
by Wu Fengguang 12 Oct '20

12 Oct '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- src/extract-stats/constants.cr | 2 +- src/extract-stats/extract_stats.cr | 10 +++++----- src/monitoring/monitoring.cr | 1 + src/scheduler/jobfile_operate.cr | 21 +++++++++++---------- src/spec/scheduler/tools_spec.cr | 8 ++++---- 5 files changed, 22 insertions(+), 20 deletions(-) diff --git a/src/extract-stats/constants.cr b/src/extract-stats/constants.cr index 332962a..70c66ad 100644 --- a/src/extract-stats/constants.cr +++ b/src/extract-stats/constants.cr @@ -5,4 +5,4 @@ EXTRACT_STATS_QUEUE_PATH = "extract_stats" STATS_WORKER_COUNT = 10 DELIMITER_TASK_QUEUE = "delimiter" -ERROR_ID_FILES = %w(build-pkg.json) +ERROR_ID_FILES = %w[build-pkg.json] diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index d3f3203..45310ad 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -7,15 +7,15 @@ require "./stats_worker" module ExtractStats # Consume scheduler queue def self.in_extract_stats - self.back_fill_task + back_fill_task STATS_WORKER_COUNT.times do - Process.fork { - self.consume_task - } + Process.fork do + consume_task + end end # keep main-process alive - sleep() + sleep end def self.consume_task diff --git a/src/monitoring/monitoring.cr b/src/monitoring/monitoring.cr index 178e4c9..e665d5f 100644 --- a/src/monitoring/monitoring.cr +++ b/src/monitoring/monitoring.cr @@ -24,6 +24,7 @@ module Monitoring socket.on_close do next unless query.as_h? + filter.remove_filter_rule(query, socket) end end diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 0533346..e0e298f 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -15,7 +15,7 @@ end module Jobfile::Operate def self.prepare_dir(file_path : String) file_path_dir = File.dirname(file_path) - if !File.exists?(file_path_dir) + unless File.exists?(file_path_dir) FileUtils.mkdir_p(file_path_dir) end end @@ -59,11 +59,12 @@ module Jobfile::Operate if val.as_h? return false end - if val.as_a? - value = shell_escape(val.as_a) - else - value = shell_escape(val.to_s) - end + + value = if val.as_a? + shell_escape(val.as_a) + else + shell_escape(val.to_s) + end script_lines << "\texport #{key}=" + value if value end end @@ -78,7 +79,7 @@ module Jobfile::Operate script_lines << "\texport LKP_SRC=/lkp/${user:-lkp}/src" script_lines << "}\n\n" - script_lines = "#{script_lines}" + script_lines = script_lines.to_s script_lines = JSON.parse(script_lines) end @@ -121,7 +122,7 @@ module Jobfile::Operate if job_sh_array.empty? lkp_src = prepare_lkp_tests(job_content["lkp_initrd_user"], - job_content["os_arch"]) + job_content["os_arch"]) cmd = "#{lkp_src}/sbin/create-job-cpio.sh #{temp_yaml}" idd = `#{cmd}` @@ -158,8 +159,8 @@ module Jobfile::Operate # update lkp-xxx.cgz if they are different target_path = update_lkp_when_different(expand_dir_base, - lkp_initrd_user, - os_arch) + lkp_initrd_user, + os_arch) # delete oldest lkp, if exists too much del_lkp_if_too_much(expand_dir_base) diff --git a/src/spec/scheduler/tools_spec.cr b/src/spec/scheduler/tools_spec.cr index 0a71af0..4c3a6aa 100644 --- a/src/spec/scheduler/tools_spec.cr +++ b/src/spec/scheduler/tools_spec.cr @@ -8,16 +8,16 @@ require "file_utils" describe Public do describe "hash replace" do it "can replace k:v" do - hash_old = {"name" => "OldValue"} - hash_new = {"name" => "NewValue"} + hash_old = { "name" => "OldValue" } + hash_new = { "name" => "NewValue" } hash_result = Public.hash_replace_with(hash_old, hash_new) (hash_result["name"]).should eq("NewValue") end it "can add k:v" do - hash_old = {"name2" => "OldValue"} - hash_new = {"name" => "NewValue"} + hash_old = { "name2" => "OldValue" } + hash_new = { "name" => "NewValue" } hash_result = Public.hash_replace_with(hash_old, hash_new) (hash_result["name"]).should eq("NewValue") -- 2.23.0

1 0

[PATCH lkp-tests] tests: tests: add get_ipmi_ip function to host-info
by Zhang Yu 12 Oct '20

12 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/host-info b/tests/host-info index 9b8bfc50..3cabebbf 100755 --- a/tests/host-info +++ b/tests/host-info @@ -80,6 +80,11 @@ get_partitions() fi } +get_ipmi_ip() +{ + echo "ipmi_ip: $(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" +} + main() { get_node @@ -87,6 +92,7 @@ main() get_partitions get_mac get_model_name + get_ipmi_ip } main -- 2.23.0

2 1

[PATCH lkp-tests] tests: add get_ipmi_ip function to host-info
by Zhang Yu 12 Oct '20

12 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/host-info b/tests/host-info index 9b8bfc50..11cbed38 100755 --- a/tests/host-info +++ b/tests/host-info @@ -80,6 +80,11 @@ get_partitions() fi } +get_ipmi_ip() +{ + echo "ipmi_ip: $(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" +} + main() { get_node @@ -87,6 +92,7 @@ main() get_partitions get_mac get_model_name + get_ipmi_ip() } main -- 2.23.0

1 0