- Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 09 Dec '20

09 Dec '20

[Why] generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install [How] Usage: ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. dst_rootfs_new_abspath: destination absolute path to create for rootfs Example: ./run /tmp/openEuler-docker/openEuler-docker..tar.xz /tmp/openeuler-rootfs/ Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..0f9390f --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_arch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

1 0

[PATCH v4 compass-ci 3/4] docker-rootfs: the function executed in host
by Wang Chenglong 09 Dec '20

09 Dec '20

the function is called by script on host. it contains get the password, download and load the docker image copy the image package to the host, create the links of vmlinuz and initrd. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 154 +++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100755 container/docker-rootfs/common diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common new file mode 100755 index 0000000..1d796ac --- /dev/null +++ b/container/docker-rootfs/common @@ -0,0 +1,154 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# This file is called by script run running on host +# define functions and set host environment variables + +check_cmd_input() +{ + ( [ $# -eq 2 ] && [ -f "$1" ] ) || { + usage + exit 1 + } + + check_docker_img_file "$1" + init_rootfs_dir "$2" + + export RUN_DIR=$(dirname $(realpath "$0")) +} + +usage() +{ + echo " + Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + + Example: + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + " +} + +check_docker_img_file() +{ + local allow_docker_img_suffix + local docker_img_name + local docker_img_suffix + + allow_docker_img_suffix=('tar.xz') + docker_img_name=$(basename "$1") + docker_img_suffix=${docker_img_name##*.} + + [ "$docker_img_suffix" == 'tar.xz' ] || { + docker_img_suffix=$(echo "$docker_img_name" |awk -F '.' '{print $(NF-1)"."$NF}') + echo "${allow_docker_img_suffix}" |grep -wq "$docker_img_suffix" || { + echo "[ERROR] Only support: .tar.xz file!" + exit 2 + } + } +} + +init_rootfs_dir() +{ + [ -d "$1" ] && return + + local limit_prompt_times + local current_prompt_time + + limit_prompt_times=3 + current_prompt_time=0 + + while true + do + read -r -p "[WARNING] Do you want to create \"$1\"? [y|n]> " if_create + + [ "$if_create" == 'y' ] && break + [ "$if_create" == 'n' ] && echo "[ERROR] User cancelled running." && exit + done + mkdir -p $1 +} + +check_passwd_file() +{ + export ROOT_NEW_PASSWD= + [ -f "$1" ] || { + echo "[INFO] No password file specified and root password kept." + return + } + export ROOT_NEW_PASSWD=$(cat "$1") +} + +load_docker_img() +{ + echo "Loading docker image..." + docker_repos_tag=$(docker load -i $1) + docker_name_tag=${docker_repos_tag#*:} +} + +cp_rootfs() +{ + docker cp -a rootfs-docker:/tmp/$1 /tmp/ + cd $2 + zcat /tmp/$1 |cpio -idmv +} + +create_get_initrd() { + echo "Creating initrd.lkp via container/dracut-initrd..." + cd "$CCI_SRC/container/dracut-initrd" || { + echo "Failed to change into $CCI_SRC/container/dracut-initrd" + exit 6 + } + ./run "$ROOTFS_DIR/lib/modules/$ROOTFS_KERNEL" + + echo "Finding initrd.lkp under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 7 + } + local initrd_lkp + initrd_lkp=$(find ./boot -name "initramfs.lkp*") && export ROOTFS_INITRD_LKP=$initrd_lkp + + [ -f "$ROOTFS_INITRD_LKP" ] || { + echo "Failed to generate \"$ROOTFS_INITRD_LKP\"" + exit 8 + } + + echo + echo "initrd_lkp: $ROOTFS_INITRD_LKP" +} + +get_rootfs_kernel() +{ + echo "Finding vmlinuz under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 5 + } + + local vmlinuz_file + local vmlinuz kernel + vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz + kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel + + echo "vmlinuz: $ROOTFS_VMLINUZ_FILE" + echo "kernel: $ROOTFS_KERNEL" +} + +create_links_vmlinuz_initrd() +{ + export ROOTFS_DIR=$1 + get_rootfs_kernel + create_get_initrd + + echo "Creating links to initrd.lkp and vmlinuz..." + cd $ROOTFS_DIR + ln -fs $ROOTFS_INITRD_LKP initrd.lkp + cd $ROOTFS_DIR/boot + ln -fs $ROOTFS_VMLINUZ vmlinuz + + echo "[INFO] Create links to initrd.lkp and vmlinuz success!" +} -- 2.23.0

2 2

[PATCH v2 lkp-tests] daemon/sshd: remove $testbox and format adjustment
by Zhang Yale 09 Dec '20

09 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index c7d7b60c..db013005 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -8,8 +8,8 @@ . $LKP_SRC/lib/http.sh sshr_ip=$LKP_SERVER -[ -n "$sshr_port" ] || sshr_port=5050 -[ -n "$sshr_port_len" ] || sshr_port_len=2000 +[ -n "$sshr_port" ] || sshr_port=5050 +[ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 run_ssh() @@ -24,11 +24,12 @@ run_ssh() data_success() { data="To: $my_email -Subject: [NOTIFY Compass-ci] $testbox ready to use +Subject: [NOTIFY Compass-ci] Applying machine success Dear $my_username: + Thanks for your participation in software ecosystem! - According to your application, $testbox has been provisioned. + According to your application, the test machine has been provisioned. The datails are as follows: Login: @@ -50,9 +51,10 @@ Compass-Ci data_failure() { data="To: $my_email -Subject: [NOTIFY Compass-ci] Applying $testbox failed +Subject: [NOTIFY Compass-ci] Applying machine failed Dear $my_username: + Sorry to inform you that your application failed. You may need to wait a moment, or check whether your pub_key exists. -- 2.23.0

2 2

[PATCH lkp-tests] lkp-tests/tests: add test case to run rpmbuild to build RPM
by Zhang Yale 09 Dec '20

09 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/rpmbuild-pkg | 81 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100755 tests/rpmbuild-pkg diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg new file mode 100755 index 00000000..9765eddb --- /dev/null +++ b/tests/rpmbuild-pkg @@ -0,0 +1,81 @@ +#!/bin/bash +# - benchmark +# - rpm_spec +# - os_mount +# - os +# - os_arch +# - os_version + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/upload.sh + +check_vars() +{ + [ -n "$benchmark" ] || die "benchmark is empty" + [ -n "$os_mount" ] || die "os_mount is empty" + [ -n "$os" ] || die "os is empty" + [ -n "$os_arch" ] || die "os_arch is empty" + [ -n "$os_version" ] || die "os_version is empty" + [ -n "$rpm_spec" ] || die "rpm_spec is empty" +} + +PKG_MNT=/initrd/rpmbuild-pkg +mkdir -p "$PKG_MNT" +pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} +sync_dest="$PKG_MNT/$pack_to" + +install_tools() +{ + yum clean all + yum -y install rpm-build + yum -y install rpmdevtools + mkdir -p ~/rpmbuild/{SPECS,SRPMS,SOURCES,BUILD,RPMS} +} + +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${rpm_spec}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" +} + +request_pkg() +{ + filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${rpm_spec}'", "git_command": ["git-ls-files", "."]}') + + local dir="SOURCES" + for pkgfile in ${filelist[*]} + do + echo $pkgfile | egrep "\.spec$" && { + dir="SPECS" + } + get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + done +} + +rpmbuild_pkg() +{ + spec_dir=~/rpmbuild/SPECS/$benchmark.spec + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $benchmark.spec` + # Install build depends + yum-builddep -y $spec_dir + # Download tar.gz to default path ~/rpmbuild/SOURCE + spectool -g -R $spec_dir + # Building rpm or srpm packages + rpmbuild -ba $spec_dir +} + +upload_pkg() +{ + #rpm package will be generated in "~/rpmbuild/SRPMS" and "~/rpmbuild/RPMS" + for file in $(find ~/rpmbuild/ -type f -name "*.rpm") + do + upload_one_curl ${file} ${sync_dest} + done +} + +check_vars +install_tools +request_pkg +rpmbuild_pkg +upload_pkg -- 2.23.0

3 4

[PATCH compass-ci] delimiter: parse bisect log and upload log file
by Cao Xueliang 09 Dec '20

09 Dec '20

1. Parse bisect log by git command. 2. Create a bisect log file. 3. Upload the bisect log file. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/constants.rb | 1 + src/delimiter/utils.rb | 22 ++++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 04f87f4..5e5406f 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -10,3 +10,4 @@ DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'compass-ci(a)qq.com' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" +RESULT_WEBDAV_URL = ENV['RESULT_WEBDAV_URL'] || "http://172.17.0.1:3080" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 90c3cce..60b64a1 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -113,5 +113,27 @@ module Utils return job end + + def parse_bisect_log(git_dir) + bisect_log = %x(git -C #{git_dir} bisect log) + bisect_log_arr = bisect_log.split(/\n/) + bisect_log_arr.keep_if { |item| item.start_with?('#') } + + return bisect_log_arr + end + + def create_bisect_log(job_id, git_dir) + log_file = File.join(git_dir, "#{job_id}_bisect.log") + log_content = parse_bisect_log(git_dir) + File.open(log_file, 'w') do |f| + log_content.each { |line| f.puts(line) } + end + + return log_file + end + + def upload_bisect_log(log_file, dest_dir, access_key) + %x(curl -sSf -T "#{log_file}" #{RESULT_WEBDAV_URL}/#{dest_dir} --cookie "ACCESSKEY=#{access_key}") + end end end -- 2.23.0

1 0

[PATCH v4 compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 09 Dec '20

09 Dec '20

generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..0f9390f --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_arch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

2 2

[PATCH v3 compass-ci 3/4] docker-rootfs: the function executed in host
by Wang Chenglong 09 Dec '20

09 Dec '20

the function is called by script on host. it contains get the password, download and load the docker image copy the image package to the host, create the links of vmlinuz and initrd. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 153 +++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100755 container/docker-rootfs/common diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common new file mode 100755 index 0000000..10ed266 --- /dev/null +++ b/container/docker-rootfs/common @@ -0,0 +1,153 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# This file is called by script run running on host +# define functions and set host environment variables + +check_cmd_input() +{ + ( [ $# -eq 2 ] && [ -f "$1" ] ) || { + usage + exit 1 + } + + check_docker_img_file "$1" + init_rootfs_dir "$2" + + export RUN_DIR=$(dirname $(realpath "$0")) +} + +usage() +{ + echo " + Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + + Example: + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + " +} + +check_docker_img_file() +{ + local allow_docker_img_suffix + local docker_img_name + local docker_img_suffix + + allow_docker_img_suffix=('tar.xz') + docker_img_name=$(basename "$1") + docker_img_suffix=${docker_img_name##*.} + + [ "$docker_img_suffix" == 'tar.xz' ] || { + docker_img_suffix=$(echo "$docker_img_name" |awk -F '.' '{print $(NF-1)"."$NF}') + echo "${allow_docker_img_suffix}" |grep -wq "$docker_img_suffix" || { + echo "[ERROR] Only support: .tar.xz file!" + exit 2 + } + } +} + +init_rootfs_dir() +{ + [ -d "$1" ] && return + + local limit_prompt_times + local current_prompt_time + + limit_prompt_times=3 + current_prompt_time=0 + + while true + do + read -r -p "[WARNING] Do you want to create \"$1\"? [y|n]> " if_create + + [ "$if_create" == 'y' ] && break + [ "$if_create" == 'n' ] && echo "[ERROR] User cancelled running." && exit + done + mkdir -p $1 +} + +check_passwd_file() +{ + export ROOT_NEW_PASSWD= + [ -f "$1" ] || { + echo "[INFO] No password file specified and root password kept." + return + } + + export ROOT_NEW_PASSWD=$(cat "$1") +} + +load_docker_img() +{ + echo "Loading docker image..." + docker_repos_tag=$(docker load -i $1) + docker_name_tag=${docker_repos_tag#*:} +} + +cp_rootfs() +{ + docker cp -a rootfs-docker:/tmp/$1 /tmp/ + cd $2 + zcat /tmp/$1 |cpio -idmv +} + +create_get_initrd() { + echo "Creating initrd.lkp via container/dracut-initrd..." + cd "$CCI_SRC/container/dracut-initrd" || { + echo "Failed to change into $CCI_SRC/container/dracut-initrd" + exit 6 + } + ./run "$ROOTFS_DIR/lib/modules/$ROOTFS_KERNEL" + + echo "Finding initrd.lkp under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 7 + } + local initrd_lkp + initrd_lkp=$(find ./boot -name "initramfs.lkp*") && export ROOTFS_INITRD_LKP=$initrd_lkp + + [ -f "$ROOTFS_INITRD_LKP" ] || { + echo "Failed to generate \"$ROOTFS_INITRD_LKP\"" + exit 8 + } + + echo + echo "initrd_lkp: $ROOTFS_INITRD_LKP" +} + +get_rootfs_kernel() { + echo "Finding vmlinuz under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 5 + } + local vmlinuz_file + local vmlinuz kernel + vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz + kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel + + echo "vmlinuz: $ROOTFS_VMLINUZ_FILE" + echo "kernel: $ROOTFS_KERNEL" +} + +create_links_vmlinuz_initrd() +{ + export ROOTFS_DIR=$1 + get_rootfs_kernel + create_get_initrd + + echo "Creating links to initrd.lkp and vmlinuz..." + cd $ROOTFS_DIR + ln -fs $ROOTFS_INITRD_LKP initrd.lkp + cd $ROOTFS_DIR/boot + ln -fs $ROOTFS_VMLINUZ vmlinuz + + echo "[INFO] Create links to initrd.lkp and vmlinuz success!" +} -- 2.23.0

2 4

[PATCH v4 compass-ci 4/4] docker-rootfs: list of packages that need to install
by Wang Chenglong 09 Dec '20

09 Dec '20

write the software in this file, it will install into rootfs. the kernel and kernel-devel is necessary. They provide the default kernel as well as kernel source code. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/packages-to-install | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 container/docker-rootfs/packages-to-install diff --git a/container/docker-rootfs/packages-to-install b/container/docker-rootfs/packages-to-install new file mode 100644 index 0000000..d7e890f --- /dev/null +++ b/container/docker-rootfs/packages-to-install @@ -0,0 +1,3 @@ +kernel +kernel-devel +openssh-server -- 2.23.0

1 0

[PATCH v4 compass-ci 2/4] docker-rootfs: setup rootfs in container
by Wang Chenglong 09 Dec '20

09 Dec '20

install software, set password and other actions in container. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/docker-rootfs/setup-image diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image new file mode 100755 index 0000000..8e2e88a --- /dev/null +++ b/container/docker-rootfs/setup-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +remove_file() +{ + rm -rf /.dockerenv +} + +setup_login() +{ + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + chmod 644 /tmp/${IMAGE_PACK} +} + +install_packages +remove_file +setup_login +pack_cgz -- 2.23.0

1 0

[PATCH compass-ci] container/es: fix failed to setting index
by Liu Yinsi 09 Dec '20

09 Dec '20

[why] when execute es/start, error message: curl: (56) Recv failure: Connection reset by peer because can't set index immediately after start es. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/start | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/es/start b/container/es/start index e43d01b..d84ad33 100755 --- a/container/es/start +++ b/container/es/start @@ -20,4 +20,7 @@ cmd=( "${cmd[@]}" +echo "Please wait 60s for setting index" +sleep 60 + set_es_indices -- 2.23.0

1 0

[PATCH v3 compass-ci 4/4] docker-rootfs: list of packages that need to install
by Wang Chenglong 09 Dec '20

09 Dec '20

write the software in this file, it will install into rootfs. the kernel and kernel-devel is necessary. They provide the default kernel as well as kernel source code. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/packages-to-install | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 container/docker-rootfs/packages-to-install diff --git a/container/docker-rootfs/packages-to-install b/container/docker-rootfs/packages-to-install new file mode 100644 index 0000000..d7e890f --- /dev/null +++ b/container/docker-rootfs/packages-to-install @@ -0,0 +1,3 @@ +kernel +kernel-devel +openssh-server -- 2.23.0

1 0

[PATCH v3 compass-ci 2/4] docker-rootfs: setup rootfs in container
by Wang Chenglong 09 Dec '20

09 Dec '20

install software, set password and other actions in container. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/docker-rootfs/setup-image diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image new file mode 100755 index 0000000..ee5d721 --- /dev/null +++ b/container/docker-rootfs/setup-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +remove_file() +{ + rm -rf /.dockerenv +} + +setup_login() +{ + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + chmod 644 /tmp/${IMAGE_PACK} +} + +install_packages +remove_file +setup_login +pack_cgz -- 2.23.0

1 0

[PATCH v3 compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 09 Dec '20

09 Dec '20

generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..0f9390f --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_arch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

1 0

[PATCH v6 compass-ci] mail-robot: add check account exists for email
by Luan Shengde 09 Dec '20

09 Dec '20

avoid re-assign accounts for a single email address [why] the mail-robot may assign a new account if user send 'apply account' ail with the same email address. and this is a big waste of account resource. [how] when receive request for applying an account, check whether the email already has already been assigned an account. assign a new one if there is none return the old one if there is already one and update info mail robot will update infos if the mail has been set new: - my_name - my_commit_url - my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 39 ++++++++++++++++++----- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..0fff3b5 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -51,6 +51,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +63,7 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + apply_my_account store_account_info send_mail('') @@ -82,15 +82,38 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp + def read_my_account_es + accountx_es = ESQuery.new(index: 'accounts') + accountx_es.query_by_id(@my_info['my_email']) + end - @my_info['my_uuid'] = my_uuid + def apply_my_account + my_account_es = read_my_account_es + apply_info = {} + if my_account_es + my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') + apply_info.update my_account_es + apply_info.update @my_info + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + end + + def check_account_es + ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) end def store_account_info -- 2.23.0

1 0

[PATCH compass-ci] lib/es_query: fix FrozenError
by Lu Weitao 09 Dec '20

09 Dec '20

[Why] /home/lwt/work3/compass-ci/lib/es_query.rb:34:in `multi_field_query': can't modify frozen String (FrozenError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index c1342c5..a250eaf 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -31,7 +31,7 @@ class ESQuery } }, size: size } - @client.search index: @index << '*', body: query + @client.search index: @index + '*', body: query end def traverse_field(size) -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/cci-makepkg: centos7 update gcc in container
by Lin Jiaxin 09 Dec '20

09 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a7a990a5..0354af89 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -98,7 +98,7 @@ centos_update_gcc() { grep -sqF "CentOS Linux release 7" /etc/centos-release || return - [ "$os_mount" = "initramfs" ] && { + [ -f /.dockerenv -o "$os_mount" = "initramfs" ] && { yum install -y centos-release-scl-rh yum install -y "devtoolset-8-gcc*" } -- 2.23.0

2 1

[PATCH v2 compass-ci] result-webdav/nginx.conf: expand client_max_body_size
by Lin Jiaxin 09 Dec '20

09 Dec '20

error: curl:(22) The requested URL returned error: 413 Request Entity Too Large Our biggest cgz file is 656M now, so i set client_max_body_size to 700m. /srv/initrd/pkg% find ./ -size +100M |xargs ls -alhS -rw-r--r--. 1 lkp lkp 656M Nov 4 17:52 ./nfs/openeuler/aarch64/20.03/ycsb/0.15.0-0.cgz /srv/initrd/build-pkg% find ./ -size +100M |xargs ls -alhS -rw-rw-rw-. 1 lkp lkp 383M Aug 29 04:36 ./quake2world-data-git/HEAD.cgz Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- container/result-webdav/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 98382bd..1ccaa92 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -12,7 +12,7 @@ http { listen 3080; server_name "result-webdav"; server_tokens off; - client_max_body_size 100m; + client_max_body_size 700m; access_log /tmp/access.log; error_log /tmp/error.log; -- 2.23.0

3 2

[PATCH compass-ci] container/git-mirror: add user lkp
by Li Yuanchao 09 Dec '20

09 Dec '20

Add user lkp, so user can put pub_key of ssh in /home/lkp when they use ssh to clone. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 1180a7b..0828be5 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -5,6 +5,7 @@ FROM alpine:3.8 MAINTAINER Li Yuanchao <lyc163mail(a)163.com> ADD kill-parent.sh / +RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories -- 2.23.0

2 1

[PATCH v2 compass-ci 4/4] docker-rootfs: list of packages that need to install
by Wang Chenglong 09 Dec '20

09 Dec '20

write the software in this file, it will install into rootfs. the kernel and kernel-devel is necessary. They provide the default kernel as well as kernel source code. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/packages-to-install | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 container/docker-rootfs/packages-to-install diff --git a/container/docker-rootfs/packages-to-install b/container/docker-rootfs/packages-to-install new file mode 100644 index 0000000..d7e890f --- /dev/null +++ b/container/docker-rootfs/packages-to-install @@ -0,0 +1,3 @@ +kernel +kernel-devel +openssh-server -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] docker-rootfs: the function executed in host
by Wang Chenglong 09 Dec '20

09 Dec '20

the function is called by script on host. it contains get the password, download and load the docker image copy the image package to the host, create the links of vmlinuz and initrd. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 153 +++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100755 container/docker-rootfs/common diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common new file mode 100755 index 0000000..292784e --- /dev/null +++ b/container/docker-rootfs/common @@ -0,0 +1,153 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# This file is called by script run running on host +# define functions and set host environment variables + +check_cmd_input() +{ + ( [ $# -eq 2 ] && [ -f "$1" ] ) || { + usage + exit 1 + } + + check_docker_img_file "$1" + init_rootfs_dir "$2" + + export RUN_DIR=$(dirname $(realpath "$0")) +} + +usage() +{ + echo " + Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + + Example: + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + " +} + +check_docker_img_file() +{ + local allow_docker_img_suffix + local docker_img_name + local docker_img_suffix + + allow_docker_img_suffix=('tar.xz') + docker_img_name=$(basename "$1") + docker_img_suffix=${docker_img_name##*.} + + [ "$docker_img_suffix" == 'tar.xz' ] || { + docker_img_suffix=$(echo "$docker_img_name" |awk -F '.' '{print $(NF-1)"."$NF}') + echo "${allow_docker_img_suffix}" |grep -wq "$docker_img_suffix" || { + echo "[ERROR] Only support: .tar.xz file!" + exit 2 + } + } +} + +init_rootfs_dir() +{ + [ -d "$1" ] && return + + local limit_prompt_times + local current_prompt_time + + limit_prompt_times=3 + current_prompt_time=0 + + while true + do + read -r -p "[WARNING] Do you want to create \"$1\"? [y|n]> " if_create + + [ "$if_create" == 'y' ] && break + [ "$if_create" == 'n' ] && echo "[ERROR] User cancelled running." && exit + done + mkdir -p $1 +} + +check_passwd_file() +{ + export ROOT_NEW_PASSWD= + [ -f "$1" ] || { + echo "[INFO] No password file specified and root password kept." + return + } + + export ROOT_NEW_PASSWD=$(cat "$1") +} + +load_docker_img() +{ + echo "Loading docker image..." + docker_repos_tag=$(docker load -i $1) + docker_name_tag=${docker_repos_tag#*:} +} + +cp_rootfs() +{ + docker cp -a rootfs-docker:/tmp/$1 /tmp/ + cd $2 + zcat /tmp/$1 |cpio -idmv +} + +create_get_initrd() { + echo "Creating initrd.lkp via container/dracut-initrd..." + cd "$CCI_SRC/container/dracut-initrd" || { + echo "Failed to change into $CCI_SRC/container/dracut-initrd" + exit 6 + } + ./run "$ROOTFS_DIR/lib/modules/$ROOTFS_KERNEL" + + echo "Finding initrd.lkp under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 7 + } + local initrd_lkp + initrd_lkp=$(find ./boot -name "initramfs.lkp*") && export ROOTFS_INITRD_LKP=$initrd_lkp + + [ -f "$ROOTFS_INITRD_LKP" ] || { + echo "Failed to generate \"$ROOTFS_INITRD_LKP\"" + exit 8 + } + + echo + echo "initrd_lkp: $ROOTFS_INITRD_LKP" +} + +get_rootfs_kernel() { + echo "Finding vmlinuz under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 5 + } + local vmlinuz_file + local vmlinuz kernel + vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz + kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel + + echo "vmlinuz: $ROOTFS_VMLINUZ_FILE" + echo "kernel: $ROOTFS_KERNEL" +} + +create_links_vmlinuz_initrd() +{ + export ROOTFS_DIR=$1 + get_rootfs_kernel + create_get_initrd + + echo "Creating links to initrd.lkp and vmlinuz..." + cd $ROOTFS_DIR + ln -fs $ROOTFS_INITRD_LKP initrd.lkp + cd $ROOTFS_DIR/boot + ln -fs $ROOTFS_VMLINUZ vmlinuz + + echo "[INFO] Create links to initrd.lkp and vmlinuz success!" +} -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] docker-rootfs: setup rootfs in container
by Wang Chenglong 09 Dec '20

09 Dec '20

install software, set password and other actions in container. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/docker-rootfs/setup-image diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image new file mode 100755 index 0000000..c9cb5e1 --- /dev/null +++ b/container/docker-rootfs/setup-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +remove_file() +{ + rm -rf /.dockerenv +} + +setup_login() +{ + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + chmod 644 /tmp/${IMAGE_PACK} +} + +install_packages +remove_file +setup_login +pack_cgz -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 09 Dec '20

09 Dec '20

generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..0f9390f --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_arch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-jobs-mapping.sh: add ss dynamic mapping
by Wang Yong 09 Dec '20

09 Dec '20

add ss field for linux, that can be used in auto build pkg Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- sbin/es-jobs-mapping.sh | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 8e19662..54b7e41 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -49,6 +49,15 @@ else } } }, + { + "ss": { + "path_match": "ss.*", + "mapping": { + "type": "keyword", + "enabled": true + } + } + }, { "default": { "match": "*", @@ -62,6 +71,10 @@ else } ], "properties": { + "ss": { + "dynamic": true, + "properties": {} + }, "pp": { "dynamic": true, "properties": {} -- 2.23.0

1 0

[PATCH compass-ci 3/4] docker-rootfs: the function executed in host
by Wang Chenglong 09 Dec '20

09 Dec '20

the function is called by script on host. it contains get the password, download and load the docker image copy the image package to the host, create the links of vmlinuz and initrd. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 148 +++++++++++++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100755 container/docker-rootfs/common diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common new file mode 100755 index 0000000..1d0e667 --- /dev/null +++ b/container/docker-rootfs/common @@ -0,0 +1,148 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# This file is called by script run running on host +# define functions and set host environment variables + +check_cmd_input() +{ + ( [ $# -eq 2 ] && [ -f "$1" ] ) || { + usage + exit 1 + } + + check_docker_img_file "$1" + init_rootfs_dir "$2" + + export RUN_DIR=$(dirname $(realpath "$0")) +} + +usage() +{ + echo " + Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + + Example: + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + " +} + +check_docker_img_file() +{ + local allow_docker_img_suffix docker_img_name docker_img_suffix + allow_docker_img_suffix=('tar.xz') + docker_img_name=$(basename "$1") + docker_img_suffix=${docker_img_name##*.} + [ "$docker_img_suffix" == 'tar.xz' ] || { + docker_img_suffix=$(echo "$docker_img_name" |awk -F '.' '{print $(NF-1)"."$NF}') + echo "${allow_docker_img_suffix}" |grep -wq "$docker_img_suffix" || { + echo "[ERROR] Only support: .tar.xz file!" + exit 2 + } + } +} + +init_rootfs_dir() +{ + [ -d "$1" ] && return + + local limit_prompt_times current_prompt_time + limit_prompt_times=3 + current_prompt_time=0 + while true + do + read -r -p "[WARNING] Do you want to create \"$1\"? [y|n]> " if_create + + [ "$if_create" == 'y' ] && break + [ "$if_create" == 'n' ] && echo "[ERROR] User cancelled running." && exit + done + + mkdir -p $1 +} + + +check_passwd_file() +{ + export ROOT_NEW_PASSWD= + [ -f "$1" ] || { + echo "[INFO] No password file specified and root password kept." + return + } + + export ROOT_NEW_PASSWD=$(cat "$1") +} + +load_docker_img() +{ + echo "Loading docker image..." + docker_repos_tag=$(docker load -i $1) + docker_name_tag=${docker_repos_tag#*:} +} + +cp_rootfs() +{ + docker cp -a rootfs-docker:/tmp/$1 /tmp/ + cd $2 + zcat /tmp/$1 |cpio -idmv + +} + +create_get_initrd() { + echo "Creating initrd.lkp via container/dracut-initrd..." + cd "$CCI_SRC/container/dracut-initrd" || { + echo "Failed to change into $CCI_SRC/container/dracut-initrd" + exit 6 + } + ./run "$ROOTFS_DIR/lib/modules/$ROOTFS_KERNEL" + + echo "Finding initrd.lkp under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 7 + } + local initrd_lkp + initrd_lkp=$(find ./boot -name "initramfs.lkp*") && export ROOTFS_INITRD_LKP=$initrd_lkp + + [ -f "$ROOTFS_INITRD_LKP" ] || { + echo "Failed to generate \"$ROOTFS_INITRD_LKP\"" + exit 8 + } + + echo + echo "initrd_lkp: $ROOTFS_INITRD_LKP" +} + +get_rootfs_kernel() { + echo "Finding vmlinuz under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 5 + } + local vmlinuz_file vmlinuz kernel + vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz + kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel + + echo "vmlinuz: $ROOTFS_VMLINUZ_FILE" + echo "kernel: $ROOTFS_KERNEL" +} + +create_links_vmlinuz_initrd() +{ + export ROOTFS_DIR=$1 + get_rootfs_kernel + create_get_initrd + + echo "Creating links to initrd.lkp and vmlinuz..." + cd $ROOTFS_DIR + ln -fs $ROOTFS_INITRD_LKP initrd.lkp + cd $ROOTFS_DIR/boot + ln -fs $ROOTFS_VMLINUZ vmlinuz + + echo "[INFO] Create links to initrd.lkp and vmlinuz success!" +} -- 2.23.0

4 6

[PATCH v2 lkp-tests 3/3] Automatic compilation and configure the variables
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- install.sh | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 install.sh diff --git a/install.sh b/install.sh new file mode 100644 index 00000000..4bcc1afb --- /dev/null +++ b/install.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +. lib/env.sh + +make + +source $(shell_profile) -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] sbin/install-dependencies.sh: modify the way to add ruby env-variables
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 7df3652f..3436ee44 100644 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,6 +1,7 @@ #!/bin/bash . lib/env.sh +. sbin/set-env.sh # choose install function base on # DISTRIBUTION @@ -30,8 +31,7 @@ esac mac_dep() { brew install ruby - echo "export PATH=/usr/local/opt/ruby/bin:$PATH" >> $(shell_profile) - source $(shell_profile) + write_shell_profile "export PATH=/usr/local/opt/ruby/bin:$PATH" gem install git activesupport rest-client } -- 2.23.0

1 0

[PATCH compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 09 Dec '20

09 Dec '20

generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..de9c3a7 --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_arch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +#docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

4 6

[PATCH compass-ci] lib/json_logger: fix initializate without env
by Wu Zhende 09 Dec '20

09 Dec '20

[Error] In lib/json_logger.cr:9:61 9 | def initialize(logdev = STDOUT, formatter = my_formatter, @env = nil) Error: instance variable @env of JSONLogger was inferred to be Nil, but Nil alone provides no information This error is reported when "JSONLogger.new" is used. Fix it. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index 9596243..dd87020 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -4,6 +4,9 @@ require "logger" require "json" require "any_merge" +require "kemal" + +add_context_storage_type(Time::Span) class JSONLogger < Logger def initialize(logdev = STDOUT, formatter = my_formatter, @env = nil) -- 2.23.0

1 0

[PATCH compass-ci 2/4] docker-rootfs: setup rootfs in container
by Wang Chenglong 09 Dec '20

09 Dec '20

install software, set password and other actions in container. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/docker-rootfs/setup-image diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image new file mode 100755 index 0000000..0fa1eb6 --- /dev/null +++ b/container/docker-rootfs/setup-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +remove_file() +{ + rm -rf /.dockerenv +} + +setup_login() +{ + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + chmod 644 /tmp/${IMAGE_PACK} +} + +install_packages +remove_file +setup_login +pack_cgz -- 2.23.0

2 2

[PATCH compass-ci] container/kibana: optimize logtrail
by Wu Zhende 09 Dec '20

09 Dec '20

1. show all ES index 2. set the maximum number of logs to be displayed to 10000 3. set the display of logs generated within 30 days Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/logtrail.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/kibana/logtrail.json b/container/kibana/logtrail.json index 304a4b3..c63f2ac 100644 --- a/container/kibana/logtrail.json +++ b/container/kibana/logtrail.json @@ -2,7 +2,7 @@ "version": 2, "index_patterns": [{ "es": { - "default_index": "scheduler-3016", + "default_index": "*", "allow_url_parameter": false, "timezone": "CST" }, @@ -12,9 +12,9 @@ "display_timestamp_format": "YYYY MMM DD HH:mm:ss", "max_buckets": 500, "nested_objects": false, - "default_time_range_in_days": 0, + "default_time_range_in_days": 30, "max_hosts": 100, - "max_events_to_keep_in_viewer": 5000, + "max_events_to_keep_in_viewer": 10000, "default_search": "", "fields": { "mapping": { -- 2.23.0

1 0

[PATCH lkp-tests 4/5] sbin/set-env.sh: take apart the method of adding env-variables
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/set-env.sh | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/sbin/set-env.sh b/sbin/set-env.sh index a8b161fa..ce1ad427 100644 --- a/sbin/set-env.sh +++ b/sbin/set-env.sh @@ -2,10 +2,17 @@ . lib/env.sh -write_shellrc() +submit_env() +{ + echo $@ >> $(shell_profile) + source $(shell_profile) +} + +write_env() { echo "export LKP_SRC=$PWD" >> $(shell_profile) echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $(shell_profile) + source $(shell_profile) } write_host() @@ -28,5 +35,5 @@ write_host() EOF } -write_shellrc +write_env write_host -- 2.23.0

4 3

[PATCH v3 compass-ci 2/3] lib/compare_data_format.rb: fix NoMethodError
by Lu Kaiyi 09 Dec '20

09 Dec '20

[why] @metrics_compare_results[x_param][metric][value_type] maybe nil, if so, @metrics_compare_results[x_param][metric][value_type][dimension] will get the error like below: /home/lukaiyi/compass-ci/lib/compare_data_format.rb:57:in `block (2 levels) in assign_echart_source': undefined method `[]' for nil:NilClass (NoMethodError) from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:55:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:55:in `block in assign_echart_source' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:53:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:53:in `assign_echart_source' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:46:in `block in assign_echart_data_set' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:39:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:39:in `assign_echart_data_set' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:30:in `block (2 levels) in convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:29:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:29:in `block in convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:28:in `each_value' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:28:in `convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:21:in `format_for_echart' from /home/lukaiyi/compass-ci/lib/compare_matrixes.rb:512:in `show_compare_result' from /home/lukaiyi/compass-ci/lib/compare.rb:103:in `compare_by_template' from /home/lukaiyi/compass-ci/sbin/compare:70:in `<main>' [how] also judge the value of @metrics_compare_results[x_param][metric][value_type] before use. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index 1959045..bf4013b 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -53,7 +53,7 @@ class FormatEchartData dimensions.each do |dimension| dimension_values = [dimension] @x_params.each do |x_param| - if @metrics_compare_results[x_param][metric] + if @metrics_compare_results[x_param][metric] && @metrics_compare_results[x_param][metric][value_type] dimension_values << @metrics_compare_results[x_param][metric][value_type][dimension] else source[0].delete(x_param) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] sbin/compare_template: friendly show compare template result in command line
by Lu Kaiyi 09 Dec '20

09 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the json format result of "compare -t <compare_template.yaml>" with third-party module in python3. [usage] compare_template <compare_template.yaml> [example] compare_template compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] Hackbench Performance Testing +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | openeuler 20.03 average | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | openeuler 20.03 standard_deviation | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ Hackbench Performance Testing +------------------------------------+-----------+----------+----------+----------+---------+---------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+---------+---------+---------+----------+ | openeuler 20.03 average | 122003.54 | 33528.53 | 31469.06 | 13870.14 | 8249.71 | 4329.45 | 1976.54 | 1141.00 | | openeuler 20.03 standard_deviation | 174.00 | 188.00 | 171.00 | 197.00 | 181.00 | 175.00 | 170.00 | 176.00 | +------------------------------------+-----------+----------+----------+----------+---------+---------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- sbin/compare_template | 53 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 sbin/compare_template diff --git a/sbin/compare_template b/sbin/compare_template new file mode 100755 index 0000000..6b702c0 --- /dev/null +++ b/sbin/compare_template @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +import os +import sys +import json +import prettytable as pt + +# receive compare_template.yaml and auto pretty show compare results +class TableShow: + def __init__(self, result_dict): + self.title = result_dict['title'] + self.tables = result_dict['tables'] + self.tb = None + + def show_table(self): + for (table_title, table) in self.tables.items(): + self.tb = pt.PrettyTable() + self.set_field_names(table, table_title) + self.set_align(table_title) + self.add_row(table) + self.print_table() + + def set_field_names(self, table, table_title): + field_names = [table_title] + field_names.extend(table['average']['source'][0]) + self.tb.field_names = field_names + + def set_align(self, table_title): + for field_name in self.tb.field_names: + self.tb.align[field_name] = 'r' + self.tb.align[table_title] = 'l' + + def add_row(self, table): + row_names = ['average', 'standard_deviation'] + for row_name in row_names: + row = table[row_name]['source'][1] + row_title = ' '.join([row[0], row_name]) + format_data_row = ["%.2f" % data for data in row[1:]] + self.tb.add_row([row_title, *format_data_row]) + + def print_table(self): + print(self.title) + print(self.tb) + print() + + +if __name__ == '__main__': + template_yaml = sys.argv[1] + result_dict = json.loads(os.popen(f"compare -t {template_yaml}").read()) + table_show = TableShow(result_dict) + table_show.show_table() -- 2.23.0

1 0

[PATCH v3 compass-ci] src/lib: add delete function in job.cr
by Zhang Yu 09 Dec '20

09 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- src/lib/job.cr | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4345e0..196516f 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -136,6 +136,15 @@ class Job update(json.as_h) end + def delete(key : String) + initialized_keys = get_initialized_keys + if initialized_keys.includes?(key) + raise "Should not delete #{key}" + else + @hash.delete(key) + end + end + private def set_defaults append_init_field() set_user_lkp_src() @@ -324,7 +333,7 @@ class Job @hash.delete("my_uuid") end - private def initialized? + private def get_initialized_keys() initialized_keys = [] of String REQUIRED_KEYS.each do |key| @@ -350,7 +359,10 @@ class Job "kernel_uri", "kernel_params", "linux_vmlinuz_path"] + end + private def initialized? + initialized_keys = get_initialized_keys initialized_keys.each do |key| return false unless @hash.has_key?(key) end -- 2.23.0

2 2

[PATCH compass-ci] lib/compare_data_format.rb: fix NoMethodError
by Lu Kaiyi 09 Dec '20

09 Dec '20

[error] /home/lukaiyi/compass-ci/lib/compare_data_format.rb:57:in `block (2 levels) in assign_echart_source': undefined method `[]' for nil:NilClass (NoMethodError) from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:55:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:55:in `block in assign_echart_source' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:53:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:53:in `assign_echart_source' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:46:in `block in assign_echart_data_set' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:39:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:39:in `assign_echart_data_set' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:30:in `block (2 levels) in convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:29:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:29:in `block in convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:28:in `each_value' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:28:in `convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:21:in `format_for_echart' from /home/lukaiyi/compass-ci/lib/compare_matrixes.rb:512:in `show_compare_result' from /home/lukaiyi/compass-ci/lib/compare.rb:103:in `compare_by_template' from /home/lukaiyi/compass-ci/sbin/compare:70:in `<main>' Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index 1959045..bf4013b 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -53,7 +53,7 @@ class FormatEchartData dimensions.each do |dimension| dimension_values = [dimension] @x_params.each do |x_param| - if @metrics_compare_results[x_param][metric] + if @metrics_compare_results[x_param][metric] && @metrics_compare_results[x_param][metric][value_type] dimension_values << @metrics_compare_results[x_param][metric][value_type][dimension] else source[0].delete(x_param) -- 2.23.0

2 1

[PATCH compass-ci] container: keep format remains consistent
by Liu Yinsi 09 Dec '20

09 Dec '20

keep start script content format consistent with others, looks more suitable. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/git-daemon/start | 12 +++++++++++- container/redis/start | 13 ++++++++++++- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/container/git-daemon/start b/container/git-daemon/start index 5b4c90b..72330aa 100755 --- a/container/git-daemon/start +++ b/container/git-daemon/start @@ -6,8 +6,18 @@ docker_rm git-daemon -docker run --restart=always -d -p 9418:9418 -v /srv/git:/git -v /etc/localtime:/etc/localtime:ro --name git-daemon apline311:git-daemon +cmd=( + docker run + --restart=always + --name git-daemon + -d + -p 9418:9418 + -v /srv/git:/git + -v /etc/localtime:/etc/localtime:ro + apline311:git-daemon +) +"${cmd[@]}" # test diff --git a/container/redis/start b/container/redis/start index 827ad0b..34266a0 100755 --- a/container/redis/start +++ b/container/redis/start @@ -10,7 +10,18 @@ docker_rm redis507s01 -docker run --restart=always -d -p 6379:6379 -v /srv/redis/alpine/server01:/srv/redis -v /etc/localtime:/etc/localtime:ro --name redis507s01 redis507n:alpine311 +cmd=( + docker run + --restart=always + --name redis507s01 + -d + -p 6379:6379 + -v /srv/redis/alpine/server01:/srv/redis + -v /etc/localtime:/etc/localtime:ro + redis507n:alpine311 +) + +"${cmd[@]}" # test echo you can use cmd: redis-cli to start a redis client -- 2.23.0

2 2

[PATCH v2 compass-ci] doc/job: add result_root.md
by Zhang Yu 09 Dec '20

09 Dec '20

Add a document for introducing the result_root Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/result_root.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/job/result_root.md diff --git a/doc/job/result_root.md b/doc/job/result_root.md new file mode 100644 index 0000000..827e40e --- /dev/null +++ b/doc/job/result_root.md @@ -0,0 +1,9 @@ +# result_root + +Meaning: +- The result_root field specifies the path for storing the result +- if this field is not specified, the default value is: '''/result/#{suite}/#{submit_date}/ + #{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}"''' +- we can specify the value of result_root in the job.yaml file, here is an example: + '''result_root: /srv/result/iperf''' + -- 2.23.0

2 3

[PATCH v2 lkp-tests] tests/build-pkg: support squid download remote files
by Liu Shaofei 09 Dec '20

09 Dec '20

When building software in PKGBUILD file, download upstream files often times out. We need to cache these files by squid server.The example is as follow: https://invisible-mirror.net/archives/conflict/conflict-20150705.tgz => http://invisible-mirror.net/archives/conflict/conflict-20150705.tgz Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index e4eca747..2729070a 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -108,6 +108,8 @@ replace_source() echo "$url" | egrep 'git\+|\.git$' && { url="\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" } + url=$(echo "$url" | sed 's|https:\/\/|http:\/\/|g') + echo "$url" >> $1 done echo ")" >> $1 @@ -126,8 +128,7 @@ build_source_pkg() repo_dir="${repo_dir%.git*}::" fi } - - replace_source "PKGBUILD.src" + replace_source PKGBUILD.src [ -n "$PKGBUILD_TAG" ] && replace_source "PKGBUILD-$PKGBUILD_TAG" -- 2.23.0

3 2

[PATCH v2 compass-ci] doc/job: add result_root.md
by Zhang Yu 09 Dec '20

09 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/result_root.md | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 doc/job/result_root.md diff --git a/doc/job/result_root.md b/doc/job/result_root.md new file mode 100644 index 0000000..26fa590 --- /dev/null +++ b/doc/job/result_root.md @@ -0,0 +1,8 @@ +# result_root + +Meaning: +- The result_root field specifies the path for storing the result +- if this field is not specified, the default value is: '''/result/#{suite}/#{submit_date}/ + #{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}"''' +- we can specify the value of result_root in the job.yaml file, here is an example: + '''result_root: /srv/result/iperf''' -- 2.23.0

2 2

[PATCH compass-ci] container: change the log storage path on container
by Bai Jing 09 Dec '20

09 Dec '20

[Why] Because the conserver-head.cf specifies the log storage path on the host machine, so to ensure that the storage path of the host machine is the same as that of the container Signed-off-by: Bai Jing <799286817(a)qq.com> --- container/conserver/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/conserver/start b/container/conserver/start index 58b2242..ae44291 100755 --- a/container/conserver/start +++ b/container/conserver/start @@ -25,7 +25,7 @@ cmd=( -e "IPMI_PASSWORD=${passwd}" --name conserver_server -v /etc/localtime:/etc/localtime:ro - -v /srv/cci/serial/logs/:/var/consoles/ + -v /srv/cci/serial/logs/:/srv/cci/serial/logs/ -p 782:782 conserver_server ) -- 2.23.0

1 0

[PATCH compass-ci] container: change the log storage path on the host machine
by Bai Jing 09 Dec '20

09 Dec '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- container/conserver/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/conserver/start b/container/conserver/start index 58b2242..ae44291 100755 --- a/container/conserver/start +++ b/container/conserver/start @@ -25,7 +25,7 @@ cmd=( -e "IPMI_PASSWORD=${passwd}" --name conserver_server -v /etc/localtime:/etc/localtime:ro - -v /srv/cci/serial/logs/:/var/consoles/ + -v /srv/cci/serial/logs/:/srv/cci/serial/logs/ -p 782:782 conserver_server ) -- 2.23.0

3 4

[PATCH v2 compass-ci] lib/es_query.rb: index error during query
by Cui Yili 08 Dec '20

08 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- lib/es_query.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3827ea8..c1342c5 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -31,7 +31,7 @@ class ESQuery } }, size: size } - @client.search index: 'jobs*', body: query + @client.search index: @index << '*', body: query end def traverse_field(size) -- 2.23.0

1 0

[PATCH lkp-tests 5/5] sbin/install-dependencies.sh: modify the way to add Ruby env-variables
by Hu Xuejiao 08 Dec '20

08 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 7df3652f..0ea852c7 100644 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,6 +1,7 @@ #!/bin/bash . lib/env.sh +. lin/set-env.sh # choose install function base on # DISTRIBUTION @@ -30,8 +31,7 @@ esac mac_dep() { brew install ruby - echo "export PATH=/usr/local/opt/ruby/bin:$PATH" >> $(shell_profile) - source $(shell_profile) + submit_env "export PATH=/usr/local/opt/ruby/bin:$PATH" gem install git activesupport rest-client } -- 2.23.0

2 1

[PATCH lkp-tests] daemon/sshd: remove $testbox and format adjustment
by Zhang Yale 08 Dec '20

08 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/daemon/sshd b/daemon/sshd index c7d7b60c..3764b19e 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -27,8 +27,9 @@ data_success() Subject: [NOTIFY Compass-ci] $testbox ready to use Dear $my_username: + Thanks for your participation in software ecosystem! - According to your application, $testbox has been provisioned. + According to your application, the machine has been provisioned. The datails are as follows: Login: @@ -53,6 +54,7 @@ data_failure() Subject: [NOTIFY Compass-ci] Applying $testbox failed Dear $my_username: + Sorry to inform you that your application failed. You may need to wait a moment, or check whether your pub_key exists. -- 2.23.0

1 0

[PATCH lkp-tests] Automatic compilation and configure the variables
by Hu Xuejiao 08 Dec '20

08 Dec '20

[why] Source shell_file based on different shells. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- install.sh | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 install.sh diff --git a/install.sh b/install.sh new file mode 100644 index 00000000..fb3f2e18 --- /dev/null +++ b/install.sh @@ -0,0 +1,6 @@ +#!/bin/bash + +. lib/env.sh + +make +source $(shell_profile) -- 2.23.0

2 1

[PATCH lkp-tests] tests/build-pkg: support squid download remote files
by Liu Shaofei 08 Dec '20

08 Dec '20

When building software in PKGBUILD file, download upstream files often times out. We need to cache these files by squid server.The example is as follow: https://invisible-mirror.net/archives/conflict/conflict-20150705.tgz => http://invisible-mirror.net/archives/conflict/conflict-20150705.tgz Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index e4eca747..c1b0513e 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -106,8 +106,12 @@ replace_source() for url in ${source[@]} do echo "$url" | egrep 'git\+|\.git$' && { - url="\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + [ -n "$repo_dir" ] && { + url="\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + } } + url=$(echo "$url" | sed 's|https:\/\/|http:\/\/|g') + echo "$url" >> $1 done echo ")" >> $1 @@ -126,8 +130,7 @@ build_source_pkg() repo_dir="${repo_dir%.git*}::" fi } - - replace_source "PKGBUILD.src" + replace_source PKGBUILD.src [ -n "$PKGBUILD_TAG" ] && replace_source "PKGBUILD-$PKGBUILD_TAG" -- 2.23.0

2 1

[PATCH lkp-tests] tests/cci-makepkg: centos7 update gcc in container
by Lin Jiaxin 08 Dec '20

08 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a7a990a5..7c6c9808 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -98,7 +98,7 @@ centos_update_gcc() { grep -sqF "CentOS Linux release 7" /etc/centos-release || return - [ "$os_mount" = "initramfs" ] && { + [ "$os_mount" = "initramfs" -o is_docker ] && { yum install -y centos-release-scl-rh yum install -y "devtoolset-8-gcc*" } -- 2.23.0

2 1

[PATCH compass-ci] lib/error_messages.rb: refactor the array structure
by Lin Jiaxin 08 Dec '20

08 Dec '20

before: #<Set: {"src/ssl_sock.c:1454:104: warning: unused parameter ‘al’ [-Wunused-parameter]\n int ssl_sock_sctl_add_cbk(SSL *ssl, unsigned ext_type, const unsigned char **out, size_t *outlen, int *al, void *add_arg)\n ^~\n"}> after: src/ssl_sock.c:1454:104: warning: unused parameter ‘al’ [-Wunused-parameter] int ssl_sock_sctl_add_cbk(SSL *ssl, unsigned ext_type, const unsigned char **out, size_t *outlen, int *al, void *add_arg) ^~ Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 3a74d29..5bbc329 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -43,9 +43,9 @@ class ErrorMessages def obtain_error_messages_by_errorid(errorid) error_messages_by_errorid = [] error_messages = obtain_error_messages - error_messages.each do | k, v | - if ("build-pkg." + build_pkg_error_id(k)) == errorid - error_messages_by_errorid << v + error_messages.each do |k, v| + if ('build-pkg.' + build_pkg_error_id(k)) == errorid + error_messages_by_errorid += v.to_a end end error_messages_by_errorid -- 2.23.0

2 1

[PATCH compass-ci] lib/es_query.rb: index error during query
by Cui Yili 08 Dec '20

08 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- lib/es_query.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3827ea8..c1342c5 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -31,7 +31,7 @@ class ESQuery } }, size: size } - @client.search index: 'jobs*', body: query + @client.search index: @index << '*', body: query end def traverse_field(size) -- 2.23.0

3 2

[PATCH compass-ci] container/kibana: support non-local ES
by Wu Zhende 08 Dec '20

08 Dec '20

The original startup mode supports only the local ES, but the ES may be deployed on other hosts. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/start | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/container/kibana/start b/container/kibana/start index 6a504fb..dbb3c56 100755 --- a/container/kibana/start +++ b/container/kibana/start @@ -3,17 +3,27 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'set' require_relative '../defconfig.rb' -docker_rm "kibana" +names = Set.new %w[ + LOGGING_ES_HOST + LOGGING_ES_PORT +] + +defaults = relevant_defaults(names) +LOGGING_ES_HOST = defaults['LOGGING_ES_HOST'] || '172.17.0.1' +LOGGING_ES_PORT = defaults['LOGGING_ES_PORT'] || '9202' + +docker_rm 'kibana' -cmd = %w[ +cmd = %W[ docker run --restart=always --name kibana -v /etc/localtime:/etc/localtime:ro -d - --link logging-es:elasticsearch + -e ELASTICSEARCH_HOSTS=http://#{LOGGING_ES_HOST}:#{LOGGING_ES_PORT} -p 11309:5601 kibana:7.6.2 ] -- 2.23.0

2 1