December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci 3/4] docker-rootfs: the function executed in host
by Wang Chenglong 09 Dec '20

09 Dec '20

the function is called by script on host. it contains get the password, download and load the docker image copy the image package to the host, create the links of vmlinuz and initrd. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 154 +++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100755 container/docker-rootfs/common diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common new file mode 100755 index 0000000..1d796ac --- /dev/null +++ b/container/docker-rootfs/common @@ -0,0 +1,154 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# This file is called by script run running on host +# define functions and set host environment variables + +check_cmd_input() +{ + ( [ $# -eq 2 ] && [ -f "$1" ] ) || { + usage + exit 1 + } + + check_docker_img_file "$1" + init_rootfs_dir "$2" + + export RUN_DIR=$(dirname $(realpath "$0")) +} + +usage() +{ + echo " + Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + + Example: + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + " +} + +check_docker_img_file() +{ + local allow_docker_img_suffix + local docker_img_name + local docker_img_suffix + + allow_docker_img_suffix=('tar.xz') + docker_img_name=$(basename "$1") + docker_img_suffix=${docker_img_name##*.} + + [ "$docker_img_suffix" == 'tar.xz' ] || { + docker_img_suffix=$(echo "$docker_img_name" |awk -F '.' '{print $(NF-1)"."$NF}') + echo "${allow_docker_img_suffix}" |grep -wq "$docker_img_suffix" || { + echo "[ERROR] Only support: .tar.xz file!" + exit 2 + } + } +} + +init_rootfs_dir() +{ + [ -d "$1" ] && return + + local limit_prompt_times + local current_prompt_time + + limit_prompt_times=3 + current_prompt_time=0 + + while true + do + read -r -p "[WARNING] Do you want to create \"$1\"? [y|n]> " if_create + + [ "$if_create" == 'y' ] && break + [ "$if_create" == 'n' ] && echo "[ERROR] User cancelled running." && exit + done + mkdir -p $1 +} + +check_passwd_file() +{ + export ROOT_NEW_PASSWD= + [ -f "$1" ] || { + echo "[INFO] No password file specified and root password kept." + return + } + export ROOT_NEW_PASSWD=$(cat "$1") +} + +load_docker_img() +{ + echo "Loading docker image..." + docker_repos_tag=$(docker load -i $1) + docker_name_tag=${docker_repos_tag#*:} +} + +cp_rootfs() +{ + docker cp -a rootfs-docker:/tmp/$1 /tmp/ + cd $2 + zcat /tmp/$1 |cpio -idmv +} + +create_get_initrd() { + echo "Creating initrd.lkp via container/dracut-initrd..." + cd "$CCI_SRC/container/dracut-initrd" || { + echo "Failed to change into $CCI_SRC/container/dracut-initrd" + exit 6 + } + ./run "$ROOTFS_DIR/lib/modules/$ROOTFS_KERNEL" + + echo "Finding initrd.lkp under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 7 + } + local initrd_lkp + initrd_lkp=$(find ./boot -name "initramfs.lkp*") && export ROOTFS_INITRD_LKP=$initrd_lkp + + [ -f "$ROOTFS_INITRD_LKP" ] || { + echo "Failed to generate \"$ROOTFS_INITRD_LKP\"" + exit 8 + } + + echo + echo "initrd_lkp: $ROOTFS_INITRD_LKP" +} + +get_rootfs_kernel() +{ + echo "Finding vmlinuz under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 5 + } + + local vmlinuz_file + local vmlinuz kernel + vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz + kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel + + echo "vmlinuz: $ROOTFS_VMLINUZ_FILE" + echo "kernel: $ROOTFS_KERNEL" +} + +create_links_vmlinuz_initrd() +{ + export ROOTFS_DIR=$1 + get_rootfs_kernel + create_get_initrd + + echo "Creating links to initrd.lkp and vmlinuz..." + cd $ROOTFS_DIR + ln -fs $ROOTFS_INITRD_LKP initrd.lkp + cd $ROOTFS_DIR/boot + ln -fs $ROOTFS_VMLINUZ vmlinuz + + echo "[INFO] Create links to initrd.lkp and vmlinuz success!" +} -- 2.23.0

2 2

[PATCH v2 lkp-tests] daemon/sshd: remove $testbox and format adjustment
by Zhang Yale 09 Dec '20

09 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index c7d7b60c..db013005 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -8,8 +8,8 @@ . $LKP_SRC/lib/http.sh sshr_ip=$LKP_SERVER -[ -n "$sshr_port" ] || sshr_port=5050 -[ -n "$sshr_port_len" ] || sshr_port_len=2000 +[ -n "$sshr_port" ] || sshr_port=5050 +[ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 run_ssh() @@ -24,11 +24,12 @@ run_ssh() data_success() { data="To: $my_email -Subject: [NOTIFY Compass-ci] $testbox ready to use +Subject: [NOTIFY Compass-ci] Applying machine success Dear $my_username: + Thanks for your participation in software ecosystem! - According to your application, $testbox has been provisioned. + According to your application, the test machine has been provisioned. The datails are as follows: Login: @@ -50,9 +51,10 @@ Compass-Ci data_failure() { data="To: $my_email -Subject: [NOTIFY Compass-ci] Applying $testbox failed +Subject: [NOTIFY Compass-ci] Applying machine failed Dear $my_username: + Sorry to inform you that your application failed. You may need to wait a moment, or check whether your pub_key exists. -- 2.23.0

2 2

[PATCH lkp-tests] lkp-tests/tests: add test case to run rpmbuild to build RPM
by Zhang Yale 09 Dec '20

09 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/rpmbuild-pkg | 81 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100755 tests/rpmbuild-pkg diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg new file mode 100755 index 00000000..9765eddb --- /dev/null +++ b/tests/rpmbuild-pkg @@ -0,0 +1,81 @@ +#!/bin/bash +# - benchmark +# - rpm_spec +# - os_mount +# - os +# - os_arch +# - os_version + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/upload.sh + +check_vars() +{ + [ -n "$benchmark" ] || die "benchmark is empty" + [ -n "$os_mount" ] || die "os_mount is empty" + [ -n "$os" ] || die "os is empty" + [ -n "$os_arch" ] || die "os_arch is empty" + [ -n "$os_version" ] || die "os_version is empty" + [ -n "$rpm_spec" ] || die "rpm_spec is empty" +} + +PKG_MNT=/initrd/rpmbuild-pkg +mkdir -p "$PKG_MNT" +pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} +sync_dest="$PKG_MNT/$pack_to" + +install_tools() +{ + yum clean all + yum -y install rpm-build + yum -y install rpmdevtools + mkdir -p ~/rpmbuild/{SPECS,SRPMS,SOURCES,BUILD,RPMS} +} + +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${rpm_spec}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" +} + +request_pkg() +{ + filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'${rpm_spec}'", "git_command": ["git-ls-files", "."]}') + + local dir="SOURCES" + for pkgfile in ${filelist[*]} + do + echo $pkgfile | egrep "\.spec$" && { + dir="SPECS" + } + get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + done +} + +rpmbuild_pkg() +{ + spec_dir=~/rpmbuild/SPECS/$benchmark.spec + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $benchmark.spec` + # Install build depends + yum-builddep -y $spec_dir + # Download tar.gz to default path ~/rpmbuild/SOURCE + spectool -g -R $spec_dir + # Building rpm or srpm packages + rpmbuild -ba $spec_dir +} + +upload_pkg() +{ + #rpm package will be generated in "~/rpmbuild/SRPMS" and "~/rpmbuild/RPMS" + for file in $(find ~/rpmbuild/ -type f -name "*.rpm") + do + upload_one_curl ${file} ${sync_dest} + done +} + +check_vars +install_tools +request_pkg +rpmbuild_pkg +upload_pkg -- 2.23.0

3 4

[PATCH compass-ci] delimiter: parse bisect log and upload log file
by Cao Xueliang 09 Dec '20

09 Dec '20

1. Parse bisect log by git command. 2. Create a bisect log file. 3. Upload the bisect log file. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/constants.rb | 1 + src/delimiter/utils.rb | 22 ++++++++++++++++++++++ 2 files changed, 23 insertions(+) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 04f87f4..5e5406f 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -10,3 +10,4 @@ DELIMITER_ACCONUT = ENV['DELIMITER_ACCONUT'] || 'compass-ci(a)qq.com' TMEP_GIT_BASE = '/c/public_git' DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" +RESULT_WEBDAV_URL = ENV['RESULT_WEBDAV_URL'] || "http://172.17.0.1:3080" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 90c3cce..60b64a1 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -113,5 +113,27 @@ module Utils return job end + + def parse_bisect_log(git_dir) + bisect_log = %x(git -C #{git_dir} bisect log) + bisect_log_arr = bisect_log.split(/\n/) + bisect_log_arr.keep_if { |item| item.start_with?('#') } + + return bisect_log_arr + end + + def create_bisect_log(job_id, git_dir) + log_file = File.join(git_dir, "#{job_id}_bisect.log") + log_content = parse_bisect_log(git_dir) + File.open(log_file, 'w') do |f| + log_content.each { |line| f.puts(line) } + end + + return log_file + end + + def upload_bisect_log(log_file, dest_dir, access_key) + %x(curl -sSf -T "#{log_file}" #{RESULT_WEBDAV_URL}/#{dest_dir} --cookie "ACCESSKEY=#{access_key}") + end end end -- 2.23.0

1 0

[PATCH v4 compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 09 Dec '20

09 Dec '20

generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..0f9390f --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_arch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

2 2

[PATCH v3 compass-ci 3/4] docker-rootfs: the function executed in host
by Wang Chenglong 09 Dec '20

09 Dec '20

the function is called by script on host. it contains get the password, download and load the docker image copy the image package to the host, create the links of vmlinuz and initrd. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 153 +++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100755 container/docker-rootfs/common diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common new file mode 100755 index 0000000..10ed266 --- /dev/null +++ b/container/docker-rootfs/common @@ -0,0 +1,153 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# This file is called by script run running on host +# define functions and set host environment variables + +check_cmd_input() +{ + ( [ $# -eq 2 ] && [ -f "$1" ] ) || { + usage + exit 1 + } + + check_docker_img_file "$1" + init_rootfs_dir "$2" + + export RUN_DIR=$(dirname $(realpath "$0")) +} + +usage() +{ + echo " + Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + + Example: + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + " +} + +check_docker_img_file() +{ + local allow_docker_img_suffix + local docker_img_name + local docker_img_suffix + + allow_docker_img_suffix=('tar.xz') + docker_img_name=$(basename "$1") + docker_img_suffix=${docker_img_name##*.} + + [ "$docker_img_suffix" == 'tar.xz' ] || { + docker_img_suffix=$(echo "$docker_img_name" |awk -F '.' '{print $(NF-1)"."$NF}') + echo "${allow_docker_img_suffix}" |grep -wq "$docker_img_suffix" || { + echo "[ERROR] Only support: .tar.xz file!" + exit 2 + } + } +} + +init_rootfs_dir() +{ + [ -d "$1" ] && return + + local limit_prompt_times + local current_prompt_time + + limit_prompt_times=3 + current_prompt_time=0 + + while true + do + read -r -p "[WARNING] Do you want to create \"$1\"? [y|n]> " if_create + + [ "$if_create" == 'y' ] && break + [ "$if_create" == 'n' ] && echo "[ERROR] User cancelled running." && exit + done + mkdir -p $1 +} + +check_passwd_file() +{ + export ROOT_NEW_PASSWD= + [ -f "$1" ] || { + echo "[INFO] No password file specified and root password kept." + return + } + + export ROOT_NEW_PASSWD=$(cat "$1") +} + +load_docker_img() +{ + echo "Loading docker image..." + docker_repos_tag=$(docker load -i $1) + docker_name_tag=${docker_repos_tag#*:} +} + +cp_rootfs() +{ + docker cp -a rootfs-docker:/tmp/$1 /tmp/ + cd $2 + zcat /tmp/$1 |cpio -idmv +} + +create_get_initrd() { + echo "Creating initrd.lkp via container/dracut-initrd..." + cd "$CCI_SRC/container/dracut-initrd" || { + echo "Failed to change into $CCI_SRC/container/dracut-initrd" + exit 6 + } + ./run "$ROOTFS_DIR/lib/modules/$ROOTFS_KERNEL" + + echo "Finding initrd.lkp under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 7 + } + local initrd_lkp + initrd_lkp=$(find ./boot -name "initramfs.lkp*") && export ROOTFS_INITRD_LKP=$initrd_lkp + + [ -f "$ROOTFS_INITRD_LKP" ] || { + echo "Failed to generate \"$ROOTFS_INITRD_LKP\"" + exit 8 + } + + echo + echo "initrd_lkp: $ROOTFS_INITRD_LKP" +} + +get_rootfs_kernel() { + echo "Finding vmlinuz under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 5 + } + local vmlinuz_file + local vmlinuz kernel + vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz + kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel + + echo "vmlinuz: $ROOTFS_VMLINUZ_FILE" + echo "kernel: $ROOTFS_KERNEL" +} + +create_links_vmlinuz_initrd() +{ + export ROOTFS_DIR=$1 + get_rootfs_kernel + create_get_initrd + + echo "Creating links to initrd.lkp and vmlinuz..." + cd $ROOTFS_DIR + ln -fs $ROOTFS_INITRD_LKP initrd.lkp + cd $ROOTFS_DIR/boot + ln -fs $ROOTFS_VMLINUZ vmlinuz + + echo "[INFO] Create links to initrd.lkp and vmlinuz success!" +} -- 2.23.0

2 4

[PATCH v4 compass-ci 4/4] docker-rootfs: list of packages that need to install
by Wang Chenglong 09 Dec '20

09 Dec '20

write the software in this file, it will install into rootfs. the kernel and kernel-devel is necessary. They provide the default kernel as well as kernel source code. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/packages-to-install | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 container/docker-rootfs/packages-to-install diff --git a/container/docker-rootfs/packages-to-install b/container/docker-rootfs/packages-to-install new file mode 100644 index 0000000..d7e890f --- /dev/null +++ b/container/docker-rootfs/packages-to-install @@ -0,0 +1,3 @@ +kernel +kernel-devel +openssh-server -- 2.23.0

1 0

[PATCH v4 compass-ci 2/4] docker-rootfs: setup rootfs in container
by Wang Chenglong 09 Dec '20

09 Dec '20

install software, set password and other actions in container. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/docker-rootfs/setup-image diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image new file mode 100755 index 0000000..8e2e88a --- /dev/null +++ b/container/docker-rootfs/setup-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +remove_file() +{ + rm -rf /.dockerenv +} + +setup_login() +{ + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + chmod 644 /tmp/${IMAGE_PACK} +} + +install_packages +remove_file +setup_login +pack_cgz -- 2.23.0

1 0

[PATCH compass-ci] container/es: fix failed to setting index
by Liu Yinsi 09 Dec '20

09 Dec '20

[why] when execute es/start, error message: curl: (56) Recv failure: Connection reset by peer because can't set index immediately after start es. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/start | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/es/start b/container/es/start index e43d01b..d84ad33 100755 --- a/container/es/start +++ b/container/es/start @@ -20,4 +20,7 @@ cmd=( "${cmd[@]}" +echo "Please wait 60s for setting index" +sleep 60 + set_es_indices -- 2.23.0

1 0

[PATCH v3 compass-ci 4/4] docker-rootfs: list of packages that need to install
by Wang Chenglong 09 Dec '20

09 Dec '20

write the software in this file, it will install into rootfs. the kernel and kernel-devel is necessary. They provide the default kernel as well as kernel source code. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/packages-to-install | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 container/docker-rootfs/packages-to-install diff --git a/container/docker-rootfs/packages-to-install b/container/docker-rootfs/packages-to-install new file mode 100644 index 0000000..d7e890f --- /dev/null +++ b/container/docker-rootfs/packages-to-install @@ -0,0 +1,3 @@ +kernel +kernel-devel +openssh-server -- 2.23.0

1 0