December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 2/4] docker-rootfs: setup rootfs in container
by Wang Chenglong 09 Dec '20

09 Dec '20

install software, set password and other actions in container. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/docker-rootfs/setup-image diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image new file mode 100755 index 0000000..ee5d721 --- /dev/null +++ b/container/docker-rootfs/setup-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum install -y --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +remove_file() +{ + rm -rf /.dockerenv +} + +setup_login() +{ + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + chmod 644 /tmp/${IMAGE_PACK} +} + +install_packages +remove_file +setup_login +pack_cgz -- 2.23.0

1 0

[PATCH v3 compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 09 Dec '20

09 Dec '20

generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..0f9390f --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_arch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

1 0

[PATCH v6 compass-ci] mail-robot: add check account exists for email
by Luan Shengde 09 Dec '20

09 Dec '20

avoid re-assign accounts for a single email address [why] the mail-robot may assign a new account if user send 'apply account' ail with the same email address. and this is a big waste of account resource. [how] when receive request for applying an account, check whether the email already has already been assigned an account. assign a new one if there is none return the old one if there is already one and update info mail robot will update infos if the mail has been set new: - my_name - my_commit_url - my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 39 ++++++++++++++++++----- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..0fff3b5 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -51,6 +51,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +63,7 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + apply_my_account store_account_info send_mail('') @@ -82,15 +82,38 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp + def read_my_account_es + accountx_es = ESQuery.new(index: 'accounts') + accountx_es.query_by_id(@my_info['my_email']) + end - @my_info['my_uuid'] = my_uuid + def apply_my_account + my_account_es = read_my_account_es + apply_info = {} + if my_account_es + my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') + apply_info.update my_account_es + apply_info.update @my_info + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + end + + def check_account_es + ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) end def store_account_info -- 2.23.0

1 0

[PATCH compass-ci] lib/es_query: fix FrozenError
by Lu Weitao 09 Dec '20

09 Dec '20

[Why] /home/lwt/work3/compass-ci/lib/es_query.rb:34:in `multi_field_query': can't modify frozen String (FrozenError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index c1342c5..a250eaf 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -31,7 +31,7 @@ class ESQuery } }, size: size } - @client.search index: @index << '*', body: query + @client.search index: @index + '*', body: query end def traverse_field(size) -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/cci-makepkg: centos7 update gcc in container
by Lin Jiaxin 09 Dec '20

09 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a7a990a5..0354af89 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -98,7 +98,7 @@ centos_update_gcc() { grep -sqF "CentOS Linux release 7" /etc/centos-release || return - [ "$os_mount" = "initramfs" ] && { + [ -f /.dockerenv -o "$os_mount" = "initramfs" ] && { yum install -y centos-release-scl-rh yum install -y "devtoolset-8-gcc*" } -- 2.23.0

2 1

[PATCH v2 compass-ci] result-webdav/nginx.conf: expand client_max_body_size
by Lin Jiaxin 09 Dec '20

09 Dec '20

error: curl:(22) The requested URL returned error: 413 Request Entity Too Large Our biggest cgz file is 656M now, so i set client_max_body_size to 700m. /srv/initrd/pkg% find ./ -size +100M |xargs ls -alhS -rw-r--r--. 1 lkp lkp 656M Nov 4 17:52 ./nfs/openeuler/aarch64/20.03/ycsb/0.15.0-0.cgz /srv/initrd/build-pkg% find ./ -size +100M |xargs ls -alhS -rw-rw-rw-. 1 lkp lkp 383M Aug 29 04:36 ./quake2world-data-git/HEAD.cgz Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- container/result-webdav/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 98382bd..1ccaa92 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -12,7 +12,7 @@ http { listen 3080; server_name "result-webdav"; server_tokens off; - client_max_body_size 100m; + client_max_body_size 700m; access_log /tmp/access.log; error_log /tmp/error.log; -- 2.23.0

3 2

[PATCH compass-ci] container/git-mirror: add user lkp
by Li Yuanchao 09 Dec '20

09 Dec '20

Add user lkp, so user can put pub_key of ssh in /home/lkp when they use ssh to clone. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 1180a7b..0828be5 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -5,6 +5,7 @@ FROM alpine:3.8 MAINTAINER Li Yuanchao <lyc163mail(a)163.com> ADD kill-parent.sh / +RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories -- 2.23.0

2 1

[PATCH v2 compass-ci 4/4] docker-rootfs: list of packages that need to install
by Wang Chenglong 09 Dec '20

09 Dec '20

write the software in this file, it will install into rootfs. the kernel and kernel-devel is necessary. They provide the default kernel as well as kernel source code. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/packages-to-install | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 container/docker-rootfs/packages-to-install diff --git a/container/docker-rootfs/packages-to-install b/container/docker-rootfs/packages-to-install new file mode 100644 index 0000000..d7e890f --- /dev/null +++ b/container/docker-rootfs/packages-to-install @@ -0,0 +1,3 @@ +kernel +kernel-devel +openssh-server -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] docker-rootfs: the function executed in host
by Wang Chenglong 09 Dec '20

09 Dec '20

the function is called by script on host. it contains get the password, download and load the docker image copy the image package to the host, create the links of vmlinuz and initrd. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 153 +++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100755 container/docker-rootfs/common diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common new file mode 100755 index 0000000..292784e --- /dev/null +++ b/container/docker-rootfs/common @@ -0,0 +1,153 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# This file is called by script run running on host +# define functions and set host environment variables + +check_cmd_input() +{ + ( [ $# -eq 2 ] && [ -f "$1" ] ) || { + usage + exit 1 + } + + check_docker_img_file "$1" + init_rootfs_dir "$2" + + export RUN_DIR=$(dirname $(realpath "$0")) +} + +usage() +{ + echo " + Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + + Example: + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + " +} + +check_docker_img_file() +{ + local allow_docker_img_suffix + local docker_img_name + local docker_img_suffix + + allow_docker_img_suffix=('tar.xz') + docker_img_name=$(basename "$1") + docker_img_suffix=${docker_img_name##*.} + + [ "$docker_img_suffix" == 'tar.xz' ] || { + docker_img_suffix=$(echo "$docker_img_name" |awk -F '.' '{print $(NF-1)"."$NF}') + echo "${allow_docker_img_suffix}" |grep -wq "$docker_img_suffix" || { + echo "[ERROR] Only support: .tar.xz file!" + exit 2 + } + } +} + +init_rootfs_dir() +{ + [ -d "$1" ] && return + + local limit_prompt_times + local current_prompt_time + + limit_prompt_times=3 + current_prompt_time=0 + + while true + do + read -r -p "[WARNING] Do you want to create \"$1\"? [y|n]> " if_create + + [ "$if_create" == 'y' ] && break + [ "$if_create" == 'n' ] && echo "[ERROR] User cancelled running." && exit + done + mkdir -p $1 +} + +check_passwd_file() +{ + export ROOT_NEW_PASSWD= + [ -f "$1" ] || { + echo "[INFO] No password file specified and root password kept." + return + } + + export ROOT_NEW_PASSWD=$(cat "$1") +} + +load_docker_img() +{ + echo "Loading docker image..." + docker_repos_tag=$(docker load -i $1) + docker_name_tag=${docker_repos_tag#*:} +} + +cp_rootfs() +{ + docker cp -a rootfs-docker:/tmp/$1 /tmp/ + cd $2 + zcat /tmp/$1 |cpio -idmv +} + +create_get_initrd() { + echo "Creating initrd.lkp via container/dracut-initrd..." + cd "$CCI_SRC/container/dracut-initrd" || { + echo "Failed to change into $CCI_SRC/container/dracut-initrd" + exit 6 + } + ./run "$ROOTFS_DIR/lib/modules/$ROOTFS_KERNEL" + + echo "Finding initrd.lkp under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 7 + } + local initrd_lkp + initrd_lkp=$(find ./boot -name "initramfs.lkp*") && export ROOTFS_INITRD_LKP=$initrd_lkp + + [ -f "$ROOTFS_INITRD_LKP" ] || { + echo "Failed to generate \"$ROOTFS_INITRD_LKP\"" + exit 8 + } + + echo + echo "initrd_lkp: $ROOTFS_INITRD_LKP" +} + +get_rootfs_kernel() { + echo "Finding vmlinuz under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 5 + } + local vmlinuz_file + local vmlinuz kernel + vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz + kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel + + echo "vmlinuz: $ROOTFS_VMLINUZ_FILE" + echo "kernel: $ROOTFS_KERNEL" +} + +create_links_vmlinuz_initrd() +{ + export ROOTFS_DIR=$1 + get_rootfs_kernel + create_get_initrd + + echo "Creating links to initrd.lkp and vmlinuz..." + cd $ROOTFS_DIR + ln -fs $ROOTFS_INITRD_LKP initrd.lkp + cd $ROOTFS_DIR/boot + ln -fs $ROOTFS_VMLINUZ vmlinuz + + echo "[INFO] Create links to initrd.lkp and vmlinuz success!" +} -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] docker-rootfs: setup rootfs in container
by Wang Chenglong 09 Dec '20

09 Dec '20

install software, set password and other actions in container. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/docker-rootfs/setup-image diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image new file mode 100755 index 0000000..c9cb5e1 --- /dev/null +++ b/container/docker-rootfs/setup-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +remove_file() +{ + rm -rf /.dockerenv +} + +setup_login() +{ + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + chmod 644 /tmp/${IMAGE_PACK} +} + +install_packages +remove_file +setup_login +pack_cgz -- 2.23.0

1 0