December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 09 Dec '20

09 Dec '20

generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..0f9390f --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_arch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-jobs-mapping.sh: add ss dynamic mapping
by Wang Yong 09 Dec '20

09 Dec '20

add ss field for linux, that can be used in auto build pkg Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- sbin/es-jobs-mapping.sh | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 8e19662..54b7e41 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -49,6 +49,15 @@ else } } }, + { + "ss": { + "path_match": "ss.*", + "mapping": { + "type": "keyword", + "enabled": true + } + } + }, { "default": { "match": "*", @@ -62,6 +71,10 @@ else } ], "properties": { + "ss": { + "dynamic": true, + "properties": {} + }, "pp": { "dynamic": true, "properties": {} -- 2.23.0

1 0

[PATCH compass-ci 3/4] docker-rootfs: the function executed in host
by Wang Chenglong 09 Dec '20

09 Dec '20

the function is called by script on host. it contains get the password, download and load the docker image copy the image package to the host, create the links of vmlinuz and initrd. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 148 +++++++++++++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100755 container/docker-rootfs/common diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common new file mode 100755 index 0000000..1d0e667 --- /dev/null +++ b/container/docker-rootfs/common @@ -0,0 +1,148 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# This file is called by script run running on host +# define functions and set host environment variables + +check_cmd_input() +{ + ( [ $# -eq 2 ] && [ -f "$1" ] ) || { + usage + exit 1 + } + + check_docker_img_file "$1" + init_rootfs_dir "$2" + + export RUN_DIR=$(dirname $(realpath "$0")) +} + +usage() +{ + echo " + Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + + Example: + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + " +} + +check_docker_img_file() +{ + local allow_docker_img_suffix docker_img_name docker_img_suffix + allow_docker_img_suffix=('tar.xz') + docker_img_name=$(basename "$1") + docker_img_suffix=${docker_img_name##*.} + [ "$docker_img_suffix" == 'tar.xz' ] || { + docker_img_suffix=$(echo "$docker_img_name" |awk -F '.' '{print $(NF-1)"."$NF}') + echo "${allow_docker_img_suffix}" |grep -wq "$docker_img_suffix" || { + echo "[ERROR] Only support: .tar.xz file!" + exit 2 + } + } +} + +init_rootfs_dir() +{ + [ -d "$1" ] && return + + local limit_prompt_times current_prompt_time + limit_prompt_times=3 + current_prompt_time=0 + while true + do + read -r -p "[WARNING] Do you want to create \"$1\"? [y|n]> " if_create + + [ "$if_create" == 'y' ] && break + [ "$if_create" == 'n' ] && echo "[ERROR] User cancelled running." && exit + done + + mkdir -p $1 +} + + +check_passwd_file() +{ + export ROOT_NEW_PASSWD= + [ -f "$1" ] || { + echo "[INFO] No password file specified and root password kept." + return + } + + export ROOT_NEW_PASSWD=$(cat "$1") +} + +load_docker_img() +{ + echo "Loading docker image..." + docker_repos_tag=$(docker load -i $1) + docker_name_tag=${docker_repos_tag#*:} +} + +cp_rootfs() +{ + docker cp -a rootfs-docker:/tmp/$1 /tmp/ + cd $2 + zcat /tmp/$1 |cpio -idmv + +} + +create_get_initrd() { + echo "Creating initrd.lkp via container/dracut-initrd..." + cd "$CCI_SRC/container/dracut-initrd" || { + echo "Failed to change into $CCI_SRC/container/dracut-initrd" + exit 6 + } + ./run "$ROOTFS_DIR/lib/modules/$ROOTFS_KERNEL" + + echo "Finding initrd.lkp under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 7 + } + local initrd_lkp + initrd_lkp=$(find ./boot -name "initramfs.lkp*") && export ROOTFS_INITRD_LKP=$initrd_lkp + + [ -f "$ROOTFS_INITRD_LKP" ] || { + echo "Failed to generate \"$ROOTFS_INITRD_LKP\"" + exit 8 + } + + echo + echo "initrd_lkp: $ROOTFS_INITRD_LKP" +} + +get_rootfs_kernel() { + echo "Finding vmlinuz under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 5 + } + local vmlinuz_file vmlinuz kernel + vmlinuz_file=$(find ./boot -name "vmlinuz-*" |grep -v rescue) && export ROOTFS_VMLINUZ_FILE=$vmlinuz_file + vmlinuz=$(basename "$vmlinuz_file") && export ROOTFS_VMLINUZ=$vmlinuz + kernel=${vmlinuz:8} && export ROOTFS_KERNEL=$kernel + + echo "vmlinuz: $ROOTFS_VMLINUZ_FILE" + echo "kernel: $ROOTFS_KERNEL" +} + +create_links_vmlinuz_initrd() +{ + export ROOTFS_DIR=$1 + get_rootfs_kernel + create_get_initrd + + echo "Creating links to initrd.lkp and vmlinuz..." + cd $ROOTFS_DIR + ln -fs $ROOTFS_INITRD_LKP initrd.lkp + cd $ROOTFS_DIR/boot + ln -fs $ROOTFS_VMLINUZ vmlinuz + + echo "[INFO] Create links to initrd.lkp and vmlinuz success!" +} -- 2.23.0

4 6

[PATCH v2 lkp-tests 3/3] Automatic compilation and configure the variables
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- install.sh | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 install.sh diff --git a/install.sh b/install.sh new file mode 100644 index 00000000..4bcc1afb --- /dev/null +++ b/install.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +. lib/env.sh + +make + +source $(shell_profile) -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] sbin/install-dependencies.sh: modify the way to add ruby env-variables
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 7df3652f..3436ee44 100644 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,6 +1,7 @@ #!/bin/bash . lib/env.sh +. sbin/set-env.sh # choose install function base on # DISTRIBUTION @@ -30,8 +31,7 @@ esac mac_dep() { brew install ruby - echo "export PATH=/usr/local/opt/ruby/bin:$PATH" >> $(shell_profile) - source $(shell_profile) + write_shell_profile "export PATH=/usr/local/opt/ruby/bin:$PATH" gem install git activesupport rest-client } -- 2.23.0

1 0

[PATCH compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 09 Dec '20

09 Dec '20

generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..de9c3a7 --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_arch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +#docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

4 6

[PATCH compass-ci] lib/json_logger: fix initializate without env
by Wu Zhende 09 Dec '20

09 Dec '20

[Error] In lib/json_logger.cr:9:61 9 | def initialize(logdev = STDOUT, formatter = my_formatter, @env = nil) Error: instance variable @env of JSONLogger was inferred to be Nil, but Nil alone provides no information This error is reported when "JSONLogger.new" is used. Fix it. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index 9596243..dd87020 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -4,6 +4,9 @@ require "logger" require "json" require "any_merge" +require "kemal" + +add_context_storage_type(Time::Span) class JSONLogger < Logger def initialize(logdev = STDOUT, formatter = my_formatter, @env = nil) -- 2.23.0

1 0

[PATCH compass-ci 2/4] docker-rootfs: setup rootfs in container
by Wang Chenglong 09 Dec '20

09 Dec '20

install software, set password and other actions in container. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/docker-rootfs/setup-image diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image new file mode 100755 index 0000000..0fa1eb6 --- /dev/null +++ b/container/docker-rootfs/setup-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +remove_file() +{ + rm -rf /.dockerenv +} + +setup_login() +{ + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + chmod 644 /tmp/${IMAGE_PACK} +} + +install_packages +remove_file +setup_login +pack_cgz -- 2.23.0

2 2

[PATCH compass-ci] container/kibana: optimize logtrail
by Wu Zhende 09 Dec '20

09 Dec '20

1. show all ES index 2. set the maximum number of logs to be displayed to 10000 3. set the display of logs generated within 30 days Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/logtrail.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/kibana/logtrail.json b/container/kibana/logtrail.json index 304a4b3..c63f2ac 100644 --- a/container/kibana/logtrail.json +++ b/container/kibana/logtrail.json @@ -2,7 +2,7 @@ "version": 2, "index_patterns": [{ "es": { - "default_index": "scheduler-3016", + "default_index": "*", "allow_url_parameter": false, "timezone": "CST" }, @@ -12,9 +12,9 @@ "display_timestamp_format": "YYYY MMM DD HH:mm:ss", "max_buckets": 500, "nested_objects": false, - "default_time_range_in_days": 0, + "default_time_range_in_days": 30, "max_hosts": 100, - "max_events_to_keep_in_viewer": 5000, + "max_events_to_keep_in_viewer": 10000, "default_search": "", "fields": { "mapping": { -- 2.23.0

1 0

[PATCH lkp-tests 4/5] sbin/set-env.sh: take apart the method of adding env-variables
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/set-env.sh | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/sbin/set-env.sh b/sbin/set-env.sh index a8b161fa..ce1ad427 100644 --- a/sbin/set-env.sh +++ b/sbin/set-env.sh @@ -2,10 +2,17 @@ . lib/env.sh -write_shellrc() +submit_env() +{ + echo $@ >> $(shell_profile) + source $(shell_profile) +} + +write_env() { echo "export LKP_SRC=$PWD" >> $(shell_profile) echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $(shell_profile) + source $(shell_profile) } write_host() @@ -28,5 +35,5 @@ write_host() EOF } -write_shellrc +write_env write_host -- 2.23.0

4 3