- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/2] sparrow/4-docker/buildall: check whether skip ssh-r
by Liu Yinsi 01 Mar '21

01 Mar '21

[why] for user locally deploy compass-ci, ssh-r is not exists, so check when ssh-r already exists, skip ssh-r. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index bb77a1f..c9edbe9 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,7 +56,8 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && exit + docker ps | grep -wq "ssh_r" + [ "$?" == "0" ] && [ "$container_name" == 'ssh-r' ] && exit [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run -- 2.23.0

2 2

[PATCH v3 compass-ci] container/assign-account: optimize config login permission
by Luan Shengde 01 Mar '21

01 Mar '21

optimize account login permission setting: before: set enable login: has key: enable_login key value: true set disable login has key: enable_login key value: false or has no key: enable_login now: set enable login: has key: enable_login key value: true set disable login: has key: enable_login key value: false keep the default login permission: has no key: enable_login Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 7e0a5a7..2922f81 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -136,7 +136,7 @@ class AccountStorage def permit_login_config(login_name) if @data.key?('enable_login') && @data['enable_login'] %x(usermod -s /usr/bin/zsh #{login_name}) - else + elsif @data.key?('enable_login') && !@data['enable_login'] %x(usermod -s /sbin/nologin #{login_name}) end end -- 2.23.0

1 0

[PATCH compass-ci] add check runtime upper limit
by Luan Shengde 01 Mar '21

01 Mar '21

add time limit for borrowing machine. the max runtime is limited to no more than 10 days. case runtime/sleep beyond the limit or both of them was set empty value. it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 52965bc..63ec8fb 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -409,6 +410,29 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time(runtime/sleep) cannot exceed 10 days. Consider re-borrow.\n" + + if @hash["pp"]["sleep"].as_i? + sleep_run_time = @hash["pp"]["sleep"] + elsif @hash["pp"]["sleep"].as_h? + sleep_run_time = @hash["pp"]["sleep"]["runtime"] + else + notice_msg = "\nPlease set runtime/sleep first for the job yaml and retry." + raise notice_msg + end + + raise error_msg if sleep_run_time.as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH compass-ci] container/scheduler: install util-linux to use uuidgen
by Wu Zhende 26 Feb '21

26 Feb '21

submit job.yaml in scheduler docker needs command "uuidgen" to generate submit_id Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/scheduler/Dockerfile b/container/scheduler/Dockerfile index 0ad813c..7fb8e35 100644 --- a/container/scheduler/Dockerfile +++ b/container/scheduler/Dockerfile @@ -8,7 +8,7 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN apk update RUN apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' 'util-linux' RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ -- 2.23.0

1 0

[PATCH compass-ci] lib/es_jobs: remove unnecessary null in raw.stats
by Wang Chenglong 26 Feb '21

26 Feb '21

before: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: [null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,"{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found\"}", null,null,null,null,null,null,null,null,null,null,null] after: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: ["{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped ifno files are found\"}"] Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/es_jobs.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index 2d2f3bf..c5c0f0d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -104,7 +104,7 @@ class ESJobs def set_default_value(result, stats, metrics) left_metrics = metrics - stats.keys - left_metrics.each { |metric| result['raw.stats'][metric] << nil } + left_metrics.each { |metric| result['raw.stats'][metric] } stats.each do |key, value| result['raw.stats'][key] << value -- 2.23.0

1 0

[PATCH compass-ci] job.cr: add has_key? function in Job class
by Cao Xueliang 26 Feb '21

26 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 4 ++++ src/scheduler/find_job_boot.cr | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..52965bc 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -332,6 +332,10 @@ class Job @hash.[key]? end + def has_key?(key : String) + @hash.has_key?(key) + end + def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" raise "Should not use []= update #{key}, use update_#{key}" diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 88db69e..8dd22a8 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,7 +160,7 @@ class Sched end private def get_boot_ipxe(job : Job) - return job["ipxe_response"] if job["suite"] == "install-iso" && job["ipxe_response"]? + return job["ipxe_response"] if job["suite"] == "install-iso" && job.has_key?("ipxe_response") response = "#!ipxe\n\n" -- 2.23.0

1 0

[PATCH v8 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 26 Feb '21

26 Feb '21

add time limit for borrowing machine. the max runtime is limited to no more than 10 days. case runtime/sleep beyond the limit or both of them was set empty value. it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..91d28fa 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -405,6 +406,29 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time(runtime/sleep) cannot exceed 10 days. Consider re-borrow.\n" + + if @hash["pp"]["sleep"].as_i? + sleep_run_time = @hash["pp"]["sleep"] + elsif @hash["pp"]["sleep"].as_h? + sleep_run_time = @hash["pp"]["sleep"]["runtime"] + else + notice_msg = "\nPlease set runtime/sleep first for the job yaml and retry." + raise notice_msg + end + + raise error_msg if sleep_run_time.as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH lkp-tests] bin/run-lkp: reduce unnecessary ucode output
by Wang Yong 25 Feb '21

25 Feb '21

reduce output when $ucode not exist Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- bin/run-lkp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/run-lkp b/bin/run-lkp index b422a7eb4..ff15b7c62 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -29,7 +29,7 @@ setup_result_root || { job_done; exit 1; } echo run-job $job set_job_state 'running' -echo "target ucode: $ucode" +[ -n "${ucode}" ] && echo "target ucode: $ucode" if [ -n "$ucode" ] && ! is_same_as_current_ucode_version "$ucode"; then echo "microcode is not matched, skip testing..." set_job_state 'microcode_is_not_matched' -- 2.23.0

2 2

git count
by Wu Fengguang 25 Feb '21

25 Feb '21

Yinsi is our code star, thanks Yinsi! wfg@crystal ~/todo/bin% ./git-count 25 41 Liu Yinsi 26 Luan Shengde 23 Cao Xueliang 17 Wei Jihui 17 Yu Chuan 16 Wang Yong 14 Li Yuanchao 13 Lu Weitao 11 Xu Xijian 10 Bai Jing 9 Xiao Shenwei 9 Cui Yili 9 Li Ping 8 Lin Jiaxin 6 Wu Zhende 5 Liu Shaofei 5 Wu Fengguang 3 Wang Chenglong 2 Zhang Dewan 0 Tong Qunfeng 0 Hu Xuejiao 0 Du Kaitian 0 Sun Yukui 0 Zhang Yale 0 Zhang Yu 0 Ren Wen 0 Lu Kaiyi 0 Shi Zhichao

1 0

review star
by Wu Fengguang 25 Feb '21

25 Feb '21

Jihui is our review star, thanks Jihui! Review email records in the last 7 days total review paragraph point people 84 40 125 44 Wu Fengguang 58 12 15 17 Wei Jihui 72 8 13 15 Cao Xueliang 33 5 18 7 Luan Shengde 7 2 8 6 Wang Yong 44 4 11 4 Yu Chuan 9 1 4 3 Xu Xijian 16 0 0 2 Li Yuanchao 17 1 0 1 Bai Jing 23 2 2 1 Li Ping 6 0 0 0 Liu Yinsi 3 1 1 0 Lin Jiaxin 2 0 0 0 Zhang Dewan 15 0 0 0 Cui Yili 6 0 0 0 Lu Weitao 22 1 2 0 Wang Chenglong

1 0

[PATCH compass-ci 1/3] sched: response ipxe_response value if ipxe_response filed in job yaml
by Cao Xueliang 25 Feb '21

25 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9e2898e..e8d590b 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,6 +160,8 @@ class Sched end private def get_boot_ipxe(job : Job) + return job["ipxe_response"] if job["ipxe_response"]? + response = "#!ipxe\n\n" _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } -- 2.23.0

2 3

[PATCH v7 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 25 Feb '21

25 Feb '21

add limit for borrowing machine, the max runtime is limited to no more than 10 days. case the runtime beyond the limit, it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..db1af2b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -405,6 +406,20 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time cannot exceed 10 days. Consider re-borrow.\n" + + raise error_msg if @hash["pp"]["sleep"]["runtime"].as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

3 4

[PATCH lkp-tests] distro/*: reduce Package Manager output
by Wang Yong 25 Feb '21

25 Feb '21

reduce unnecessary output to get effective information ref: commit 16cef539c517d0e6bd3495a1e3079214a8b4bd24 yum/dnf: -q, --quiet ref: https://linux.die.net/man/8/yum ref: https://www.man7.org/linux/man-pages/man8/dnf.8.html apt-get: -q, --quiet Quiet. Produces output suitable for logging, omitting progress indicators. More q's will produce more quiet up to a maximum of two. Note that quiet level 2 implies -y. ref: https://linux.die.net/man/8/apt-get pacman: -q, --quiet ref: https://archlinux.org/pacman/pacman.8.html zypper: --quiet, -q ref: https://en.opensuse.org/SDB:Zypper_usage Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/aliyun | 24 ++++++++++++------------ distro/archlinux | 11 ++++++----- distro/debian | 15 ++++++++------- distro/installer/archlinux | 3 ++- distro/installer/centos | 8 ++++---- distro/installer/debian | 5 +++-- distro/installer/fedora | 2 +- distro/installer/opensuse | 2 +- distro/installer/oracle | 6 +++--- distro/installer/ubuntu | 2 +- 10 files changed, 41 insertions(+), 37 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 83727ce32..83f058c33 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -9,17 +9,17 @@ shopt -s nullglob support_repo_epel() { if ! rpm -q epel-release >/dev/null; then - yum install -y epel-release + yum install -yq epel-release fi } update_yum_into_dnf() { if ! rpm -q dnf >/dev/null; then - yum install -y dnf + yum install -yq dnf ln -sf /usr/bin/dnf /usr/bin/yum else - dnf makecache timer + dnf makecache timer >/dev/null fi } @@ -46,7 +46,7 @@ distro_install_depends() for pkg in $packages do - rpm --quiet --query $pkg || yum install -y $pkg + rpm --quiet --query $pkg || yum install -yq $pkg done done } @@ -73,8 +73,8 @@ yum_download() [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -y --downloadonly $*" - yum install -y --downloadonly "$@" + echo "yum install -yq --downloadonly $*" + yum install -yq --downloadonly "$@" } yum_pack() @@ -137,9 +137,9 @@ install_gcc7() wget -O /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm https://rpmfind.net/linux/centos/7/extras/x86_64/Packages/centos-release-sc… rpm -ivh --ignoresize /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm else - yum install -y centos-release-scl-rh + yum install -yq centos-release-scl-rh fi - yum install -y "devtoolset-7-gcc*" + yum install -yq "devtoolset-7-gcc*" source /opt/rh/devtoolset-7/enable } @@ -148,7 +148,7 @@ download_ocfs2_tools() local pkgs=$1 mkdir -p /opt/rpms - yum install -y wget + yum install -yq wget wget -nc -P /opt/rpms https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/ocfs2-… @@ -160,7 +160,7 @@ download_ocfs2_tools() fi # Try to download the dependences of ocfs2-tools/ocfs2-tools-devel - yum install -y --downloadonly "$download_pkgs" + yum install -yq --downloadonly "$download_pkgs" } install_ocfs2_tools() @@ -169,8 +169,8 @@ install_ocfs2_tools() download_ocfs2_tools "$pkgs" - yum install -y /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm + yum install -yq /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm echo "$pkgs" | grep -q 'ocfs2-tools-devel' && \ - yum install -y /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm + yum install -yq /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm } diff --git a/distro/archlinux b/distro/archlinux index 59c8da1d6..dd4b2331c 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -43,8 +43,8 @@ find_pacman_package() pacman_download() { - echo "pacman -Sw --noconfirm $*" - pacman -Sw --noconfirm "$@" + echo "pacman -Swq --noconfirm $*" + pacman -Swq --noconfirm "$@" } pacman_pack() @@ -88,7 +88,8 @@ distro_install_depends() for pkg in $packages do - pacman -Qq --noconfirm $pkg > /dev/null || pacman -Sy --noconfirm $pkg + pacman -Qq --noconfirm $pkg > /dev/null || pacman -Syq --noconfirm --needed $pkg >/tmp/pacman_info 2>&1 + [ -f /tmp/pacman_info ] && grep -v "warning: could not get file information for *" /tmp/pacman_info done done } @@ -115,13 +116,13 @@ add_aur_helper() { echo "lkp ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers - pacman -Q yay 2>/dev/null && return + pacman -Qq yay >/dev/null && return local user=lkp local dir=/home/${user} mkdir -p ${dir} && chown ${user}:${user} ${dir} - pacman -S --needed --noconfirm git base-devel 2>/dev/null + pacman -Sq --needed --noconfirm git base-devel 2>/dev/null su - ${user} -c "git clone https://aur.archlinux.org/yay.git" su - ${user} -c "cd yay && makepkg -si --needed --noconfirm" } diff --git a/distro/debian b/distro/debian index 21dafe21b..a1642cc1f 100755 --- a/distro/debian +++ b/distro/debian @@ -8,7 +8,7 @@ shopt -s nullglob update() { - apt-get update + apt-get -qq update } add_i386_package() @@ -18,8 +18,8 @@ add_i386_package() download() { - echo apt-get download $* - apt-get download $* + echo apt-get -qq download $* + apt-get -qq download $* } hardlink_to_symlink() @@ -128,7 +128,7 @@ resolve_depends() local deps=$* local command - command="apt-get --simulate --no-install-recommends install $deps" + command="apt-get --simulate --no-install-recommends install -qq $deps" # using "bash -c" to support wildcard in apt-get. # if use wildcard in the package version variable, # apt-get will treat the wildcard as a normal character. @@ -193,7 +193,8 @@ distro_install_depends() do $chronic apt-get -o Dpkg::Options::=--force-confdef \ -o Dpkg::Options::=--force-confold \ - install -y --no-upgrade $package + install -qq --no-upgrade $package >/tmp/apt-get_info 2>&1 + grep -v "dpkg: warning: files list file for package '.*' missing;" /tmp/apt-get_info done done } @@ -225,7 +226,7 @@ fixup_distro_mirror() fixup_preinstall() { - apt-get install -yf + apt-get install -qqf } fixup_arch_install() @@ -235,7 +236,7 @@ fixup_arch_install() dpkg --add-architecture i386 update - apt -y --allow-remove-essential --fix-broken install -o=APT::Architecture="i386" + apt --allow-remove-essential --fix-broken install -qq -o=APT::Architecture="i386" } pack_benchmark_deps() diff --git a/distro/installer/archlinux b/distro/installer/archlinux index 3be9e7953..81ca82f73 100755 --- a/distro/installer/archlinux +++ b/distro/installer/archlinux @@ -13,4 +13,5 @@ add_aur_helper regular_user=$(grep 1090 /etc/passwd | awk -F':' '{ print $1 }') -sudo -u $regular_user yay -S --needed --noconfirm $* +sudo -u $regular_user yay -Sq --needed --noconfirm $* >/tmp/pacman_info 2>&1 +[ -f /tmp/pacman_info ] && grep -v "warning: could not get file information for *" /tmp/pacman_info diff --git a/distro/installer/centos b/distro/installer/centos index d9ad60038..704d3e3cc 100755 --- a/distro/installer/centos +++ b/distro/installer/centos @@ -1,12 +1,12 @@ #!/bin/bash if ! rpm -q epel-release >/dev/null; then - [ ${DISTRO^^} == "OPENEULER" ] || yum install -y epel-release + [ ${DISTRO^^} == "OPENEULER" ] || yum install -yq epel-release if [ ${DISTRO^^} == "OPENEULER" ]; then - yum makecache timer + yum makecache timer >/dev/null else - yum makecache fast + yum makecache fast >/dev/null fi fi -yum install -y $* +yum install -yq $* diff --git a/distro/installer/debian b/distro/installer/debian index 35b3209bd..8bf166458 100755 --- a/distro/installer/debian +++ b/distro/installer/debian @@ -2,7 +2,8 @@ # Debian package installation export DEBIAN_FRONTEND=noninteractive -dpkg --add-architecture i386 && apt-get update +dpkg --add-architecture i386 && apt-get -qq update apt-get -o Dpkg::Options::="--force-confdef" \ -o Dpkg::Options::="--force-confold" \ - -y install $* >/tmp/apt-get_info 2>&1 + -qq install $* +grep -v "dpkg: warning: files list file for package '.*' missing;" /tmp/apt-get_info diff --git a/distro/installer/fedora b/distro/installer/fedora index 1da708c2a..61176b361 100755 --- a/distro/installer/fedora +++ b/distro/installer/fedora @@ -1,3 +1,3 @@ #!/bin/sh -dnf -y install $* +dnf -yq install $* diff --git a/distro/installer/opensuse b/distro/installer/opensuse index cbdbf4eff..392d77b4a 100755 --- a/distro/installer/opensuse +++ b/distro/installer/opensuse @@ -1,3 +1,3 @@ #!/bin/bash -zypper install -y $* +zypper -q install -y $* diff --git a/distro/installer/oracle b/distro/installer/oracle index 11df716d6..c6881070c 100755 --- a/distro/installer/oracle +++ b/distro/installer/oracle @@ -1,8 +1,8 @@ #!/bin/sh if ! rpm -q epel-release >/dev/null; then - yum install -y epel-release - yum makecache fast + yum install -yq epel-release + yum makecache fast >/dev/null fi -yum install -y $* +yum install -yq $* diff --git a/distro/installer/ubuntu b/distro/installer/ubuntu index 541ba74f6..0ce801a79 100755 --- a/distro/installer/ubuntu +++ b/distro/installer/ubuntu @@ -6,4 +6,4 @@ dpkg --add-architecture i386 && apt-get update apt-get -o Dpkg::Options::="--force-confdef" \ -o Dpkg::Options::="--force-confold" \ -o Dpkg::Options::="--force-overwrite" \ - -y install $* + -qq install $* -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/compare_matrixes.rb: fix rubocop warning
by Lu Weitao 25 Feb '21

25 Feb '21

[Why] /home/lwt/compass-ci/lib/compare_matrixes.rb:586:14: C: Style/StringLiterals: Prefer single-quoted strings when you don't need string interpolation or special symbols. return " < 1% " if stddev_percent < 1 Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d004099..6700126 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -584,7 +584,7 @@ def format_stddev_percent(stddev_percent, average_width) percent_width = SUB_LONG_COLUMN_WIDTH - average_width if stddev_percent if stddev_percent != 0 - return " < 1% " if stddev_percent < 1 + return ' < 1% ' if stddev_percent < 1 percent_str = get_suitable_number_str( stddev_percent.abs, -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/compare_matrixes.rb skip .message|.log metric
by Lu Weitao 25 Feb '21

25 Feb '21

[Why] the metric end with .message|.log, no need to compare Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index ffc6500..d004099 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -163,6 +163,7 @@ def get_matrixes_values(matrixes_list, latest_jobs, options) matrixes_size = get_matrixes_size(matrixes_list) get_matrixes_fields(matrixes_list).each do |field| next if field == 'stats_source' + next if field.end_with?('.message', '.log') success = successful?(field) next unless success || latest_failure?(field, latest_jobs) -- 2.23.0

1 0

[PATCH v5 lkp-tests] jobs: add os-matrix-*.yaml
by Li Ping 25 Feb '21

25 Feb '21

1 0

[PATCH lkp-tests] pkg/mysql: change mysql default port
by Wei Jihui 25 Feb '21

25 Feb '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/mysql/PKGBUILD | 2 +- pkg/mysql/my.cnf | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD index 0f7d8bc8..977ad47e 100644 --- a/pkg/mysql/PKGBUILD +++ b/pkg/mysql/PKGBUILD @@ -9,7 +9,7 @@ source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" "https://github.com/thkukuk/rpcsvc-proto/releases/download/v1.4/rpcsvc-proto…" "my.cnf") -md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '7bb0372ec4e9281813a9afc67c6cfc20') +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '6d545f2b9a5ef6989c675063a328e407') prepare() { diff --git a/pkg/mysql/my.cnf b/pkg/mysql/my.cnf index 9732af40..acca75ae 100644 --- a/pkg/mysql/my.cnf +++ b/pkg/mysql/my.cnf @@ -13,7 +13,7 @@ socket=/data/mysql/run/mysql.sock tmpdir=/data/mysql/tmp datadir=/data/mysql/data default_authentication_plugin=mysql_native_password -port=3309 +port=3306 user=mysql memlock=on bind-address=0.0.0.0 -- 2.23.0

3 2

[PATCH lkp-tests] daemon: add mysql
by Wei Jihui 25 Feb '21

25 Feb '21

daemon/mysql is for start mysql server Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/mysql | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100755 daemon/mysql diff --git a/daemon/mysql b/daemon/mysql new file mode 100755 index 00000000..0291d3e9 --- /dev/null +++ b/daemon/mysql @@ -0,0 +1,104 @@ +#!/bin/sh +# - nic +# - device +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=$server}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=mysql_password}" + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$device" + mount /dev/"$device" /data + + mkdir -p /data/mysql/data /data/mysql/share /data/mysql/tmp /data/mysql/run /data/mysql/log + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +start_mysql_server() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "$mysql is not installed!" >&2 && exit 1 + fi + + "$mysqld" --user=mysql --basedir=/data/mysql --datadir=/data/mysql/data/ --initialize + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +disable_irqbalance() +{ + systemctl stop firewalld + iptables -F + pkill -9 irqbalance + systemctl start tuned +} + +intrrupt_nic_binding() +{ + local core_ranges='55-79 93-95' + ethtool -L $nic combined 28 + for core_range in $core_ranges + do + $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic + done + ethtool -l $nic +} + +net_io_cache_setting() +{ + echo deadline > /sys/block/"$device"/queue/scheduler + echo 2048 >/sys/block/"$device"/queue/nr_requests +} + +getcpu_process_bind_core() +{ + cd /getcpu || exit + insmod getcpu-hi.ko + numactl -C 0-92 -i 0-3 "mysqld" --defaults-file=/etc/my.cnf & + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + create_basedir + start_mysql_server + create_database + disable_irqbalance + intrrupt_nic_binding + net_io_cache_setting + getcpu_process_bind_core +} +main -- 2.23.0

2 1

[PATCH lkp-tests 2/3] tests/sysbench-mysql: skip clean up data
by Wei Jihui 25 Feb '21

25 Feb '21

comment out unnecessary opration: cleanup data Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/sysbench-mysql | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 1d424aaa..5dd8686f 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -85,9 +85,13 @@ run_sysbench_step() run_sysbench_mysql() { - run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + echo "data is being prepared, this will take some time..." + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare > /dev/null + run_sysbench_step oltp.lua ${args2[@]} run - run_sysbench_step oltp.lua ${args2[@]} cleanup + + # clean up test data will take some time, this step can be skipped + # run_sysbench_step oltp.lua ${args2[@]} cleanup > /dev/null } stop_firewalld -- 2.23.0

2 1

[PATCH lkp-tests 1/3] jobs/sysbench-mysql.yaml: udpate test args
by Wei Jihui 25 Feb '21

25 Feb '21

1. change cluster to cs-s1-a102-c1 2. add kernel args 3. add mysql args 4. add sysctl for performance optimized 5. add server nic and device, this can be sured by cluster Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/sysbench-mysql.yaml | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml index 83795a18..607acf12 100644 --- a/jobs/sysbench-mysql.yaml +++ b/jobs/sysbench-mysql.yaml @@ -1,10 +1,35 @@ suite: sysbench-mysql category: benchmark -cluster: cs-lkp-hsw-ep5 +cluster: cs-s1-a102-c1 +kernel_version: 4.19.90-mysql +kernel_custom_params: sched_steal_node_limit=8 + +mysql_user: +mysql_host: +mysql_port: +mysql_db: +mysql_password: if role server: + sysctl: + sc_numa_balancing: 0 + sc_sched_autogroup_enabled: 0 + sc_sched_min_granularity_ns: 10000000 + sc_sched_wakeup_granularity_ns: 15000000 + sc_threads-max: 52130 + sc_tcp_max_syn_backlog: 360000 + sc_dirty_ratio: 5 + sc_somaxconn: 1024 + sc_rmem_max: 16777216 + sc_wmem_max: 16777216 + sc_swappiness: 1 + sc_ipv4_tcp_rmem: 4096 87380 16777216 + sc_ipv4_tcp_wmem: 4096 87380 16777216 + sc_sched_features: "NO_WAKEUP_PREEMPTION STEAL" mysql: + nic: enp189s0f1 + device: nvme0n1 if role client: sysbench-mysql: @@ -17,6 +42,6 @@ if role client: runtime: 600 rand_type: special rand_spec_pct: 100 - oltp_read_only: off + oltp_read_only: false report_interval: 1 events: 2000000000 -- 2.23.0

2 1

[PATCH v3 compass-ci] doc/stats-rule.md
by Wang Chenglong 25 Feb '21

25 Feb '21

[Why] The rules to follow when writing the stats script. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/stats-rule.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 doc/stats-rule.md diff --git a/doc/stats-rule.md b/doc/stats-rule.md new file mode 100644 index 0000000..eaa2586 --- /dev/null +++ b/doc/stats-rule.md @@ -0,0 +1,47 @@ +${LKP_SERVER}/stats/${script} stat�� +========================================== + +## ��es�� + +1.��error_id�� + - original: {"changed": false, "failures": ["No package yum-plugin-priorities available."], "msg": "Failed to + install some of the specified packages", "rc": 1, "results": []} + + - error_id: error.No-package-yum-plugin-priorities-available: 1 + +2.�� + - ��error_id�� + # �� + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + end + + - ��jobs��'��'��es��error_id.��stats/${script}�� + +3.��error message�� + # original error message + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + end + +4.��error_id�� + - ��'-'�� + - ��'#'. + - ��'-',��. + + def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/-+/, '-') # ��'-', �� + line.gsub!(/\b-[a-z][0-9]\.[0-9]\.[0-9]+/, "#") # dgraph-linux-amd64-v0.8.2.tar.gz + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/{%/, ' ') + line.gsub!(/%}/, ' ') + line + end -- 2.23.0

2 1

[PATCH compass-ci] container/dracut-initrd: add mke2fs to module 98dracut-systemd
by Xu Xijian 25 Feb '21

25 Feb '21

1. align cmd "inst /usr/bin/awk" when insert it into module 40network/module-setup.sh. 2. add mke2fs to module 98dracut-systemd, it's used to make filesystem type. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 8f03fa8..849480d 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -16,4 +16,5 @@ rm -rf /var/lib/apt/lists/* cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh -sed -i "/install() {/ainst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh +sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh +sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh -- 2.23.0

1 0

[PATCH v3 lkp-tests] jobs: add os-matrix-arch.yaml
by Li Ping 25 Feb '21

25 Feb '21

1 0

[PATCH v5 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 25 Feb '21

25 Feb '21

[How] ansible_stats_openeuler < ${job_output} wcl@crystal ~/lkp-tests/stats% ./ansible_test < /srv/result/ansible_test/2021-02-04/ vm-2p16g--wcl/openeuler-20.03-aarch64/jgeusebroek-ansible-role-docker/crystal.974042/ansible_test error.Failed-to-download-metadata-for-repo-dockerrepo: 1 error.Failed-to-download-metadata-for-repo-dockerrepo.message: {"ansible_loop_var": "item", "changed": false, "item": "docker-engine", "msg": "Failed to download metadata for repo 'dockerrepo': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 138 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 127 insertions(+), 11 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..67e312f47 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,131 @@ #!/usr/bin/env ruby -while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line +require 'json' + +def output_error(error_msg) + error_id = common_error_id error_msg + puts "error.#{error_id}: 1" + puts "error.#{error_id}.message: #{@ansible_failed_info}" +end + +def parse_msg_1(ansible_failed_json) + case ansible_failed_json.chomp + + # When running some plug-in, the file was not found + when /(No file was found when using (.+))\./ + output_error $1 + + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + + # The task includes an option with an undefined variable + # {"msg": "The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined + # The error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, + # but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: + # Define mysql_packages. here"} + when /(.+ is undefined)/ + output_error $1 + + # {"changed": true, "cmd": ["packaging/installer/install-required-packages.sh", "--non-interactive", "netdata"], + # "msg": "non-zero return code", "rc": 1} + when /(non-zero return code)/ + output_error $1 + + # Some service could not find the requested host + when /(Could not find the requested service .+): host/ + output_error $1 + + # Unable to start some services + when /(Unable to start service .+):/ + output_error $1 + end +end + +def parse_msg_2(ansible_failed_json) + case ansible_failed_json.chomp + + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + + when /(Failure downloading .+) / + output_error $1 + + # Failed to download the yum repodata + # {"changed": false, "msg": "Failed to download metadata for repo 'docker-ce-stable': Cannot download repomd.xml", + # "rc": 1, "results": []} + when /(Failed to download metadata for repo.+)':/ + output_error $1 + + # Some role in not sipported this system + # {"changed": false, "msg": "Distribution openEuler is not supported by this role!"} + when /(.+ is not supported .+)/ + output_error $1 + + # failed to create temporary content file: The read operation timed out. + when /(.+ The read operation timed out)/ + output_error $1 + + # {"msg": "The conditional check 'java_version is defined' failed. The error was: error while evaluating conditional + # (java_version is defined): {{ java_default_version }}: {{ _java_default_version[ansible_distribution ~ '-' ~ + # ansible_distribution_major_version] | default(_java_default_version[ansible_distribution]) }}: 'dict object' has + # no attribute 'openEuler'\n\nThe error appears to be in '/root/.ansible/roles/ansible-role-java/tasks/assert.yml': + # line 19, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line + # appears to be:\n\n\n- name: test if java_version is set correctly\n ^ here\n"} + when /(The conditional check .+ failed)/ + output_error $1 + end +end + +def parse_failures(ansible_failed_json) + ansible_failed_json.each do |items| + case items.chomp + # Failed to install some of the specified packages. + # {"changed": false, "failures": ["No package erlang available."], "msg": + # "Failed to install some of the specified packages", "rc": 1, "results": []} + when /(No package .+ available)/ + output_error $1 + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json.chomp + + # Could not find or access ansible yaml + # {"ansible_facts": {}, "ansible_included_var_files": [], "changed": false, "message": + # "Could not find or access 'openEuler.yml' Searched in: + # /root/.ansible/roles/ansible-unzip/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/openEuler.yml i + # /root/.ansible/roles/ansible-unzip/tasks/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/tasks/openEuler.yml + # /root/.ansible/vars/openEuler.yml /root/.ansible/openEuler.yml on the Ansible Controller. + # If you are using a module and expect the file to exist on the remote, see the remote_src option"} + when /(Could not find or access .+')/ + output_error $1 end end + +def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line +end + +while (line = STDIN.gets) + next unless line =~ /(FAILED!|failed:).*=>(.*)/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.parse @ansible_failed_info + + parse_msg_1 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_msg_2 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_failures ansible_failed_json['failures'] unless ansible_failed_json['failures'].nil? + parse_message ansible_failed_json['message'] unless ansible_failed_json['message'].nil? +end -- 2.23.0

2 4

[PATCH v4 lkp-tests] jobs: add os-multi-dimension-*.yaml
by Li Ping 24 Feb '21

24 Feb '21

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: fix continue error
by Liu Yinsi 24 Feb '21

24 Feb '21

[error] ./buildall: line 59: continue: only meaningful in a `for', `while', or `until' loop [how] before: [ "$container_name" == 'ssh-r' ] && continue after: [ "$container_name" == 'ssh-r' ] && exit [why] 'continue' can not use independently without `for', `while', or `until', so use 'exit' to fix. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 96c7385..bb77a1f 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,7 +56,7 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && continue + [ "$container_name" == 'ssh-r' ] && exit [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/es/start: call check_es_ready()
by Liu Yinsi 24 Feb '21

24 Feb '21

call container/defconfig.sh:check_es_ready() to accelerate es/logging-es startup time. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/start | 7 +++---- container/logging-es/start | 5 ++--- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/container/es/start b/container/es/start index d84ad33..a1b81f9 100755 --- a/container/es/start +++ b/container/es/start @@ -20,7 +20,6 @@ cmd=( "${cmd[@]}" -echo "Please wait 60s for setting index" -sleep 60 - -set_es_indices +# set index +check_es_ready 9200 +find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; diff --git a/container/logging-es/start b/container/logging-es/start index 66464e6..13bf25d 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -20,9 +20,8 @@ cmd=( "${cmd[@]}" -echo "Please wait 60s for setting index" -sleep 60 - +# set index +check_es_ready 9202 curl -sSH "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' { "index": { -- 2.23.0

1 0

[PATCH lkp-tests] setup/simply-ci: fix git clone error
by Liu Yinsi 24 Feb '21

24 Feb '21

[how] before: git clone https://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git after: git clone https://gitee.com/wu_fengguang/compass-ci.git [why] refer code: git_ci(): ... if [ -n "$GIT_SERVER" ]; then cat >> /etc/gitconfig <<-EOF [url "git://$GIT_SERVER/gitee.com"] insteadOf=https://gitee.com ... if $GIT_SERVER exists, "http://gitee.com" will be instead of "git://$GIT_SERVER/gitee.com", so the old code is wrong. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index 076c357d9..c8f8a5a53 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -14,7 +14,7 @@ git_ci() EOF fi - git clone https://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git /c/compass-ci + git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci } dev_env() -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] delimiter: use secrets field to submit process job
by Cao Xueliang 24 Feb '21

24 Feb '21

Before: Search account info from ES. After: Use secrets field to get the my_token. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.rb | 8 ++++++++ src/delimiter/utils.rb | 18 ++++-------------- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/src/delimiter.rb b/src/delimiter.rb index 1ea5de5..cb43cb4 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -5,8 +5,16 @@ require_relative './delimiter/delimiter' require_relative '../lib/config_account' +def config_secrets_yaml + account = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/defaults/account.yaml" + lab = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/include/lab/#{account['lab']}.yaml" + secrets = Hash["secrets" => lab] + File.open("#{ENV['HOME']}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + begin config_yaml('delimiter') + config_secrets_yaml delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 0930dd7..dbc42ef 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -125,30 +125,20 @@ module Utils File.open(commit_jobs, 'a+') { |f| f.puts content } end - def get_account_info - ESQuery.new(index: 'accounts').query_by_id(DELIMITER_EMAIL) - end - def init_job_content(job_id) job_yaml = AssistResult.new.get_job_yaml(job_id) raise "get job yaml failed, job id: #{job_id}" unless job_yaml job = JSON.parse job_yaml - - account_info = get_account_info - raise "query delimiter account info failed: #{DELIMITER_EMAIL}" unless account_info - record_jobs(job['id'], job['upstream_commit']) job['suite'] = 'bisect' - job['my_name'] = account_info['my_name'] - job['my_email'] = account_info['my_email'] - job['my_token'] = account_info['my_token'] + job['my_name'] = ENV['my_name'] + job['my_email'] = ENV['my_email'] + job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id + job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" - job.delete('tboxgroup') - job.delete('subqueue') - job.delete('queue') job.delete('id') return job -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] sched: response ipxe_response value if ipxe_response field in job yaml
by Cao Xueliang 24 Feb '21

24 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9e2898e..e8d590b 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,6 +160,8 @@ class Sched end private def get_boot_ipxe(job : Job) + return job["ipxe_response"] if job["ipxe_response"]? + response = "#!ipxe\n\n" _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] delimiter: use secrets field to submit process job
by Cao Xueliang 24 Feb '21

24 Feb '21

Before: Search account info from ES. After: Use secrets field to get the my_token. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.rb | 8 ++++++++ src/delimiter/utils.rb | 10 ++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/src/delimiter.rb b/src/delimiter.rb index 1ea5de5..cb43cb4 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -5,8 +5,16 @@ require_relative './delimiter/delimiter' require_relative '../lib/config_account' +def config_secrets_yaml + account = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/defaults/account.yaml" + lab = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/include/lab/#{account['lab']}.yaml" + secrets = Hash["secrets" => lab] + File.open("#{ENV['HOME']}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + begin config_yaml('delimiter') + config_secrets_yaml delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 0930dd7..cd026f9 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -141,14 +141,12 @@ module Utils record_jobs(job['id'], job['upstream_commit']) job['suite'] = 'bisect' - job['my_name'] = account_info['my_name'] - job['my_email'] = account_info['my_email'] - job['my_token'] = account_info['my_token'] + job['my_name'] = ENV['my_name'] + job['my_email'] = ENV['my_email'] + job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id + job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" - job.delete('tboxgroup') - job.delete('subqueue') - job.delete('queue') job.delete('id') return job -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] lib/git_bisect.rb: fix: syntax error, unexpected string literal, expecting']'
by Cao Xueliang 24 Feb '21

24 Feb '21

need add ',' after 'upstream_url' => @upstream_url Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index dd0b782..6aedda5 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -113,7 +113,7 @@ class GitBisect 'work_dir' => @work_dir, 'bisect_error' => bisect_error, 'all_errors' => all_errors, - 'upstream_url' => @upstream_url + 'upstream_url' => @upstream_url, 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end -- 2.23.0

1 0

[PATCH compass-ci 3/3] delimiter: use secrets field to submit process job
by Cao Xueliang 24 Feb '21

24 Feb '21

Before: Search account info from ES. After: Use secrets field to get the my_token. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.rb | 8 ++++++++ src/delimiter/utils.rb | 10 ++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/src/delimiter.rb b/src/delimiter.rb index 1ea5de5..cb43cb4 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -5,8 +5,16 @@ require_relative './delimiter/delimiter' require_relative '../lib/config_account' +def config_secrets_yaml + account = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/defaults/account.yaml" + lab = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/include/lab/#{account['lab']}.yaml" + secrets = Hash["secrets" => lab] + File.open("#{ENV['HOME']}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + begin config_yaml('delimiter') + config_secrets_yaml delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 0930dd7..cd026f9 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -141,14 +141,12 @@ module Utils record_jobs(job['id'], job['upstream_commit']) job['suite'] = 'bisect' - job['my_name'] = account_info['my_name'] - job['my_email'] = account_info['my_email'] - job['my_token'] = account_info['my_token'] + job['my_name'] = ENV['my_name'] + job['my_email'] = ENV['my_email'] + job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id + job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" - job.delete('tboxgroup') - job.delete('subqueue') - job.delete('queue') job.delete('id') return job -- 2.23.0

1 0

[PATCH compass-ci 2/3] lib/git_bisect.rb: fix: syntax error, unexpected string literal, expecting']'
by Cao Xueliang 24 Feb '21

24 Feb '21

need add ',' after 'upstream_url' => @upstream_url Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index dd0b782..6aedda5 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -113,7 +113,7 @@ class GitBisect 'work_dir' => @work_dir, 'bisect_error' => bisect_error, 'all_errors' => all_errors, - 'upstream_url' => @upstream_url + 'upstream_url' => @upstream_url, 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/stats-rule.md
by Wang Chenglong 24 Feb '21

24 Feb '21

[Why] The rules to follow when writing the stats script. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/stats-rule.md | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 doc/stats-rule.md diff --git a/doc/stats-rule.md b/doc/stats-rule.md new file mode 100644 index 0000000..0fb557a --- /dev/null +++ b/doc/stats-rule.md @@ -0,0 +1,42 @@ +${LKP_SERVER}/stats/${script} stat�� +========================================== + +## ��es�� + +1.��error_id�� + - original: {"changed": false, "failures": ["No package yum-plugin-priorities available."], "msg": "Failed to + install some of the specified packages", "rc": 1, "results": []} + + - error_id: error.No-package-yum-plugin-priorities-available: 1 + +2.�� + - ��error_id�� + # �� + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + end + + - ��jobs��'��'��es��error_id.��stats/${script}�� + +3.��error message�� + # original error message + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + end + +4.��error_id�� + - �� + def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/{%/, ' ') + line.gsub!(/%}/, ' ') + line + end -- 2.23.0

2 2

[PATCH lkp-tests 3/3] tests/sysbench-mysql: fix mysql_port, mysql_password default value
by Wei Jihui 24 Feb '21

24 Feb '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/sysbench-mysql | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 5dd8686f..d6f5dcda 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -20,9 +20,9 @@ : "${mysql_user:=root}" : "${mysql_host:=$server}" -: "${mysql_port:=$mysql_port}" +: "${mysql_port:=3306}" : "${mysql_db:=sysbench_1}" -: "${mysql_password:=$mysql_password}" +: "${mysql_password:=mysql_password}" : "${db_driver:=mysql}" : "${oltp_test_mode:=complex}" : "${oltp_tables_count:=1000}" -- 2.23.0

1 0

[PATCH v3 lkp-tests] jobs: add os-multi-dimension-arch.yaml
by Li Ping 24 Feb '21

24 Feb '21

1 0

[PATCH v2 compass-ci] container/assign-account: optimize config login permission
by Luan Shengde 24 Feb '21

24 Feb '21

optimize account login permission setting: before: set enable login: has key: enable_login key value: true set disable login has key: enable_login key value: false or has no key: enable_login now: set enable login: has key: enable_login key value: true set disable login: has key: enable_login key value: false keep the default login permission: has no key: enable_login Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 7e0a5a7..228d468 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -136,7 +136,7 @@ class AccountStorage def permit_login_config(login_name) if @data.key?('enable_login') && @data['enable_login'] %x(usermod -s /usr/bin/zsh #{login_name}) - else + elsif @data.key?('enable_login') && ! @data['enable_login'] %x(usermod -s /sbin/nologin #{login_name}) end end -- 2.23.0

1 0

[PATCH lkp-tests] add tests script for openeuler os migration
by Bai Jing 24 Feb '21

24 Feb '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/openeuler_docker | 114 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100755 tests/openeuler_docker diff --git a/tests/openeuler_docker b/tests/openeuler_docker new file mode 100755 index 000000000..f79610b34 --- /dev/null +++ b/tests/openeuler_docker @@ -0,0 +1,114 @@ +#!/bin/sh +# - dockerfile_url + +. $LKP_SRC/lib/debug.sh + +[ -n "$dockerfile_url" ] || die "dockerfile_url is empty" + +cd $BENCHMARK_ROOT + +fit_repo() +{ + yum reinstall -y git + git clone "$dockerfile_url" || exit 0 + file_dir=$(find ./ -type f -name "*Dockerfile*" | xargs grep -E '^FROM centos:(centos|)[78]') + [ -n "$file_dir" ] || exit 0 +} + +openeuler_ready() +{ + docker load < $BENCHMARK_ROOT/openeuler_docker/openeuler-20.03-lts-v1.tar +} + +centos_docker() +{ + flag=0 + for file in $(find ./ -type f -name "*Dockerfile*" | xargs grep -E '^FROM centos:(centos|)[78]' | awk -F: '{print $1}') + do + image=$(echo "$file" | awk -F '/' '{print $3}' | tr 'A-Z' 'a-z') + if [ -f "build.sh" ] && [ -x "build.sh" ]; then + ./build.sh > /dev/null 2> /dev/null && handle_dockerfile + else + docker_build > /dev/null 2> /dev/null && handle_dockerfile + fi + done +} + +handle_FROM() +{ + sed -i "s/^FROM .*/FROM openeuler-20.03-lts:v1/g" "$file" + +} + +handle_epel() +{ + grep -qw "epel-release" "$file" && { + sed -i "s/epel-release/bash/g" "$file" + } +} + +handle_epelrpm() +{ + + grep -qw "https://.*epel-release.*rpm" "$file" && { + sed -i "s|https://.*epel-release.*rpm|bash|g" "$file" + + } +} + +handle_groupadd() +{ + grep -qw "groupadd*" "$file" && { + sed -i '/groupadd*/i\ yum -y install shadow && \\' "$file" + } + +} + +handle_epel_repo() +{ + grep -qwE '.* wget .*(epel|CentOS-Base)\.repo' "$file" && { + sed 's|^|#|g' "$file" + } +} + +handle_dockerfile() +{ + flag=1 + handle_FROM + handle_epel + handle_groupadd + docker_build "-openeuler" + [ $? -ne 0 ] || docker_run "-openeuler" +} + +docker_build() +{ + docker build -f "$file" -t "${image}$1" . + [ $? -ne 0 ] && { + [ "$flag" == "1" ] && echo "${image}$1.build.fail: 1" + return 1 + } + [ "$flag" == "1" ] && echo "${image}$1.build.success: 1" + flag=0 +} + +docker_run() +{ + docker run -d --name "${image}" "${image}$1" + docker_stats=$(docker inspect ${image} | grep Running | awk -F ' ' '{print $2}' | tr -d ',') + [ $docker_stats == false ] && { + echo "${image}.run.fail: 1" + return 1 + } + + echo "${image}.run.success: 1" +} + +docker_test() +{ + fit_repo + openeuler_ready + centos_docker +} + +docker_test -- 2.23.0

1 1

[PATCH v2 lkp-tests] jobs/: add os-multi-dimension-arch.yaml
by Li Ping 24 Feb '21

24 Feb '21

3 7

[PATCH v3 compass-ci] answerback-email: add store lab to ES
by Luan Shengde 24 Feb '21

24 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 8dceee1..f573f0f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -40,7 +40,8 @@ my_info = { 'my_commit_url' => nil, 'my_token' => nil, 'my_login_name' => nil, - 'my_ssh_pubkey' => [] + 'my_ssh_pubkey' => [], + 'lab' => LAB } # stdin_info is used to store infos added with option: @@ -179,7 +180,6 @@ def apply_account(my_info, conf_info) apply_info = {} apply_info.update my_info apply_info.update conf_info - apply_info['lab'] = LAB assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" account_info_str = RestClient.post assign_account_url, apply_info.to_json -- 2.23.0

1 0

[PATCH v6 compass-ci] doc: optimize os_mount.md
by Yu Chuan 24 Feb '21

24 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 255 +++++++++++++++++++++++++++++++------------- 1 file changed, 178 insertions(+), 77 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..f4b28bb4b4c7 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,9 +1,9 @@ # Summary -========= +--------- -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: +The below types are supported: - nfs - cifs - initramfs @@ -11,84 +11,185 @@ It has the following optional values: - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. - -2. scheduler return the ipxe_str to testbox + + ```shell + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +# os_mount=local +---------------- + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. + +So if you want to run your job in disk file system, you need to specify `os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + + +## temporary rootfs data by default + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follows: + ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot from this lv, just use it as a source data, and take a snapshot of it to get the real boot logical volume. + - src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + - ${timestamp} is just a version number, which is the latest rootfs version on the nfs server in cluster. + ${boot_lv} -- boot logical volume: + - will boot from this lv. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it. + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. ``` -3. dracut step of boot +## persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follows: + + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the + rootfs data of job-20210218.yaml so that it can be used by the subsequent + jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs + data of job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, `save_root_partition` and `use_root_partition` must be the same, and unique. + - you must `save_root_partition` firstly, then you can `use_root_partition`. otherwise, your job will fail. + +## work flow + +1. user submit job + + job fields: + ```yaml + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` + +2. scheduler return the ipxe_response to testbox + + ``` + #!ipxe + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ``` + +3. dracut step of boot + + ```bash + analyze_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analyze_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main + ``` 4. execute the job -- 2.23.0

1 0

[PATCH v5 compass-ci] doc: optimize os_mount.md
by Yu Chuan 24 Feb '21

24 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 264 ++++++++++++++++++++++++++++++-------------- 1 file changed, 180 insertions(+), 84 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..64e86b0ecb5f 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,94 +1,190 @@ -# Summary -========= +Summary +======= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: - - nfs - - cifs - - initramfs - - container - - local +The below types are supported: + - nfs + - cifs + - initramfs + - container + - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +os_mount=local +============== + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. +So if you want to run your job in disk file system, you need to specify +`os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + + +## temporary rootfs data by default + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs +will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot from this lv, just use it as a source data, and take a + snapshot of it to get the real boot logical volume. + - src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + - ${timestamp} is just a version number, which is the latest rootfs version + on the nfs server in cluster. + ${boot_lv} -- boot logical volume: + - will boot from this lv. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +## persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), +two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + job-20210218.yaml so that it can be used by the subsequent jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, ${save_root_partition} and ${use_root_partition} must be the same, and unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox - ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot - ``` + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ``` 3. dracut step of boot - ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi - ``` + ``` + analyze_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analyze_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main + ``` 4. execute the job -- 2.23.0

2 2

[PATCH compass-ci] doc/manual/submit-job.md: fix link error
by Wei Jihui 24 Feb '21

24 Feb '21

the links file name has changed to english. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- doc/manual/submit-job.en.md | 2 +- doc/manual/submit-job.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/submit-job.en.md b/doc/manual/submit-job.en.md index 2748fbe..de73f8f 100644 --- a/doc/manual/submit-job.en.md +++ b/doc/manual/submit-job.en.md @@ -2,7 +2,7 @@ ### Prerequisites -The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/本地…. +The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/in…. ### Purpose diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 92faa5c..77e9566 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -2,7 +2,7 @@ ### 前提条件 -已经按照[本地安装compass-ci客户端.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/本地安装compass-ci客户端.md )完成安装 +已经按照[本地安装compass-ci客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/install-cci-client.md )完成安装 ### 概述 -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/submit-job.md: fix link error
by Wei Jihui 24 Feb '21

24 Feb '21

the links file name has changed to english. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- doc/manual/submit-job.en.md | 2 +- doc/manual/submit-job.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/submit-job.en.md b/doc/manual/submit-job.en.md index 2748fbe..de73f8f 100644 --- a/doc/manual/submit-job.en.md +++ b/doc/manual/submit-job.en.md @@ -2,7 +2,7 @@ ### Prerequisites -The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��…. +The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/in…. ### Purpose diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 92faa5c..77e9566 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -2,7 +2,7 @@ ### �� -��[��compass-ci��.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��compass-ci��.md )�� +��[��compass-ci��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/install-cci-client.md )�� ### �� -- 2.23.0

1 0

[PATCH compass-ci] doc/stats-rule.md
by Wang Chenglong 24 Feb '21

24 Feb '21

[Why] The rules to follow when writing the stats script. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/stats-rule.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 doc/stats-rule.md diff --git a/doc/stats-rule.md b/doc/stats-rule.md new file mode 100644 index 0000000..94affa7 --- /dev/null +++ b/doc/stats-rule.md @@ -0,0 +1,37 @@ +${LKP_SERVER}/stats/${script} stat�� +========================================== + +## ��es�� + +1.��error_id�� + - original: {"changed": false, "failures": ["No package yum-plugin-priorities available."], "msg": "Failed to + install some of the specified packages", "rc": 1, "results": []} + + - error_id: error.No-package-yum-plugin-priorities-available: 1 + +2.�� + - catch all kind errors + ��error_id�� + ��jobs��'��'��es��error_id.��stats/${script}�� + +3.��error message�� + # original error message + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + end + +4.��error_id�� + - �� + def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/{%/, ' ') + line.gsub!(/%}/, ' ') + line + end -- 2.23.0

2 2

[PATCH lab-z9] hosts/*: add default hdd_partitions
by Cui Yili 24 Feb '21

24 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- hosts/vm-1p1g | 1 + hosts/vm-2p16g | 1 + hosts/vm-2p1g | 1 + hosts/vm-2p4g | 1 + hosts/vm-2p8g | 1 + 5 files changed, 5 insertions(+) diff --git a/hosts/vm-1p1g b/hosts/vm-1p1g index 3af3646..b278e80 100644 --- a/hosts/vm-1p1g +++ b/hosts/vm-1p1g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 1 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p16g b/hosts/vm-2p16g index 10847a7..cd49afc 100644 --- a/hosts/vm-2p16g +++ b/hosts/vm-2p16g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 16G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g b/hosts/vm-2p1g index 0293ce9..4fffd2a 100644 --- a/hosts/vm-2p1g +++ b/hosts/vm-2p1g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g b/hosts/vm-2p4g index 219c4c9..41d0a3b 100644 --- a/hosts/vm-2p4g +++ b/hosts/vm-2p4g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g b/hosts/vm-2p8g index 6249322..8bc5cad 100644 --- a/hosts/vm-2p8g +++ b/hosts/vm-2p8g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: /dev/vda -- 2.23.0

2 1

[PATCH compass-ci] doc/manual/submit-job.md: fix link error
by Wei Jihui 24 Feb '21

24 Feb '21

the links file name has changed to english. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- doc/manual/submit-job.en.md | 2 +- doc/manual/submit-job.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/submit-job.en.md b/doc/manual/submit-job.en.md index 2748fbe..de73f8f 100644 --- a/doc/manual/submit-job.en.md +++ b/doc/manual/submit-job.en.md @@ -2,7 +2,7 @@ ### Prerequisites -The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��…. +The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/in…. ### Purpose diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 92faa5c..77e9566 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -2,7 +2,7 @@ ### �� -��[��compass-ci��.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��compass-ci��.md )�� +��[��compass-ci��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/install-cci-client.md )�� ### �� -- 2.23.0

1 0

[PATCH v4 compass-ci] doc: optimize os_mount.md
by Yu Chuan 24 Feb '21

24 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 265 ++++++++++++++++++++++++++++++-------------- 1 file changed, 181 insertions(+), 84 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..1e6b349a136d 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,94 +1,191 @@ -# Summary -========= +Summary +======= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: - - nfs - - cifs - - initramfs - - container - - local +The below types are supported: + - nfs + - cifs + - initramfs + - container + - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +os_mount=local +============== + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. +So if you want to run your job in disk file system, you need to specify +`os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + +## description + +### temporary rootfs data *Defaultly* + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs +will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot from this lv, just use it as a source data, and take a + snapshot of it to get the real boot logical volume. + - src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + - ${timestamp} is just a version number, which is the latest rootfs version + on the nfs server in cluster. + ${boot_lv} -- boot logical volume: + - will boot from this lv. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +### persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), +two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + job-20210218.yaml so that it can be used by the subsequent jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, ${save_root_partition} and ${use_root_partition} must be the same, and unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox - ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot - ``` + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ``` 3. dracut step of boot - ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi - ``` + ``` + analysis_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analysis_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main() + ``` 4. execute the job -- 2.23.0

2 2

[PATCH lkp-tests] jobs/: add os-multi-dimension-arch.yaml
by Li Ping 23 Feb '21

23 Feb '21

2 2

[PATCH lkp-tests] lib: export secrets.yaml in job_init
by Cao Xueliang 23 Feb '21

23 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/job-init.sh | 2 ++ lib/job.sh | 18 +++++++++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/lib/job-init.sh b/lib/job-init.sh index 1fb898872..27d3c6d0b 100755 --- a/lib/job-init.sh +++ b/lib/job-init.sh @@ -1,4 +1,5 @@ . $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/job.sh mount_cgroup() { @@ -270,4 +271,5 @@ job_init() job_redirect_stdout_stderr job_env + read_secret_vars } diff --git a/lib/job.sh b/lib/job.sh index dc65f68d8..87ca7ee67 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -1,8 +1,20 @@ #!/bin/sh +read_secret_vars() +{ + read_yaml_vars "$(dirname $job_file)/secrets.yaml" secrets_ +} + read_env_vars() { - [ -f "$TMP/env.yaml" ] || return 0 + read_yaml_vars "$TMP/env.yaml" +} + +read_yaml_vars() +{ + local file=$1 + local prefix=$2 + [ -f "$file" ] || return 0 local key local val @@ -11,8 +23,8 @@ read_env_vars() do [ "${key%[a-zA-Z0-9_]:}" != "$key" ] || continue key=${key%:} - export "$key=$val" - done < $TMP/env.yaml + export "$prefix$key=$val" + done < $file return 0 } -- 2.23.0

1 0

find latest github git pushes
by Wu Fengguang 23 Feb '21

23 Feb '21

Hi Yuanchao, I researched github and have 2 finding 1) a registered user can watch a number of repos and get notifications. however the notification type does not include "git push/new commit". There is only "new pull request", which looks much less useful for us. 2) we can still take advantage of its search API to find latest updated repos, like this https://github.com/search?l=&o=desc&q=pushed%3A%3E2021-02-21+filename%3AREA… 7,543 repository results clean-code-craft-tcq-1/modular-c-AnPra7 modular-c-AnPra7 created by GitHub Classroom C MIT license Updated 3 minutes ago trpo2021/chessviz-Jamesss1337 chessviz-Jamesss1337 created by GitHub Classroom C Updated 3 minutes ago mubu43/ExploringC C Updated 3 minutes ago BlankyShi/gitTest gitTest C Updated 4 minutes ago aabelque/malloc C Updated 4 minutes ago ythan88/r-card C Updated 4 minutes ago koipkoi/pokeemerald-kr A korean translation project for Pokémon Emerald. C Updated 4 minutes ago OpenMPT/openmpt Official read-only git mirror of the OpenMPT and libopenmpt subversion repository at https://source.openmpt.org/. Thi… libopenmpt tracker audio music demoscene music-creation protracker impulsetracker screamtracker fasttracker 223 C Updated 4 minutes ago silabs-DenverL/ADCxSyncCapture ADC0 and ADC1 Synchronization Capture C MIT license Updated 4 minutes ago gopro/gopro-lib-node.gl OpenGL engine for building and rendering graph-based scenes 32 C Apache-2.0 license Updated 4 minutes ago Thanks, Fengguang

1 2

[PATCH compass-ci] 2-network/iptables: fix iptables holding the xtables lock
by Cui Yili 23 Feb '21

23 Feb '21

[why] error: Another app is currently holding the xtables lock. Perhaps you want to use the -w option? [how] add lock wait time. iptables -w 10... Signed-off-by: Cui Yili <2268260388(a)qq.com> --- sparrow/2-network/iptables | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 48794f5..73f9749 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -8,9 +8,9 @@ BR0_IFACE=br0 BR0_SUBNET=172.18.0.0/16 # iptables -t nat -F -iptables -I FORWARD 1 -j ACCEPT -iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE -iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE +iptables -w 10 -I FORWARD 1 -j ACCEPT +iptables -w 10 -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE +iptables -w 10 -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE command -v firewall-cmd > /dev/null || { exit 0 -- 2.23.0

1 0

[PATCH compass-ci] support-os-os_arch-os_version.md: delete the cifs method of archlinux
by Wang Chenglong 23 Feb '21

23 Feb '21

[Why] When start archlinux with CIFS, the system will get mangled names error. So only show the NFS method for user. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/support-os-os_arch-os_version.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/support-os-os_arch-os_version.md b/doc/support-os-os_arch-os_version.md index 5d30486..93fb47b 100644 --- a/doc/support-os-os_arch-os_version.md +++ b/doc/support-os-os_arch-os_version.md @@ -18,5 +18,5 @@ centos aarch64 7 container, 7.8.2003 cifs/nfs, debian aarch64 sid cifs/nfs, ,initramfs x86 sid cifs/nfs, -archlinux aarch64 2020-11-12 cifs/nfs, +archlinux aarch64 2020-11-12 nfs, ``` -- 2.23.0

1 0

[PATCH v3 compass-ci] doc: optimize os_mount.md
by Yu Chuan 23 Feb '21

23 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 263 ++++++++++++++++++++++++++++++-------------- 1 file changed, 179 insertions(+), 84 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..0d0afd6469a8 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,94 +1,189 @@ -# Summary -========= +Summary +======= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: - - nfs - - cifs - - initramfs - - container - - local +The below types are supported: + - nfs + - cifs + - initramfs + - container + - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +os_mount=local +============== + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. +So if you want to run your job in disk file system, you need to specify +`os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + +## description + +### temporary rootfs data *Defaultly* + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs +will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follow: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot form this lv, just use it as a source data, and take a + snapshot of it to get the real boot logical volume. + - srv_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - will boot from this. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +### persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), +two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follow: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + job-20210218.yaml so that it can be used by the subsequent jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, ${save_root_partition} and ${use_root_partition} must be the same, and unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox - ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot - ``` + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ``` 3. dracut step of boot - ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi - ``` + ``` + analysis_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analysis_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main() + ``` 4. execute the job -- 2.23.0

2 2

[PATCH v4 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 23 Feb '21

23 Feb '21

[How] ansible_stats_openeuler < ${job_output} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 135 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 124 insertions(+), 11 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..bd60a3ebc 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,128 @@ #!/usr/bin/env ruby -while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line +require 'json' + +def parse_msg_1(ansible_failed_json) + case ansible_failed_json.chomp + + # When running some plug-in, the file was not found + when /No file was found when using (.+)\./ + puts "files.error.no-file-was-found-when-using.#{$1}: 1" + puts "files.error.no-file-was-found-when-using.#{$1}.message: #{@ansible_failed_info}" + + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /No such file or directory: b (.+) :/ + puts "error.no-such-file-or-directory.#{$1}: 1" + puts "error.no-such-file-or-directory.#{$1}.message: #{@ansible_failed_info}" + + # The task includes an option with an undefined variable + when /: (.+) is undefined/ + puts "syntax.error.an-undefined-variable.#{$1}: 1" + puts "syntax.error.an-undefined-variable.#{$1}.message: #{@ansible_failed_info}" + + # {"changed": true, "cmd": ["packaging/installer/install-required-packages.sh", "--non-interactive", "netdata"], + # "msg": "non-zero return code", "rc": 1} + when /non-zero return code/ + puts "error.non-zero-return-code.#{ansible_failed_json}: 1" + puts "error.non-zero-return-code.#{ansible_failed_json}.message: #{@ansible_failed_info}" + + # Some service could not find the requested host + when /Could not find the requested service (.+): host/ + puts "request.error.could-not-find-service-host.#{$1}: 1" + puts "request.error.could-not-find-service-host.#{$1}.messgae: #{@ansible_failed_info}" + + # Unable to start some services + when /Unable to start service (.+):/ + puts "service.error.unable-to-start-service.#{$1}: 1" + puts "service.error.unable-to-start-service.#{$1}.message: #{@ansible_failed_info}" + end +end + +def parse_msg_2(ansible_failed_json) + case ansible_failed_json.chomp + + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /Failed to download packages: Cannot download (.+):/ + puts "download.error.failed-to-download-package.#{$1}: 1" + puts "download.error.failed-to-download-package.#{$1}.message: #{@ansible_failed_info}" + + # {"changed": false, "msg": "Failure downloading https://rpms.remirepo.net/enterprise/remi-release-20.rpm, + # HTTP Error 404: Not Found"} + when /Failure downloading (.+), / + puts "download.error.Failure-downloading.[#{$1}]: 1" + puts "download.error.Failure-downloading.[#{$1}]: #{@ansible_failed_info}" + + # Failed to download the yum repodata + # {"changed": false, "msg": "Failed to download metadata for repo 'docker-ce-stable': Cannot download repomd.xml", + # "rc": 1, "results": []} + when /Failed to download metadata for repo (.+) :/ + puts "download.error.failed-to-download-metadata.#{$1}: 1" + puts "download.error.failed-to-download-metadata.#{$1}.message: #{@ansible_failed_info}" + + # Some role in not sipported this system + # {"changed": false, "msg": "Distribution openEuler is not supported by this role!"} + when /(.+) is not supported/ + puts "system.error.not-supported.#{$1.gsub(/ /, '-')}: 1" + puts "system.error.not-supported.message: #{@ansible_failed_info}" + + # failed to create temporary content file: The read operation timed out. + when /The read operation timed out/ + puts 'connection.error.read-timed-out: 1' + puts "connection.error.read-timed-out.message: #{@ansible_failed_info}" end end + +def parse_failures(ansible_failed_json) + ansible_failed_json.each do |items| + case items.chomp + # Failed to install some of the specified packages. + # {"changed": false, "failures": ["No package erlang available."], "msg": + # "Failed to install some of the specified packages", "rc": 1, "results": []} + when /No package (.+) available/ + puts "yum.error.no-package-available.#{$1}: 1" + puts "yum.error.no-package-available.#{$1}.message: #{@ansible_failed_info}" + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json.chomp + + # Could not find or access ansible yaml + # {"ansible_facts": {}, "ansible_included_var_files": [], "changed": false, "message": + # "Could not find or access 'openEuler.yml' Searched in: + # /root/.ansible/roles/ansible-unzip/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/openEuler.yml i + # /root/.ansible/roles/ansible-unzip/tasks/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/tasks/openEuler.yml + # /root/.ansible/vars/openEuler.yml /root/.ansible/openEuler.yml on the Ansible Controller. + # If you are using a module and expect the file to exist on the remote, see the remote_src option"} + when /Could not find or access (.+) / + puts "yaml.error.not-find-or-access-yaml.#{$1}: 1" + puts "yaml.error.not-find-or-access-yaml.#{$1}.message: #{@ansible_failed_info}" + end +end + +def common_error_id(line) + line = line.chomp + line.gsub(/'/, ' ') + line.gsub(/{%/, ' ') + line.gsub(/%}/, ' ') + line.gsub(/{},/, '[],') + line.gsub(/\\n\\t/, ' ') + line +end + +while (line = STDIN.gets) + line = common_error_id(line) + next unless line =~ /(FAILED!|failed:).*({.*})/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.load(@ansible_failed_info) + + parse_msg_1 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_msg_2 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_failures ansible_failed_json['failures'] unless ansible_failed_json['failures'].nil? + parse_message ansible_failed_json['message'] unless ansible_failed_json['message'].nil? +end -- 2.23.0

4 6

[PATCH lkp-tests] lib/job2sh.rb: delete secrets field when job2sh
by Cao Xueliang 23 Feb '21

23 Feb '21

We don't show the secrets field in job.sh file, so delete secrets field when job2sh. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/job2sh.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7f9cf562f..82ec7d249 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -318,6 +318,7 @@ class Job2sh < Job unless job job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job job.delete_if { |key, val| parse_one([], key, val, :PASS_EXPORT_ENV) } + job.delete_if { |key, val| key == "secrets" } end @cur_func = :run_job -- 2.23.0

1 0

[PATCH compass-ci] sched: add the secrets field processing logic
by Cao Xueliang 23 Feb '21

23 Feb '21

You can add the secrets field which you don't want to make public but the SUT need it in job.yaml. secrets: my_token:xxx The scheduler will save the secrets value to the redis id2secrets table, then delete the field from job.yaml. When consume the job, the scheduler will read the value from redis, then create a secrets.yaml in job.cgz. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/create-job-cpio.sh | 1 + src/scheduler/create_job_cpio.cr | 13 +++++++++++++ src/scheduler/submit_job.cr | 7 +++++++ 3 files changed, 21 insertions(+) diff --git a/sbin/create-job-cpio.sh b/sbin/create-job-cpio.sh index e18bc0d..afb316c 100755 --- a/sbin/create-job-cpio.sh +++ b/sbin/create-job-cpio.sh @@ -9,6 +9,7 @@ cd "$1" || exit install -m775 -D -t lkp/scheduled job.sh install -m664 -D -t lkp/scheduled job.yaml +[ -f secrets.yaml ] && install -m664 -D -t lkp/scheduled secrets.yaml find lkp | cpio --quiet -o -H newc | gzip > job.cgz diff --git a/src/scheduler/create_job_cpio.cr b/src/scheduler/create_job_cpio.cr index 51d79f7..5634f67 100644 --- a/src/scheduler/create_job_cpio.cr +++ b/src/scheduler/create_job_cpio.cr @@ -74,8 +74,21 @@ class Sched script_lines = JSON.parse(script_lines) end + def create_secrets_yaml(job_id, base_dir) + secrets = @redis.hash_get("id2secrets", job_id) + return nil unless secrets + + secrets_yaml = base_dir + "/#{job_id}/secrets.yaml" + prepare_dir(secrets_yaml) + + File.open(secrets_yaml, "w") do |file| + YAML.dump(JSON.parse(secrets), file) + end + end + def create_job_cpio(job_content : JSON::Any, base_dir : String) job_content = job_content.as_h + create_secrets_yaml(job_content["id"], base_dir) # put job2sh in an array if job_content.has_key?("job2sh") diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 1837cb1..fe9efa1 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -144,8 +144,15 @@ class Sched JSON.parse(response[1].to_json)["id"].to_s if response[0] == 200 end + def save_secrets(job, job_id) + return nil unless job["secrets"]? + + @redis.hash_set("id2secrets", job_id, job["secrets"]?.to_json) + job.delete("secrets") + end # add job content to es and return a response def add_job(job, job_id) + save_secrets(job, job_id) job.update_id(job_id) @es.set_job_content(job) end -- 2.23.0

1 0

[PATCH lab-z9] add muti-qemu hosts and alter the multi-qemu.yaml
by Bai Jing 23 Feb '21

23 Feb '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- allot/idle/taishan200-2280-2s48p-256g--a63 | 1 + allot/idle/taishan200-2280-2s48p-256g--a64 | 1 + allot/idle/taishan200-2280-2s48p-256g--a65 | 1 + allot/idle/taishan200-2280-2s48p-256g--a66 | 1 + allot/idle/taishan200-2280-2s48p-256g--a67 | 1 + allot/idle/taishan200-2280-2s48p-256g--a68 | 1 + allot/idle/taishan200-2280-2s48p-256g--a69 | 1 + allot/idle/taishan200-2280-2s48p-256g--a70 | 1 + .../multi-qemu.yaml | 15 ++++++++++++--- 9 files changed, 20 insertions(+), 3 deletions(-) create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a63 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a64 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a65 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a66 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a67 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a68 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a69 create mode 120000 allot/idle/taishan200-2280-2s48p-256g--a70 diff --git a/allot/idle/taishan200-2280-2s48p-256g--a63 b/allot/idle/taishan200-2280-2s48p-256g--a63 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a63 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a64 b/allot/idle/taishan200-2280-2s48p-256g--a64 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a64 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a65 b/allot/idle/taishan200-2280-2s48p-256g--a65 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a65 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a66 b/allot/idle/taishan200-2280-2s48p-256g--a66 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a66 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a67 b/allot/idle/taishan200-2280-2s48p-256g--a67 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a67 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a68 b/allot/idle/taishan200-2280-2s48p-256g--a68 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a68 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a69 b/allot/idle/taishan200-2280-2s48p-256g--a69 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a69 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--a70 b/allot/idle/taishan200-2280-2s48p-256g--a70 new file mode 120000 index 0000000..3c3d0c3 --- /dev/null +++ b/allot/idle/taishan200-2280-2s48p-256g--a70 @@ -0,0 +1 @@ +taishan200-2280-2s48p-256g--multi-qemu \ No newline at end of file diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml index ed55c08..87f684e 100644 --- a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml @@ -12,14 +12,23 @@ fs: simplify-ci: multi-qemu-0: - nr_vm: 20 - tbox_group: vm-2p8g + nr_vm: 2 + tbox_group: vm-1p1g multi-qemu-1: nr_vm: 2 tbox_group: vm-2p4g multi-qemu-2: + nr_vm: 12 + tbox_group: vm-2p8g +multi-qemu-3: + nr_vm: 6 + tbox_group: vm-2p16g +multi-qemu-4: nr_vm: 2 - tbox_group: vm-1p1g + tbox_group: vm-2p32g +multi-qemu-5: + nr_vm: 2 + tbox_group: vm-4p64g sleep: 100d os: openeuler -- 2.23.0

1 0

[PATCH lkp-tests] setup/simplify-ci: git failed to exit directly
by Cui Yili 23 Feb '21

23 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- setup/simplify-ci | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index c8f8a5a53..5aede9cdb 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -14,7 +14,7 @@ git_ci() EOF fi - git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci + git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci || return 1 } dev_env() @@ -68,7 +68,7 @@ run_service() main() { - git_ci + git_ci || return 1 install_env run_service } -- 2.23.0

1 0

[PATCH compass-ci] sbin/auto_submit: Adapting to new config style of DEFAULTS files
by Li Yuanchao 23 Feb '21

23 Feb '21

As the config style of DEFAULTS files in upstream-repos changes, the auto_submit need change to adapt to it. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 53 ++++++++++++++++-------------------------------- 1 file changed, 18 insertions(+), 35 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index b408158..857af29 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -57,28 +57,13 @@ class AutoSubmit system(real_argvs.join(' ')) end - def get_branch_commands(submit_commands) - submit_commands.each do |element| - return element['branch_commands'] if element['branch_commands'] - end - return nil - end + def submit_specific_branch(submit_argv, newrefs_info, branches) + branches.each do |branch_name| + branch = "refs/heads/#{branch_name}" + next unless newrefs_info['new_refs']['heads'][branch] - def submit_specific_branch(submit_argv, newrefs_info, branch_commands) - return newrefs_info['new_refs']['heads'] if branch_commands.nil? - - non_specific_commits = {} - newrefs_info['new_refs']['heads'].each do |branch, commit_id| - branch_name = branch.delete_prefix('refs/heads/') - if branch_commands[branch_name].nil? - non_specific_commits[branch] = commit_id - next - end - argvs = Array.new(submit_argv) - argvs.push(branch_commands[branch_name]) - submit_one_job(argvs, newrefs_info['git_repo'], branch_name, commit_id) + submit_one_job(submit_argv, newrefs_info['git_repo'], branch_name, newrefs_info['new_refs']['heads'][branch]) end - return non_specific_commits end def submit_non_specific(non_specific_commits, git_repo, submit_argv) @@ -90,13 +75,13 @@ class AutoSubmit # Add support of different branch different job # newrefs_info['submit'] is like: - # ["submit command 1", - # "submit command 2", + # [ + # { "command" => "submit command 1", + # "branches" => [ "master", "next", ... ] + # }, # ..., - # { "branch_commands" => { "branch_1" => "submit command 1", - # ..., - # "branch_n" => "submit command n" - # } + # { "command" => "submit command n", + # "branches" => [ "branch_name_a", "branch_name_b", ... ] # } # ] def submit_job(newrefs_info) @@ -105,16 +90,14 @@ class AutoSubmit return unless newrefs_info['submit'] - branch_commands = get_branch_commands(newrefs_info['submit']) - non_specific_commits = submit_specific_branch(submit_argv, newrefs_info, branch_commands) - return if non_specific_commits.empty? - - newrefs_info['submit'].each do |argv_config| - next if argv_config.is_a? Hash - + newrefs_info['submit'].each do |element| argvs = Array.new(submit_argv) - argvs.push(argv_config) - submit_non_specific(non_specific_commits, newrefs_info['git_repo'], argvs) + argvs.push(element['command']) + if element['branches'] + submit_specific_branch(argvs, newrefs_info, element['branches']) + else + submit_non_specific(newrefs_info['new_refs']['heads'], newrefs_info['git_repo'], argvs) + end end end -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: redirect the sync_dest directory
by Li Ping 23 Feb '21

23 Feb '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- tests/rpmbuild-pkg | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 9f455f0d0..38de16e66 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,6 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} +[[ "$os_mount" = "cifs" ]] && os_mount="nfs" sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${package_name}" init_workspace() -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs/: add os-arch-version-mount.yaml for multi-dimension test
by Li Ping 23 Feb '21

23 Feb '21

1 1

[PATCH compass-ci] container/scheduler/start: fix read config
by Wei Jihui 23 Feb '21

23 Feb '21

scheduler service need read config from /etc/compass-ci/[service|scheduler], not only /etc/compass-ci/default. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/scheduler/start | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/container/scheduler/start b/container/scheduler/start index c32a7a8..4aaa71d 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -46,7 +46,6 @@ DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" LAB = defaults['lab'] || 'z9' SCHED_PORT = defaults['SCHED_PORT'] SCHED_HOST = defaults['SCHED_HOST'] -DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' docker_rm "s001-alpine-#{SCHED_PORT}" @@ -68,9 +67,9 @@ cmd = %W[ -e CCI_SRC=#{DEFAULT_CCI} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} - -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -v /srv/cci/scheduler/alpine:/srv/cci/scheduler + -v /etc/compass-ci:/etc/compass-ci:ro -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result -v /srv/initrd:/srv/initrd:ro -- 2.23.0

1 0

[PATCH compass-ci] container/assign-account: optimize account login permission
by Luan Shengde 23 Feb '21

23 Feb '21

[why] the mail robot sets no keyword 'enable_login' when assigning account. the applied account via sending 'apply account' email was set no permission to login. before: set enable login: has key: enable_login key value: true set disable login has key: enable_login key value: false or has no key: enable_login now: set enable login: has key: enable_login key value: true set disable login: has key: enable_login key value: false keep the default: has no key: enable_login Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 7e0a5a7..228d468 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -136,7 +136,7 @@ class AccountStorage def permit_login_config(login_name) if @data.key?('enable_login') && @data['enable_login'] %x(usermod -s /usr/bin/zsh #{login_name}) - else + elsif @data.key?('enable_login') && ! @data['enable_login'] %x(usermod -s /sbin/nologin #{login_name}) end end -- 2.23.0

1 0

[PATCH lab-z9] idle/*/multi-qemu.yaml: raid the added disk
by Cui Yili 23 Feb '21

23 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- .../multi-qemu.yaml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml index 8182a97..ed55c08 100644 --- a/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml +++ b/allot/idle/taishan200-2280-2s48p-256g--multi-qemu/multi-qemu.yaml @@ -1,8 +1,14 @@ suite: multi-qemu category: functional -disk: 1HDD -fs: ext4 +nr_hdd: {{nr_hdd_partitions}} +disk: + +md: + - RAID0 + +fs: + - ext4 simplify-ci: multi-qemu-0: -- 2.23.0

2 1

[PATCH compass-ci] sched: delete job.cgz from result root dir
by Cao Xueliang 22 Feb '21

22 Feb '21

We has the job.yaml and job.sh in result root dir, no need to save the job.cgz. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/create_job_cpio.cr | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/scheduler/create_job_cpio.cr b/src/scheduler/create_job_cpio.cr index 1101a0f..51d79f7 100644 --- a/src/scheduler/create_job_cpio.cr +++ b/src/scheduler/create_job_cpio.cr @@ -134,8 +134,7 @@ class Sched # the job.yaml is not final version files = ["#{src_dir}/job.sh", - "#{src_dir}/job.yaml", - "#{src_dir}/job.cgz"] + "#{src_dir}/job.yaml"] FileUtils.cp(files, dst_dir) end end -- 2.23.0

1 0

[PATCH lkp-tests 2/2] spec/submit/link_jobs: replace latency_stats to perf-sched
by Wei Jihui 22 Feb '21

22 Feb '21

commit 4b1eaa97a0d7dd2455929c0734ed88631b1aa263: include/category/benchmark: replace latency_stats to perf-ddsched Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- ...4-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 5457a131..0c3eb69d 100644 --- a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -51,7 +51,7 @@ meminfo: slabinfo: interrupts: lock_stat: -latency_stats: +perf-sched: softirqs: bdi_dev_mapping: diskstats: @@ -103,7 +103,7 @@ monitors: slabinfo: interrupts: lock_stat: - latency_stats: + perf-sched: softirqs: bdi_dev_mapping: diskstats: -- 2.23.0

1 0

[PATCH lkp-tests 1/2] spec/submit: update submit spec result
by Wei Jihui 22 Feb '21

22 Feb '21

all vm testboxs add hdd_partitions Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/job_on_fail/job-3000.yaml | 1 + ...1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml | 1 + spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml | 1 + spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml | 1 + spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml | 1 + spec/submit/matrix/job-1-300-whetstone-double.yaml | 1 + spec/submit/matrix/job-3-300-shell1.yaml | 1 + spec/submit/merge_yaml/job-defaults.yaml | 1 + spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml | 1 + spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml | 1 + spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml | 1 + 11 files changed, 11 insertions(+) diff --git a/spec/submit/job_on_fail/job-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml index 11d89b2b..5d766ba0 100644 --- a/spec/submit/job_on_fail/job-3000.yaml +++ b/spec/submit/job_on_fail/job-3000.yaml @@ -11,6 +11,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 466a2054..5457a131 100644 --- a/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_jobs1/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -34,6 +34,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 kmsg: boot-time: diff --git a/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml b/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml index 9a613c00..d20eed7e 100644 --- a/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs2/job-1000-2-100000ss-add.yaml @@ -15,6 +15,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml b/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml index 8bf52d0f..8df242aa 100644 --- a/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs2/job-10000-2-100000ss-add.yaml @@ -15,6 +15,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml b/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml index 8cb78287..cf3a162b 100644 --- a/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml +++ b/spec/submit/link_jobs2/job-20-2-6000ss-cache.yaml @@ -15,6 +15,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/matrix/job-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml index 78f79fa7..4ba40ba5 100644 --- a/spec/submit/matrix/job-1-300-whetstone-double.yaml +++ b/spec/submit/matrix/job-1-300-whetstone-double.yaml @@ -12,6 +12,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/matrix/job-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml index afe5a953..294cbdd3 100644 --- a/spec/submit/matrix/job-3-300-shell1.yaml +++ b/spec/submit/matrix/job-3-300-shell1.yaml @@ -12,6 +12,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/merge_yaml/job-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml index f72fd6ac..ae2f3c39 100644 --- a/spec/submit/merge_yaml/job-defaults.yaml +++ b/spec/submit/merge_yaml/job-defaults.yaml @@ -9,6 +9,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml index a2355b50..b7b0ef0e 100644 --- a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml +++ b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml @@ -13,6 +13,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml index 8a6317cb..144fba5a 100644 --- a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml +++ b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml @@ -13,6 +13,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: diff --git a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml index 15701ad0..74c4e380 100644 --- a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml +++ b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml @@ -13,6 +13,7 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: "/dev/vda" arch: aarch64 cpufreq_governor: need_kconfig: -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: only store necessary info to es
by Li Yuanchao 22 Feb '21

22 Feb '21

As the config become more, the old way will store more and more info to es, and they are not all necessary, and their changes sometimes will cause conflict with old data in es. So only store necessary info now. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 346352b..76cdd1b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -288,8 +288,8 @@ class MirrorMain end def es_repo_update(git_repo) - repo_info = @git_info[git_repo].merge(@fork_stat[git_repo]) - repo_info.delete(:cur_refs) if repo_info.key?(:cur_refs) + repo_info = { 'git_repo' => git_repo, 'url' => @git_info[git_repo]['url'] } + repo_info = repo_info.merge(@fork_stat[git_repo]) body = { "doc": repo_info, "doc_as_upsert": true -- 2.23.0

1 0

[PATCH v3 compass-ci 4/4] container/mail-robot: update mail model
by Luan Shengde 22 Feb '21

22 Feb '21

[why] key work is renamed: 'my oss commit' => 'my_oss_commit' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-fail-eamil.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 3597159..3169235 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -11,17 +11,17 @@ def email_err_message(message) You should add a commit url for the 'apply account' email. For example: - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + my_oss_commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… EMAIL_MESSAGE when 'URL_PREFIX_ERR' err_message = <<~EMAIL_MESSAGE Please add a correct prefix for the commit url. - my oss commit + my_oss_commit For example: - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + my_oss_commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… EMAIL_MESSAGE when 'NOT_REGISTERED' err_message = <<~EMAIL_MESSAGE -- 2.23.0

1 0

[PATCH v3 compass-ci 3/4] container/mail-robot: add login message
by Luan Shengde 22 Feb '21

22 Feb '21

add login command for account VM server. case forwarded user info has key account_vm with value: yes/true, the email will send the login command. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/assign-account-email.rb | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index b71b3fe..2171de7 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -3,13 +3,26 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_email(my_info) +def build_apply_account_email(my_info, account_info, assign_account_vm) email_greeting = if my_info['bisect'] 'We have automatically created the following information for you.' else 'Thank you for joining us.' end + login_msg = <<~LOGIN_MSG + + You can use the following command to login the jumper server: + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + LOGIN_MSG + + login_account_vm = assign_account_vm ? login_msg : '' + email_msg = <<~EMAIL_MESSAGE To: #{my_info['my_email']} Subject: [compass-ci] Account Ready @@ -17,8 +30,8 @@ def build_apply_account_email(my_info) Dear #{my_info['my_name']}, #{email_greeting} - - You can use the following info to submit jobs: + #{login_account_vm} + You need to do the following configurations before submitting a job: notice: (1-2) are ONE-TIME setup -- 2.23.0

1 0

[PATCH v3 compass-ci 2/4] container/mail-robot: add parsing forwarded email
by Luan Shengde 22 Feb '21

22 Feb '21

for forwarded 'apply account' email, my_email/my_name is store in a fixed format in the mail content. extract a list of users info from the mail content update regular expression: 'my oss commit' is renamed to 'my_oss_commit'. both support 'my oss commit' and 'my_oss_commit' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 92db1da..272f1a1 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -57,18 +57,27 @@ class ParseApplyAccountEmail end def extract_users - users = extract_mail_content_body.split(/\r|\n/) + users_info = [] + + users = extract_mail_content_body.split(/---+/) users.delete('') - users + users.each do |user| + user_info = YAML.safe_load(user) + + users_info << user_info if user_info.include?('my_email') + end + + return users_info end def extract_commit_url mail_content_line = extract_mail_content_body.gsub(/\n/, '') - # the commit url should be headed with a prefix: my oss commit + # the commit url should be headed with a prefix: my_oss_commit # the commit url should be in a standart format, example: - # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… - unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) - raise 'URL_PREFIX_ERR' unless mail_content_line.match?(%r{my oss commit:\s*https?://}) + # my_oss_commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + # the prefix is renamed to 'my_oss_commit', but 'my oss commit' is still supportted. + unless mail_content_line.match?(%r{my[ _]oss[ _]commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + raise 'URL_PREFIX_ERR' unless mail_content_line.match?(%r{my[ _]oss[ _]commit:\s*https?://}) raise 'NO_COMMIT_URL' unless mail_content_line.match?(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/4] container/mail-robot: add handle forwarded my_name/my_email(s)
by Luan Shengde 22 Feb '21

22 Feb '21

[why] the forwarded email from account issuer may will contain my_name/my_email(s). it allows contain multi my_name/my_email(s), just loop them. the forwarded email content may like: --- my_name: name_1 my_email: email_1 account_vm: true/false/yes/no --- my_name: name_2 my_email: email_2 --- available values for key_word: account_vm - yes/no - true/false only if the section for a user has a key_word account_vm, and the key_word has a value: yes/true, the replied email from mail robot will contains the login command for account vm server. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 66 ++++++++++++++++++++--- 1 file changed, 58 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index e8288d7..f3f61c3 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -55,6 +55,9 @@ class ApplyAccount @es_host = @send_mail_host @es_port = ES_PORT + # email info file for account issuers. + @account_issuer = YAML.load_file '/etc/compass-ci/account-issuer.yaml' + @my_info = {} end @@ -63,11 +66,36 @@ class ApplyAccount # in order to successfully send email for failed parsing # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] - parse_mail_content - apply_my_account - store_account_info - send_mail('') + # for the forwarded email, it contains my_email/my_name .etc. + # all emails from account issuer will be treated as forwarded emails. + # the mail_content may like: + # --- + # my_name: name_1 + # my_name: email_1 + # account_vm: true/false/yes/no + # --- + # my_name: name_2 + # my_email: email_2 + # --- + # the forwarded email allowed to contain multi my_email/my_name(s) + # in this case, we will loop them + if @account_issuer.include? @my_info['my_email'] + users_info = forward_users + users_info.each do |user_info| + # for forwarded email for multi users, avoid rezidual information from the last, + # need to clear the old data for my_info. + @my_info.clear + assign_account_vm = user_info.delete('account_vm') + @my_info.update user_info + + applying_account(assign_account_vm) + sleep 5 + end + else + parse_mail_content + applying_account(false) + end rescue StandardError => e puts e.message puts e.backtrace @@ -75,6 +103,24 @@ class ApplyAccount send_mail(e.message) end + def forward_users + forward_email_content = ParseApplyAccountEmail.new(@mail_content) + users_info = forward_email_content.extract_users + + users_info.clone.each_index do |i| + users_info[i]['my_ssh_pubkey'] = [] + users_info[i]['lab'] = ENV['lab'] + end + + users_info + end + + def applying_account(assign_account_vm) + account_info = apply_my_account + store_account_info + send_mail('', account_info, assign_account_vm) + end + def parse_mail_content parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) @@ -93,7 +139,9 @@ class ApplyAccount apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? apply_info.update my_account_es apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + if my_ssh_pubkey_new + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + end @my_info.update apply_info apply_info['is_update_account'] = true apply_info @@ -102,14 +150,15 @@ class ApplyAccount def apply_my_account my_account_es = read_my_account_es apply_info = {} + if my_account_es build_apply_info(apply_info, my_account_es) else my_token = %x(uuidgen).chomp @my_info['my_token'] = my_token apply_info.update @my_info + apply_info['enable_login'] = true end - apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end @@ -118,6 +167,7 @@ class ApplyAccount acct_info = apply_account.apply_jumper_account @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + acct_info end def store_account_info @@ -125,9 +175,9 @@ class ApplyAccount es.put_source_by_id(@my_info['my_email'], @my_info) end - def send_mail(error_message) + def send_mail(error_message, account_info, assign_account_vm) email_message = if error_message.empty? - build_apply_account_email(@my_info) + build_apply_account_email(@my_info, account_info, assign_account_vm) else build_apply_account_fail_email(@my_info, error_message) end -- 2.23.0

1 0

[PATCH lkp-tests] jobs/: add os-arch-version-mount.yaml for multi-dimension test
by Li Ping 22 Feb '21

22 Feb '21

1 0

[PATCH compass-ci] sbin/compare: update help message
by Lu Weitao 22 Feb '21

22 Feb '21

[Why] the option -d/--dimensions no specific instructions [How] Usage: ... options: -d, --dimension dimensions this option activate group compare mode, require filter and dimenssion filter like: os=openeuler,centos suite=iperf dimensions like: "tbox_group os_arch os ..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/compare | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/sbin/compare b/sbin/compare index e8d1ac4..2ee3268 100755 --- a/sbin/compare +++ b/sbin/compare @@ -11,8 +11,8 @@ # compare "id=6000,6001" "id=7000,7001" # compare "commit=a12d232e" "commit=b3bacc31" # compare "os=debian" "os=centos" -c "suite=iperf" -# compare "os=centos" -d "os_version os_arch" -# compare "os=centos" -d "os_version os_arch" --theme="classic" +# compare os=centos -d "os_version os_arch" +# compare os=centos suite=iperf -d "os_version os_arch" --theme="classic" require 'optparse' require_relative '../lib/compare.rb' @@ -37,7 +37,8 @@ opt_parser = OptionParser.new do |opts| common_conditions = c end - opts.on('-d', '--dimension dimensions', 'dimensions to group compare: "tbox_group os_arch os ..."') do |d| + opts.on('-d', '--dimension dimensions', 'this option activate group compare mode, require filter and dimenssion', + 'filter like: os=openeuler,centos suite=iperf', 'dimensions like: "tbox_group os_arch os ..."') do |d| dimensions = d is_group = true end -- 2.23.0

1 0

[PATCH compass-ci] container/fluentd-base/Dockerfile: fix failed to build container
by Liu Yinsi 22 Feb '21

22 Feb '21

[error] root@taishan200-2280-2s64p-256g--a7 /c/compass-ci/container/fluentd-base# ./build ... ERROR: Error installing async: ERROR: Failed to build gem native extension. current directory: /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5/ext/nio4r /usr/bin/ruby -I /usr/lib/ruby/2.6.0 -r ./siteconf20210222-482-mws95t.rb extconf.rb checking for unistd.h... yes checking for linux/aio_abi.h... no checking for linux/io_uring.h... no checking for sys/select.h... yes checking for port_event_t in poll.h... no checking for sys/epoll.h... yes checking for sys/event.h... no checking for port_event_t in port.h... no checking for sys/resource.h... yes creating Makefile current directory: /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5/ext/nio4r make "DESTDIR=" clean current directory: /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5/ext/nio4r make "DESTDIR=" compiling bytebuffer.c compiling monitor.c compiling nio4r_ext.c In file included from nio4r_ext.c:6: ../libev/ev.c:479:11: fatal error: linux/version.h: No such file or directory 479 | # include <linux/version.h> | ^~~~~~~~~~~~~~~~~ compilation terminated. make: *** [Makefile:245: nio4r_ext.o] Error 1 make failed, exit code 2 Gem files will remain installed in /usr/lib/ruby/gems/2.6.0/gems/nio4r-2.5.5 for inspection. Results logged to /usr/lib/ruby/gems/2.6.0/extensions/aarch64-linux/2.6.0/nio4r-2.5.5/gem_make.out The command '/bin/sh -c umask 002 && gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && gem install fluentd && gem install fluent-plugin-rabbitmq && gem install fluent-plugin-elasticsearch && gem install fluent-plugin-tail-ex && gem install fluent-plugin-tail-multiline && gem install json && gem install async && gem install webrick && gem install io-console && gem install etc' returned a non-zero code: 1 [how] 'gem install async' in Dockerfile cause error, apk add linux-headers can fix it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/fluentd-base/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/fluentd-base/Dockerfile b/container/fluentd-base/Dockerfile index 08bcd9c..2d181f5 100644 --- a/container/fluentd-base/Dockerfile +++ b/container/fluentd-base/Dockerfile @@ -10,7 +10,7 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN apk update RUN apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' 'linux-headers' RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ -- 2.23.0

1 0

[PATCH v2 compass-ci] doc: optimize os_mount.md
by Yu Chuan 22 Feb '21

22 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 207 +++++++++++++++++++++++++++++--------------- 1 file changed, 137 insertions(+), 70 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..ebe8566b9443 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,9 +1,9 @@ # Summary ========= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: +The below types are supported: - nfs - cifs - initramfs @@ -11,84 +11,151 @@ It has the following optional values: - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. +# os_mount=local +================ + +## description + +If you specifies os_mount=local, the rootfs type of your job will be a logical volume in the testbox. + +> logical volume format: `/dev/mapper/os-${os}_${os_arch}_${os_version}` + +> optional vars in ${kernel_append_root}: + - ${save_root_partition}, ${use_root_partition} + - purpose: + - when os_mount=local, rootfs is stored in the logical volume on the disk, + but it will be wiped by default in the preprocessing action performed by each job. + just is initrd stage. + - so we need a solution to solve scenarios that rootfs data needs to be persisted. + - how to implement: + - design a unique string for your job, it will be used as a unique symbol for your job. + the rootfs data of your job will be persistently saved to the corresponding testbox. + - so, when you need to use the rootfs data of this job in subsequent jobs, you only need + to use this unique string to submit subsequent-job.yaml, and then the subsequent-job.yaml + will be executed based on the prsistent rootfs data. + - how to use: + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + this job-20210218 so that it can be used by the subsequent jobs. So you need add the + follow field in your job-20210218.yaml. + ``` + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + ``` + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml, So you need add the follow field in your job-20210219.yaml + ``` + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + - description: + - ${save_root_partition} and ${use_root_partition} must be the same, and obviously, they need be unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox ``` dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} boot ``` 3. dracut step of boot ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi + rsync_src_lv() + { + lvdisplay ${SRC_LV} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${SRC_LV}) os || exit + + # rsync nfsroot to ${SRC_LV} + mount -t nfs ${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} /mnt + mkdir /mnt1 && mount ${SRC_LV} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${SRC_LV} to readonly + lvchange --permission r ${SRC_LV} + } + + check_src_lv() + { + lvdisplay ${SRC_LV} > /dev/null || exit + } + + snapshot_boot_lv() + { + [ "$SRC_LV" == "$BOOT_LV" ] && return + + lvremove --force ${BOOT_LV} + lvcreate --size 10G --name $(basename ${BOOT_LV}) --snapshot ${SRC_LV} || exit + } + + no_save_no_use() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03" + + rsync_src_lv ${SRC_LV} + } + + only_save() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + rsync_src_lv ${SRC_LV} + } + + only_use() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03" + + check_src_lv ${SRC_LV} + } + + save_and_use() + { + export SRC_LV="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + export BOOT_LV="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + check_src_lv ${SRC_LV} + } + + main() + { + [ -z ${use_root_partition} ] && [ -z ${save_root_partition} ] && no_save_no_use + + [ -z ${use_root_partition} ] && [ -n ${save_root_partition} ] && only_save + + [ -n ${use_root_partition} ] && [ -z ${save_root_partition} ] && only_use + + [ -n ${use_root_partition} ] && [ -n ${save_root_partition} ] && save_and_use + + snapshot_boot_lv + boot_from_lv + } + + main() ``` 4. execute the job -- 2.23.0

2 4

[PATCH compass-ci] doc: update document
by Luan Shengde 22 Feb '21

22 Feb '21

rename key word: my oss commit => my_oss_commit Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 4 ++-- doc/manual/apply-account.md | 6 +++--- doc/manual/apply-account.zh.md | 6 +++--- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/mail-robot.md b/doc/mail-robot.md index 6158b65..4d38e78 100644 --- a/doc/mail-robot.md +++ b/doc/mail-robot.md @@ -17,9 +17,9 @@ send mail to: compass-ci-robot(a)qq.com subject: apply account mail content - my oss commit: commit_url + my_oss_commit: commit_url example: - my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + my_oss_commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… attahcment ssh pub_key diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 974fe63..542251d 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -4,7 +4,7 @@ You'll need an account to submit jobs. The account is mainly for tracking your r We have limited pool of testboxes, so there have to be some usage control. We offer free accounts to OSS (Open Source Software) contributors where the OSS project is in our test coverage. -You can prove yourself as OSS contributor by providing "my oss commit" in the below email. +You can prove yourself as OSS contributor by providing "my_oss_commit" in the below email. We offer free accounts to our collaborators too. If you are one of our collaborators, you should know the collaboration channel to apply account. @@ -19,7 +19,7 @@ Email template: To: compass-ci-robot(a)qq.com Subject: apply account - my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} + my_oss_commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} {{ attach your ssh pubkey file(s), e.g. ~/.ssh/id_rsa.pub }} @@ -29,7 +29,7 @@ Email template: - the email name should be in English or Chinese Pinyin, as: "David Rientjes" in the template. -- my oss commit: a git commit URL that has your name and email. +- my_oss_commit: a git commit URL that has your name and email. The git repo should be in our test coverage and registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). If your contributed OSS project is not in upstream-repos, you may refer to diff --git a/doc/manual/apply-account.zh.md b/doc/manual/apply-account.zh.md index 5e6605d..849364f 100644 --- a/doc/manual/apply-account.zh.md +++ b/doc/manual/apply-account.zh.md @@ -4,7 +4,7 @@ 因为目前我们的测试机资源池有限，因此需要做一些使用限制。我们为平台测试所覆盖的开源软件（OSS）的贡献者提供了免费的账号。 -你可以通过在下面邮件中提供‘my oss commit’来验明正身您是一个开源软件（OSS）的贡献者。 +你可以通过在下面邮件中提供‘my_oss_commit’来验明正身您是一个开源软件（OSS）的贡献者。我们同时也为我们的合作者提供了免费的账号。这种情况需要提供用的邮件，名字，ssh公钥，以及申请测试机的目的。 @@ -18,7 +18,7 @@ 收件人: compass-ci-robot(a)qq.com 邮件标题: apply account - my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} + my_oss_commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} {{ 将您的ssh公钥添加为邮件附件，例如 ~/.ssh/id_rsa.pub }} @@ -26,7 +26,7 @@ - 使用您的信息替换{{ }}中的内容 - 邮件名应该使用英文或中文拼音，例如模板中的“David Rientjes” -- my oss commit: 使用您的邮件和名字提交的代码的地址。 +- my_oss_commit: 使用您的邮件和名字提交的代码的地址。您的仓库需要在平台测试覆盖的范围内，并且注册到： -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends/sysbench-mysql
by Wei Jihui 22 Feb '21

22 Feb '21

1. when pack pkg/sysbench-mysql, it need mysql so add libmariadb-dev(mariadb-devel) at sysbench-mysql-dev 2. when exec test/sysbench-mysql, it will install mysql at client, no need install as depends, so delete libmariadb3 at sysbench-mysql Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/sysbench-mysql | 1 - distro/depends/sysbench-mysql-dev | 3 ++- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/distro/depends/sysbench-mysql b/distro/depends/sysbench-mysql index b0d33ffe..a248fd07 100644 --- a/distro/depends/sysbench-mysql +++ b/distro/depends/sysbench-mysql @@ -1,4 +1,3 @@ libnuma1 xfsprogs ethtool -libmariadb3 diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev index 65633b33..992db26d 100644 --- a/distro/depends/sysbench-mysql-dev +++ b/distro/depends/sysbench-mysql-dev @@ -1,2 +1,3 @@ +libtool pkg-config -mysql-server +libmariadb-dev -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix fs parameter parse
by Wei Jihui 22 Feb '21

22 Feb '21

[input] fs: ext4 [error] this fs will parse error by lkp-tests/params/fs fs: fs: &1 fs: ext4 [why] job.rb will parse fs twice by lkp-test/params/fs: /(.*)/{ fs: $1} this will create two fs. [how] only parse all parameters once, that is only exec expand_params once. not at the first time is for job should split array params at the first time. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index eacb7ec9..7fc7cb5b 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -447,7 +447,7 @@ class Job job = deepcopy(@job) job['___'] = nil @jobx = job - if @job.include?(:expand_params) + if @job.include?(:expand_params) && !first_time expand_params @job = deepcopy(@jobx) @job.delete('___') -- 2.23.0

2 1

[PATCH v2 compass-ci 1/4] container/mail-robot: add handle forwarded my_name/my_email(s)
by Luan Shengde 22 Feb '21

22 Feb '21

[why] for forwarded email from account issuer, the email content will contain my_name/my_email(s). a forwarded email may contain multi my_name/my_email(s), just loop them. the forwarded email content may like: --- my_name: name_1 my_email: email_1 account_vm: true/false/yes/no --- my_name: name_2 my_email: email_2 --- the replied 'apply account' email will return login command for account VM server if account_vm has a value: yes/true add store lab to ES Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 65 ++++++++++++++++++++--- 1 file changed, 57 insertions(+), 8 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index e8288d7..818f900 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -55,6 +55,9 @@ class ApplyAccount @es_host = @send_mail_host @es_port = ES_PORT + # email info file for account issuers. + @account_issuer = YAML.load_file '/etc/compass-ci/account-issuer.yaml' + @my_info = {} end @@ -63,11 +66,35 @@ class ApplyAccount # in order to successfully send email for failed parsing # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] - parse_mail_content - apply_my_account - store_account_info - send_mail('') + # for the forwarded email, it contains my_email/my_name .etc. + # all emails from account issuer will be treated as forwarded emails. + # the mail_content may like: + # --- + # my_name: name_1 + # my_name: email_1 + # account_vm: true/false/yes/no + # --- + # my_name: name_2 + # my_email: email_2 + # --- + # the forwarded email allowed to contain multi my_email/my_name(s) + # in this case, we will loop them + if @account_issuer.include? @my_info['my_email'] + users_info = forward_users + users_info.each do |user_info| + # for forwarded email for multi user, avoid rezidual information from the last. + @my_info.clear + assign_account_vm = user_info.delete('account_vm') + @my_info.update user_info + + applying_account(assign_account_vm) + sleep 5 + end + else + parse_mail_content + applying_account(false) + end rescue StandardError => e puts e.message puts e.backtrace @@ -75,6 +102,24 @@ class ApplyAccount send_mail(e.message) end + def forward_users + forward_email_content = ParseApplyAccountEmail.new(@mail_content) + users_info = forward_email_content.extract_users + + users_info.clone.each_index do |i| + users_info[i]['my_ssh_pubkey'] = [] + users_info[i]['lab'] = ENV['lab'] + end + + users_info + end + + def applying_account(assign_account_vm) + account_info = apply_my_account + store_account_info + send_mail('', account_info, assign_account_vm) + end + def parse_mail_content parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) @@ -93,7 +138,9 @@ class ApplyAccount apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? apply_info.update my_account_es apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + if my_ssh_pubkey_new + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + end @my_info.update apply_info apply_info['is_update_account'] = true apply_info @@ -102,6 +149,8 @@ class ApplyAccount def apply_my_account my_account_es = read_my_account_es apply_info = {} + apply_info['enable_login'] = true + if my_account_es build_apply_info(apply_info, my_account_es) else @@ -109,7 +158,6 @@ class ApplyAccount @my_info['my_token'] = my_token apply_info.update @my_info end - apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end @@ -118,6 +166,7 @@ class ApplyAccount acct_info = apply_account.apply_jumper_account @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + acct_info end def store_account_info @@ -125,9 +174,9 @@ class ApplyAccount es.put_source_by_id(@my_info['my_email'], @my_info) end - def send_mail(error_message) + def send_mail(error_message, account_info, assign_account_vm) email_message = if error_message.empty? - build_apply_account_email(@my_info) + build_apply_account_email(@my_info, account_info, assign_account_vm) else build_apply_account_fail_email(@my_info, error_message) end -- 2.23.0

2 4

[PATCH compass-ci] doc: optimize os_mount.md
by Yu Chuan 21 Feb '21

21 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 181 +++++++++++++++++++++++++++++--------------- 1 file changed, 120 insertions(+), 61 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..b06b27237c75 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,9 +1,9 @@ # Summary ========= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: +The below types are supported: - nfs - cifs - initramfs @@ -11,84 +11,143 @@ It has the following optional values: - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local + - submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + - submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + - submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + - submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + - submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local -# Optional Values -================= +# os_mount=local +================ -## Work flow when os_mount=local +## description -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. +If you specifies os_mount=local, the rootfs type of your job will be a logical volume in the testbox. - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. +> logical volume format: `/dev/mapper/os-${os}_${os_arch}_${os_version}` + +> optional vars in ${kernel_append_root}: + - ${save_root_partition}, ${use_root_partition} + - you can save the rootfs data of any job, and use it in subsequent job. + - ${save_root_partition} and ${use_root_partition} must be the same, and obviously, they need be unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + - demo kernel_append_root field of job.yaml: + ``` + # demo 1. save the rootfs data of this job + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210219 + + # demo 2. use the rootfs data of previous job + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + + # demo 3. use the rootfs data of previous job, and save the rootfs data of this job + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` + +## work flow + +1. user submit job + - job fields: + - os: openeuler + - os_arch: aarch64 + - os_version: 20.03 + - os_mount: local + - kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 2. scheduler return the ipxe_str to testbox ``` dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} boot ``` 3. dracut step of boot ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { + copy_nfsroot_to_src_lv() + { + src_lv=$1 + + lvdisplay ${src_lv} > /dev/null || { # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 + lvcreate -L 10G -n $(basename ${src_lv}) os || exit 1 - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 + # rsync nfsroot to ${src_lv} + mount -t nfs ${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 cp -a /mnt/. /mnt1/ umount /mnt /mnt1 - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} + # change premission of ${src_lv} to readonly + lvchange -p r ${src_lv} } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi + } + + check_src_lv() + { + lvdisplay $1 > /dev/null || exit 1 + } + + clone_boot_lv_from_src_lv() + { + src_lv=$1 + boot_lv=$2 + + lvremove -f ${boot_lv} + lvcreate ${boot_lv} from ${src_lv} || exit 1 + } + + no_save_no_use() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + + copy_nfsroot_to_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + only_save() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${timestamp}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + copy_nfsroot_to_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + only_use() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + + check_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + save_and_use() + { + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + + check_src_lv ${src_lv} + clone_boot_lv_from_src_lv ${src_lv} ${boot_lv} + boot_from_lv ${boot_lv} + } + + main() + { + [ -z ${use_root_partition} ] && [ -z ${save_root_partition} ] && no_save_no_use + + [ -z ${use_root_partition} ] && [ -n ${save_root_partition} ] && only_save + + [ -n ${use_root_partition} ] && [ -z ${save_root_partition} ] && only_use + + [ -n ${use_root_partition} ] && [ -n ${save_root_partition} ] && save_and_use + } + + main() ``` 4. execute the job -- 2.23.0

2 10

[PATCH v2 compass-ci] sbin/build-my-info: fix no key: my_ssh_pubkey
by Luan Shengde 20 Feb '21

20 Feb '21

[why] it will read my_ssh_pubkey and check it when submit jobs. $ submit -m -c job.yaml -i ~/lkp-tests/jobs/ssh-on-fail.yaml submit_id=f87d2951-c226-4e51-a3e4-c71818a4c974 submit job.yaml failed, got job id=0, error: Missing hash key: "my_ssh_pubkey" [how] add initialize my_ssh_pubkey when building my_info. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/build-my-info.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index c7a7798..ecfcb42 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -12,6 +12,7 @@ def build_my_info(option) info_es = build_my_info.search_my_info info_es.update option info_es['my_token'] = %x(uuidgen).chomp if info_es['my_token'].nil? + info_es['my_ssh_pubkey'] = [] build_my_info.config_my_info(info_es) end -- 2.23.0

1 0

[PATCH v2 compass-ci] answerback-email: add store lab to ES
by Luan Shengde 20 Feb '21

20 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 8dceee1..f573f0f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -40,7 +40,8 @@ my_info = { 'my_commit_url' => nil, 'my_token' => nil, 'my_login_name' => nil, - 'my_ssh_pubkey' => [] + 'my_ssh_pubkey' => [], + 'lab' => LAB } # stdin_info is used to store infos added with option: @@ -179,7 +180,6 @@ def apply_account(my_info, conf_info) apply_info = {} apply_info.update my_info apply_info.update conf_info - apply_info['lab'] = LAB assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" account_info_str = RestClient.post assign_account_url, apply_info.to_json -- 2.23.0

1 0

[PATCH compass-ci] auto-submit: use the lib script to config the account
by Cao Xueliang 20 Feb '21

20 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/auto_submit | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 6d29543..b408158 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -7,8 +7,7 @@ require 'bunny' require 'json' require 'yaml' require 'set' -require_relative '../lib/build_my_info_client' -require "#{ENV['CCI_SRC']}/container/defconfig" +require_relative '../lib/config_account' # receive message and auto submit job class AutoSubmit @@ -130,23 +129,7 @@ class AutoSubmit end end -def my_email - names = Set.new %w[auto-submit] - - defaults = relevant_defaults(names) - defaults['auto-submit']['my_email'] -end - -def config_yaml - build_my_info = BuildMyInfo.new(my_email) - - my_info = build_my_info.search_my_info - build_my_info.config_default_yaml(my_info) - build_my_info.config_lab_yaml(my_info) -end - -# fix verification problems when submit jobs by auto-submit service -config_yaml +config_yaml('auto-submit') auto_submitter = AutoSubmit.new auto_submitter.listen -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] ner/mail-robot: optimize start
by Luan Shengde 20 Feb '21

20 Feb '21

attach the directory instead of the single file. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/start b/container/mail-robot/start index 3827e2c..bd2a62d 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -31,7 +31,7 @@ cmd = %w[ ] + env + %W[ -v /c/upstream-repos:/c/upstream-repos:ro -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw - -v /etc/compass-ci/conf/sec_admins.yaml:/etc/compass-ci/conf/sec_admins.yaml:ro + -v /etc/compass-ci:/etc/compass-ci:ro -v #{ENV['CCI_SRC']}:/c/compass-ci:rw -w /c/compass-ci/container/mail-robot mail-robot -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/mail-robot: add login command
by Luan Shengde 20 Feb '21

20 Feb '21

add login command for account VM server. case user info has key account_vm with value: yes/true, the email will send the login command. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/assign-account-email.rb | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index b71b3fe..8e3aa6e 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -3,13 +3,26 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_email(my_info) +def build_apply_account_email(my_info, account_info, assign_account_vm) email_greeting = if my_info['bisect'] 'We have automatically created the following information for you.' else 'Thank you for joining us.' end + logging_msg = <<~LOGIN_MSG + + You can use the following command to login the jumper server: + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + LOGIN_MSG + + logging_account_vm = assign_account_vm ? logging_msg : '' + email_msg = <<~EMAIL_MESSAGE To: #{my_info['my_email']} Subject: [compass-ci] Account Ready @@ -17,7 +30,7 @@ def build_apply_account_email(my_info) Dear #{my_info['my_name']}, #{email_greeting} - + #{logging_account_vm} You can use the following info to submit jobs: notice: -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container/mail-robot: add parsing forwarded email
by Luan Shengde 20 Feb '21

20 Feb '21

for forwarded 'apply account' email, my_email/my_name is store in a fixed format in the mail content. extract a list of users info from the mail content Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/parse-apply-account-email.rb | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 92db1da..3de2fb5 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -57,9 +57,18 @@ class ParseApplyAccountEmail end def extract_users - users = extract_mail_content_body.split(/\r|\n/) + users_info = [] + + users = extract_mail_content_body.split(/---+/) users.delete('') - users + users.each do |user| + user_info = YAML.safe_load(user) + next unless user_info.include?('my_email') + + users_info << user_info + end + + return users_info end def extract_commit_url -- 2.23.0

1 0

[PATCH compass-ci] delimter: Use delimiter account submit the bisect job
by Cao Xueliang 20 Feb '21

20 Feb '21

Add the config_account.rb to create the account info in container, it will use the specified account when submit the job in container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/start | 1 + lib/config_account.rb | 22 ++++++++++++++++++++++ src/delimiter.rb | 2 ++ 3 files changed, 25 insertions(+) create mode 100644 lib/config_account.rb diff --git a/container/delimiter/start b/container/delimiter/start index 669923b..3cff23a 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -31,6 +31,7 @@ cmd = %w[ -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro -v /etc/localtime:/etc/localtime:ro + -v /etc/compass-ci/register:/etc/compass-ci/register:ro -w #{DEFAULT_CCI}/src delimiter ] diff --git a/lib/config_account.rb b/lib/config_account.rb new file mode 100644 index 0000000..dc0067c --- /dev/null +++ b/lib/config_account.rb @@ -0,0 +1,22 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "./build_my_info_client" +require_relative "../container/defconfig" + +def my_email(account) + names = %W(#{account}) + + defaults = relevant_defaults(names) + defaults[account]['my_email'] +end + +def config_yaml(account) + build_my_info = BuildMyInfo.new(my_email(account)) + + my_info = build_my_info.search_my_info + build_my_info.config_default_yaml(my_info) + build_my_info.config_lab_yaml(my_info) +end diff --git a/src/delimiter.rb b/src/delimiter.rb index bd40775..1ea5de5 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -3,8 +3,10 @@ # frozen_string_literal: true require_relative './delimiter/delimiter' +require_relative '../lib/config_account' begin + config_yaml('delimiter') delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: fix undefined method `/'
by Cao Xueliang 20 Feb '21

20 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index d43785b..104f098 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -47,7 +47,7 @@ class MailBisectResult Hi #{(a)git_commit.author_name}, We found some error/warning(s) and the first bad commit in the below project: - git url: #{@upstream_url/@commit_id} + git url: #{@upstream_url}/#{@commit_id} git commit: #{@commit_id} ("#{(a)git_commit.subject}") All error/warning(s) (new ones prefixed by >>): -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] sparrow: separate service config
by Cao Xueliang 20 Feb '21

20 Feb '21

Extract service config from defaults/yaml to the service/yaml. Read service/yaml in defconfig.rb and defconfig.sh. Add scheduler/yaml for scheduler set the content to the job. Adjust the order of service config. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/defconfig.rb | 1 + container/defconfig.sh | 1 + sparrow/3-code/dev-env | 80 +++++++++++++++++++++++++++++------------- 3 files changed, 57 insertions(+), 25 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index 602b498..b121de8 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -8,6 +8,7 @@ require 'yaml' def cci_defaults hash = {} Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/service/*.yaml', '/etc/compass-ci/accounts/*.yaml', '/etc/compass-ci/register/*.yaml', "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| diff --git a/container/defconfig.sh b/container/defconfig.sh index 7c575ef..c33a790 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -10,6 +10,7 @@ load_cci_defaults() yaml_file=( /etc/compass-ci/defaults/*.yaml + /etc/compass-ci/service/*.yaml /etc/compass-ci/accounts/*.yaml "$HOME"/.config/compass-ci/defaults/*.yaml ) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8812839..cc94e01 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -11,52 +11,82 @@ server_ip=$(awk '/^server_ip:\s/ {print $2; exit}' /etc/compass-ci/setup.yaml) server_name=$(hostname | cut -f1 -d.) : ${sched_host:=$server_ip} : ${sched_port:=3000} -: ${os_http_host:=$server_ip} -: ${initrd_http_host:=$server_ip} -: ${initrd_http_port:=8800} +: ${ES_HOST:=$server_ip} +: ${ES_PORT:=9200} +: ${OS_HTTP_HOST:=$server_ip} +: ${SEND_MAIL_HOST:=$server_ip} +: ${SEND_MAIL_PORT:=49000} +: ${SRV_HTTP_HOST:=$server_ip} +: ${SRV_HTTP_PORT:=11300} +: ${LOGGING_ES_HOST:=$server_ip} +: ${LOGGING_ES_PORT:=9202} +: ${INITRD_HTTP_HOST:=$server_ip} +: ${INITRD_HTTP_PORT:=8800} +: ${ASSIST_RESULT_HOST:=$server_ip} +: ${ASSIST_RESULT_PORT:=8102} +: ${RESULT_WEBDAV_HOST:=$server_ip} +: ${RESULT_WEBDAV_PORT:=3080} : ${MASTER_FLUENTD_HOST:=$server_ip} : ${MASTER_FLUENTD_PORT:=24224} : ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} -: ${LOGGING_ES_HOST:=$server_ip} -: ${LOGGING_ES_PORT:=9202} -: ${SEND_MAIL_PORT:=49000} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} -: ${ASSIST_RESULT_PORT:=8102} -: ${RESULT_WEBDAV_PORT:=3080} -: ${ES_HOST:=$server_ip} -: ${ES_PORT:=9200} -: ${SRV_HTTP_PORT:=11300} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF +lab: nolab +EOF + +mkdir -p /etc/compass-ci/service +cat > /etc/compass-ci/service/service-env.yaml <<EOF +ES_HOST: $ES_HOST +ES_PORT: $ES_PORT SCHED_HOST: $sched_host SCHED_PORT: $sched_port LKP_SERVER: $server_ip GIT_SERVER: $GIT_SERVER -TASKQUEUE_HOST: $sched_host -OS_HTTP_HOST: $os_http_host -INITRD_HTTP_HOST: $initrd_http_host -INITRD_HTTP_PORT: $initrd_http_port +OS_HTTP_HOST: $OS_HTTP_HOST GITCACHE_HOST: $gitcache_host GITCACHE_PORT: $gitcache_port +SRV_HTTP_HOST: $SRV_HTTP_HOST +SRV_HTTP_PORT: $SRV_HTTP_PORT +TASKQUEUE_HOST: $sched_host +SEND_MAIL_HOST: $SEND_MAIL_HOST +SEND_MAIL_PORT: $SEND_MAIL_PORT +LOGGING_ES_HOST: $LOGGING_ES_HOST +LOGGING_ES_PORT: $LOGGING_ES_PORT +INITRD_HTTP_HOST: $INITRD_HTTP_HOST +INITRD_HTTP_PORT: $INITRD_HTTP_PORT +ASSIST_RESULT_HOST: $ASSIST_RESULT_HOST +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +RESULT_WEBDAV_HOST: $RESULT_WEBDAV_HOST +RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT DOCKER_REGISTRY_HOST: $DOCKER_REGISTRY_HOST DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT -LOGGING_ES_HOST: $LOGGING_ES_HOST -LOGGING_ES_PORT: $LOGGING_ES_PORT -SEND_MAIL_PORT: $SEND_MAIL_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT -LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci -ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT -RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT -lab: nolab -ES_HOST: $ES_HOST -ES_PORT: $ES_PORT -SRV_HTTP_PORT: $SRV_HTTP_PORT +LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +EOF + +mkdir -p /etc/compass-ci/scheduler +cat > /etc/compass-ci/scheduler/local-testbox-env.yaml <<-EOF +LKP_SERVER: +GIT_SERVER: +GITCACHE_HOST: +GITCACHE_PORT: +SRV_HTTP_HOST: +SRV_HTTP_PORT: +SEND_MAIL_HOST: +SEND_MAIL_PORT: +INITRD_HTTP_HOST: +INITRD_HTTP_PORT: +ASSIST_RESULT_HOST: +ASSIST_RESULT_PORT: +RESULT_WEBDAV_HOST: +RESULT_WEBDAV_PORT: EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] scheduler: add testbox env in scheduler
by Cao Xueliang 20 Feb '21

20 Feb '21

We add the testbox env use the submit script in lkp-tests, but the submit don't need to do that, so, we add the testbox env in scheduler. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 2 ++ src/scheduler/testbox_env.cr | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+) create mode 100644 src/scheduler/testbox_env.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index 1afc22f..c3eeaf4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "scheduler/testbox_env.cr" require "../scheduler/elasticsearch_client" require "./json_logger" @@ -52,6 +53,7 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @hash.merge!(testbox_env) @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new diff --git a/src/scheduler/testbox_env.cr b/src/scheduler/testbox_env.cr new file mode 100644 index 0000000..bf8cd8a --- /dev/null +++ b/src/scheduler/testbox_env.cr @@ -0,0 +1,33 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + def get_service_env + hash = Hash(String, JSON::Any).new + yaml_any = File.open("/etc/compass-ci/service/service-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + return Hash(String, JSON::Any).from_json(yaml_any.to_json) + end + + def testbox_env(flag = "local") + service_env = get_service_env + hash = Hash(String, JSON::Any).new + + yaml_any = File.open("/etc/compass-ci/scheduler/#{flag}-testbox-env.yaml") do |content| + YAML.parse(content).as_h? + end + return hash unless yaml_any + + hash.merge!(Hash(String, JSON::Any).from_json(yaml_any.to_json)) + hash.each do |key, value| + if value == nil + hash[key] = service_env[key] + end + end + + hash + end +end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: modify the function get_hosts_file
by Li Ping 20 Feb '21

20 Feb '21

[why]: if the hosts_file 2288hv5-2s44p-38g--a6 already in lkp-tests/hosts/ submit borrow-10d.yaml testbox=2288hv5-2s44p-38g--a6 Traceback (most recent call last): 7: from /home/liping/lkp-tests/sbin/submit:136:in `<main>' 6: from /home/liping/lkp-tests/sbin/submit:136:in `each' 5: from /home/liping/lkp-tests/sbin/submit:150:in `block in <main>' 4: from /home/liping/lkp-tests/lib/job.rb:743:in `each_jobs' 3: from /home/liping/lkp-tests/lib/job.rb:447:in `load_defaults' 2: from /home/liping/lkp-tests/lib/job.rb:432:in `load_self_config' 1: from /home/liping/lkp-tests/lib/job.rb:351:in `load_hosts_config' /home/liping/lkp-tests/lib/job.rb:378:in `get_hosts_file': hosts file not exist: 2288hv5-2s44p-38g, maybe need check testbox field (ArgumentError) [reason]: get_hosts_file use tbox_group to search for hosts_file --- lib/job.rb | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index eacb7ec95..8f8c91b28 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -366,12 +366,17 @@ class Job end def get_hosts_file - testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" - return testbox_file if File.file?(testbox_file) - + lab_host = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts" + lkp_host = "#{LKP_SRC}/hosts" tbox_group = @job['tbox_group'].split(/\.|--/)[0] - tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" - return tbox_group_file if File.file?(tbox_group_file) + check_files = ["#{lab_host}/#{@job['testbox']}", + "#{lkp_host}/#{@job['testbox']}", + "#{lab_host}/#{tbox_group}", + "#{lkp_host}/#{tbox_group}"] + + check_files.each do |file| + return file if File.file?(file) + end raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end -- 2.23.0

2 2

[PATCH lkp-tests] tests/rpmbuild-pkg: various cleanup
by Wu Fengguang 20 Feb '21

20 Feb '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- tests/rpmbuild-pkg | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 781d04a0d..9f455f0d0 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -5,8 +5,8 @@ . $LKP_SRC/lib/upload.sh [ -n "$rpm_repo" ] || die "rpm_repo is empty" -benchmark=${rpm_repo##*/} -sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark}" +package_name=${rpm_repo##*/} +sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${package_name}" init_workspace() { @@ -16,15 +16,15 @@ init_workspace() get_pkgfile() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" } download_rpm_repo() { - filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') - for pkgfile in ${filelist[*]} + for pkgfile in "${filelist[@]}" do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" @@ -34,20 +34,29 @@ download_rpm_repo() build_rpm() { - spec_dir=${HOME}/rpmbuild/SPECS/$benchmark.spec - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_dir` + local spec_file=${HOME}/rpmbuild/SPECS/$package_name.spec + + # HTTP is proxy cache friendly + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_file` + # Install build depends - yum-builddep -y $spec_dir + yum-builddep -y $spec_file || exit # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE - spectool -g -R $spec_dir + spectool -g -R $spec_file || exit # Building rpm or srpm packages - rpmbuild -ba $spec_dir + rpmbuild -ba $spec_file || exit +} + +show_rpm_files() +{ + find ${HOME}/rpmbuild/RPMS -type f -name "*.rpm" + find ${HOME}/rpmbuild/SRPMS -type f -name "*.rpm" } upload_rpm_pkg() { - # rpm package will be generated in "${HOME}/rpmbuild/SRPMS" and "${HOME}/rpmbuild/RPMS" - for file in $(find ${HOME}/rpmbuild/ -type f -name "*.rpm") + local file + for file in $(show_rpm_files) do upload_one_curl ${file} ${sync_dest} done -- 2.23.0

1 0

[PATCH v2 compass-ci] rsync-server: add support upload function for ks file
by Yu Chuan 20 Feb '21

20 Feb '21

[Why] We need to achieve automatic installation through 'pxe + kickstart + dnsmasq + httpd'. So we need to rsync the ks file by rsync-server. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/rsync-server/rsyncd.conf | 6 ++++++ container/rsync-server/start | 1 + 2 files changed, 7 insertions(+) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index cdba89cf20c8..5a9bd2ddadec 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -10,3 +10,9 @@ path = /srv/result/upload/ read only = no max connections = 200 refuse options = delete + +[ks] +path = /srv/os/install/ks/ +read only = no +max connections = 200 +refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 956013268f89..334f5c2c4a69 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -11,6 +11,7 @@ cmd=( --restart=always --name=rsync_server -v /srv/result/upload:/srv/result/upload + -v /srv/os/install/ks:/srv/os/install/ks -v /etc/localtime:/etc/localtime:ro -p 11387:11387 rsync_server:latest -- 2.23.0

1 0

[PATCH lkp-tests] bootstrap.sh: add a test ping condition to network_ok
by Yu Chuan 20 Feb '21

20 Feb '21

[Why] In physical machine, there will be a delay between the start of network card and the availability of the network. So we add a test ping condition to network_ok. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 06aa09bf65b2..38965ff60755 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -85,14 +85,19 @@ net_devices_link() done } +test_ping() +{ + ping -c 1 -W 1 $LKP_SERVER > /dev/null || return 1 +} + network_ok() { local i for i in /sys/class/net/*/ do [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 + [ "$(cat $i/operstate)" = 'up' ] && test_ping && return 0 + [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && test_ping && return 0 done is_clearlinux && { -- 2.23.0

2 7

[PATCH compass-ci] lib/git_mirror.rb: change the way to load DEFAULTS file
by Li Yuanchao 19 Feb '21

19 Feb '21

Before: DEFAULTS files are individual. After: Load DEFAULTS file and merge it with the parent directory DEFAULTS files. So the outer DEFAULTS files would keep common things for the inner ones. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 39b1619..346352b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -130,6 +130,7 @@ class MirrorMain defaults_key = repodir == REPO_DIR ? 'default' : repodir.delete_prefix("#{REPO_DIR}/") @defaults[defaults_key] = YAML.safe_load(File.open(defaults_file)) + @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key]) end def traverse_repodir(repodir) @@ -224,7 +225,7 @@ class MirrorMain git_repo = "#{project}/#{fork_name}" @git_info[git_repo] = YAML.safe_load(File.open(repodir)) @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo] = merge_defaults(git_repo) + @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo]) fork_stat_init(git_repo) @priority_queue.push git_repo, @priority @priority += 1 @@ -449,15 +450,17 @@ end # main thread class MirrorMain - def merge_defaults(git_repo) - defaults_key = File.dirname(git_repo) + def merge_defaults(object_key, object) + return object if object_key == 'default' + + defaults_key = File.dirname(object_key) while defaults_key != '.' - return @defaults[defaults_key].merge(@git_info[git_repo]) if @defaults[defaults_key] + return @defaults[defaults_key].merge(object) if @defaults[defaults_key] defaults_key = File.dirname(defaults_key) end - return @defaults['default'].merge(@git_info[git_repo]) if @defaults['default'] + return @defaults['default'].merge(object) if @defaults['default'] - return @git_info[git_repo] + return object end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] lkp-bootstrap: delete unnecessary exit
by Xu Xijian 19 Feb '21

19 Feb '21

The exit cmd after reboot is unnecessary, delete it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- rootfs/addon/etc/init.d/lkp-bootstrap | 1 - 1 file changed, 1 deletion(-) diff --git a/rootfs/addon/etc/init.d/lkp-bootstrap b/rootfs/addon/etc/init.d/lkp-bootstrap index f66ec4eb5..091b29890 100755 --- a/rootfs/addon/etc/init.d/lkp-bootstrap +++ b/rootfs/addon/etc/init.d/lkp-bootstrap @@ -24,7 +24,6 @@ if [ -s /lkp/run/lkp-bootstrap.pid ]; then # the previous lkp-bootstrap init.d script failed to restart the machine echo "LKP bootstrap: first run may failed, retry rebooting" reboot - exit fi read_kernel_cmdline_vars() -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/2] bin/lkp-setup-rootfs: refactor the chmod_root_ssh()
by Xu Xijian 19 Feb '21

19 Feb '21

Add blank line after return cmd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- bin/lkp-setup-rootfs | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cf48c8f6b..bc0f04b2a 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -245,6 +245,7 @@ chmod_root_ssh() { [ -d "/root/.ssh" ] || return [ -f "/root/.ssh/id_rsa" ] || return + chmod 700 /root/.ssh/ chmod 600 /root/.ssh/id_rsa } -- 2.23.0

2 2