- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] add check runtime upper limit
by Luan Shengde 01 Mar '21

01 Mar '21

add time limit for borrowing machine. the max runtime is limited to no more than 10 days. case runtime/sleep beyond the limit or both of them was set empty value. it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 52965bc..63ec8fb 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -409,6 +410,29 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time(runtime/sleep) cannot exceed 10 days. Consider re-borrow.\n" + + if @hash["pp"]["sleep"].as_i? + sleep_run_time = @hash["pp"]["sleep"] + elsif @hash["pp"]["sleep"].as_h? + sleep_run_time = @hash["pp"]["sleep"]["runtime"] + else + notice_msg = "\nPlease set runtime/sleep first for the job yaml and retry." + raise notice_msg + end + + raise error_msg if sleep_run_time.as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH compass-ci] container/scheduler: install util-linux to use uuidgen
by Wu Zhende 26 Feb '21

26 Feb '21

submit job.yaml in scheduler docker needs command "uuidgen" to generate submit_id Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/scheduler/Dockerfile b/container/scheduler/Dockerfile index 0ad813c..7fb8e35 100644 --- a/container/scheduler/Dockerfile +++ b/container/scheduler/Dockerfile @@ -8,7 +8,7 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN apk update RUN apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' 'util-linux' RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ -- 2.23.0

1 0

[PATCH compass-ci] lib/es_jobs: remove unnecessary null in raw.stats
by Wang Chenglong 26 Feb '21

26 Feb '21

before: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: [null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,"{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found\"}", null,null,null,null,null,null,null,null,null,null,null] after: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: ["{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped ifno files are found\"}"] Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/es_jobs.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index 2d2f3bf..c5c0f0d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -104,7 +104,7 @@ class ESJobs def set_default_value(result, stats, metrics) left_metrics = metrics - stats.keys - left_metrics.each { |metric| result['raw.stats'][metric] << nil } + left_metrics.each { |metric| result['raw.stats'][metric] } stats.each do |key, value| result['raw.stats'][key] << value -- 2.23.0

1 0

[PATCH compass-ci] job.cr: add has_key? function in Job class
by Cao Xueliang 26 Feb '21

26 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 4 ++++ src/scheduler/find_job_boot.cr | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..52965bc 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -332,6 +332,10 @@ class Job @hash.[key]? end + def has_key?(key : String) + @hash.has_key?(key) + end + def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" raise "Should not use []= update #{key}, use update_#{key}" diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 88db69e..8dd22a8 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,7 +160,7 @@ class Sched end private def get_boot_ipxe(job : Job) - return job["ipxe_response"] if job["suite"] == "install-iso" && job["ipxe_response"]? + return job["ipxe_response"] if job["suite"] == "install-iso" && job.has_key?("ipxe_response") response = "#!ipxe\n\n" -- 2.23.0

1 0

[PATCH v8 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 26 Feb '21

26 Feb '21

add time limit for borrowing machine. the max runtime is limited to no more than 10 days. case runtime/sleep beyond the limit or both of them was set empty value. it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..91d28fa 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -405,6 +406,29 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time(runtime/sleep) cannot exceed 10 days. Consider re-borrow.\n" + + if @hash["pp"]["sleep"].as_i? + sleep_run_time = @hash["pp"]["sleep"] + elsif @hash["pp"]["sleep"].as_h? + sleep_run_time = @hash["pp"]["sleep"]["runtime"] + else + notice_msg = "\nPlease set runtime/sleep first for the job yaml and retry." + raise notice_msg + end + + raise error_msg if sleep_run_time.as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH lkp-tests] bin/run-lkp: reduce unnecessary ucode output
by Wang Yong 25 Feb '21

25 Feb '21

reduce output when $ucode not exist Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- bin/run-lkp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/run-lkp b/bin/run-lkp index b422a7eb4..ff15b7c62 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -29,7 +29,7 @@ setup_result_root || { job_done; exit 1; } echo run-job $job set_job_state 'running' -echo "target ucode: $ucode" +[ -n "${ucode}" ] && echo "target ucode: $ucode" if [ -n "$ucode" ] && ! is_same_as_current_ucode_version "$ucode"; then echo "microcode is not matched, skip testing..." set_job_state 'microcode_is_not_matched' -- 2.23.0

2 2

git count
by Wu Fengguang 25 Feb '21

25 Feb '21

Yinsi is our code star, thanks Yinsi! wfg@crystal ~/todo/bin% ./git-count 25 41 Liu Yinsi 26 Luan Shengde 23 Cao Xueliang 17 Wei Jihui 17 Yu Chuan 16 Wang Yong 14 Li Yuanchao 13 Lu Weitao 11 Xu Xijian 10 Bai Jing 9 Xiao Shenwei 9 Cui Yili 9 Li Ping 8 Lin Jiaxin 6 Wu Zhende 5 Liu Shaofei 5 Wu Fengguang 3 Wang Chenglong 2 Zhang Dewan 0 Tong Qunfeng 0 Hu Xuejiao 0 Du Kaitian 0 Sun Yukui 0 Zhang Yale 0 Zhang Yu 0 Ren Wen 0 Lu Kaiyi 0 Shi Zhichao

1 0

review star
by Wu Fengguang 25 Feb '21

25 Feb '21

Jihui is our review star, thanks Jihui! Review email records in the last 7 days total review paragraph point people 84 40 125 44 Wu Fengguang 58 12 15 17 Wei Jihui 72 8 13 15 Cao Xueliang 33 5 18 7 Luan Shengde 7 2 8 6 Wang Yong 44 4 11 4 Yu Chuan 9 1 4 3 Xu Xijian 16 0 0 2 Li Yuanchao 17 1 0 1 Bai Jing 23 2 2 1 Li Ping 6 0 0 0 Liu Yinsi 3 1 1 0 Lin Jiaxin 2 0 0 0 Zhang Dewan 15 0 0 0 Cui Yili 6 0 0 0 Lu Weitao 22 1 2 0 Wang Chenglong

1 0

[PATCH compass-ci 1/3] sched: response ipxe_response value if ipxe_response filed in job yaml
by Cao Xueliang 25 Feb '21

25 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9e2898e..e8d590b 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,6 +160,8 @@ class Sched end private def get_boot_ipxe(job : Job) + return job["ipxe_response"] if job["ipxe_response"]? + response = "#!ipxe\n\n" _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } -- 2.23.0

2 3

[PATCH v7 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 25 Feb '21

25 Feb '21

add limit for borrowing machine, the max runtime is limited to no more than 10 days. case the runtime beyond the limit, it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..db1af2b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -405,6 +406,20 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time cannot exceed 10 days. Consider re-borrow.\n" + + raise error_msg if @hash["pp"]["sleep"]["runtime"].as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

3 4

[PATCH lkp-tests] distro/*: reduce Package Manager output
by Wang Yong 25 Feb '21

25 Feb '21

reduce unnecessary output to get effective information ref: commit 16cef539c517d0e6bd3495a1e3079214a8b4bd24 yum/dnf: -q, --quiet ref: https://linux.die.net/man/8/yum ref: https://www.man7.org/linux/man-pages/man8/dnf.8.html apt-get: -q, --quiet Quiet. Produces output suitable for logging, omitting progress indicators. More q's will produce more quiet up to a maximum of two. Note that quiet level 2 implies -y. ref: https://linux.die.net/man/8/apt-get pacman: -q, --quiet ref: https://archlinux.org/pacman/pacman.8.html zypper: --quiet, -q ref: https://en.opensuse.org/SDB:Zypper_usage Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/aliyun | 24 ++++++++++++------------ distro/archlinux | 11 ++++++----- distro/debian | 15 ++++++++------- distro/installer/archlinux | 3 ++- distro/installer/centos | 8 ++++---- distro/installer/debian | 5 +++-- distro/installer/fedora | 2 +- distro/installer/opensuse | 2 +- distro/installer/oracle | 6 +++--- distro/installer/ubuntu | 2 +- 10 files changed, 41 insertions(+), 37 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 83727ce32..83f058c33 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -9,17 +9,17 @@ shopt -s nullglob support_repo_epel() { if ! rpm -q epel-release >/dev/null; then - yum install -y epel-release + yum install -yq epel-release fi } update_yum_into_dnf() { if ! rpm -q dnf >/dev/null; then - yum install -y dnf + yum install -yq dnf ln -sf /usr/bin/dnf /usr/bin/yum else - dnf makecache timer + dnf makecache timer >/dev/null fi } @@ -46,7 +46,7 @@ distro_install_depends() for pkg in $packages do - rpm --quiet --query $pkg || yum install -y $pkg + rpm --quiet --query $pkg || yum install -yq $pkg done done } @@ -73,8 +73,8 @@ yum_download() [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -y --downloadonly $*" - yum install -y --downloadonly "$@" + echo "yum install -yq --downloadonly $*" + yum install -yq --downloadonly "$@" } yum_pack() @@ -137,9 +137,9 @@ install_gcc7() wget -O /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm https://rpmfind.net/linux/centos/7/extras/x86_64/Packages/centos-release-sc… rpm -ivh --ignoresize /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm else - yum install -y centos-release-scl-rh + yum install -yq centos-release-scl-rh fi - yum install -y "devtoolset-7-gcc*" + yum install -yq "devtoolset-7-gcc*" source /opt/rh/devtoolset-7/enable } @@ -148,7 +148,7 @@ download_ocfs2_tools() local pkgs=$1 mkdir -p /opt/rpms - yum install -y wget + yum install -yq wget wget -nc -P /opt/rpms https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/ocfs2-… @@ -160,7 +160,7 @@ download_ocfs2_tools() fi # Try to download the dependences of ocfs2-tools/ocfs2-tools-devel - yum install -y --downloadonly "$download_pkgs" + yum install -yq --downloadonly "$download_pkgs" } install_ocfs2_tools() @@ -169,8 +169,8 @@ install_ocfs2_tools() download_ocfs2_tools "$pkgs" - yum install -y /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm + yum install -yq /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm echo "$pkgs" | grep -q 'ocfs2-tools-devel' && \ - yum install -y /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm + yum install -yq /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm } diff --git a/distro/archlinux b/distro/archlinux index 59c8da1d6..dd4b2331c 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -43,8 +43,8 @@ find_pacman_package() pacman_download() { - echo "pacman -Sw --noconfirm $*" - pacman -Sw --noconfirm "$@" + echo "pacman -Swq --noconfirm $*" + pacman -Swq --noconfirm "$@" } pacman_pack() @@ -88,7 +88,8 @@ distro_install_depends() for pkg in $packages do - pacman -Qq --noconfirm $pkg > /dev/null || pacman -Sy --noconfirm $pkg + pacman -Qq --noconfirm $pkg > /dev/null || pacman -Syq --noconfirm --needed $pkg >/tmp/pacman_info 2>&1 + [ -f /tmp/pacman_info ] && grep -v "warning: could not get file information for *" /tmp/pacman_info done done } @@ -115,13 +116,13 @@ add_aur_helper() { echo "lkp ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers - pacman -Q yay 2>/dev/null && return + pacman -Qq yay >/dev/null && return local user=lkp local dir=/home/${user} mkdir -p ${dir} && chown ${user}:${user} ${dir} - pacman -S --needed --noconfirm git base-devel 2>/dev/null + pacman -Sq --needed --noconfirm git base-devel 2>/dev/null su - ${user} -c "git clone https://aur.archlinux.org/yay.git" su - ${user} -c "cd yay && makepkg -si --needed --noconfirm" } diff --git a/distro/debian b/distro/debian index 21dafe21b..a1642cc1f 100755 --- a/distro/debian +++ b/distro/debian @@ -8,7 +8,7 @@ shopt -s nullglob update() { - apt-get update + apt-get -qq update } add_i386_package() @@ -18,8 +18,8 @@ add_i386_package() download() { - echo apt-get download $* - apt-get download $* + echo apt-get -qq download $* + apt-get -qq download $* } hardlink_to_symlink() @@ -128,7 +128,7 @@ resolve_depends() local deps=$* local command - command="apt-get --simulate --no-install-recommends install $deps" + command="apt-get --simulate --no-install-recommends install -qq $deps" # using "bash -c" to support wildcard in apt-get. # if use wildcard in the package version variable, # apt-get will treat the wildcard as a normal character. @@ -193,7 +193,8 @@ distro_install_depends() do $chronic apt-get -o Dpkg::Options::=--force-confdef \ -o Dpkg::Options::=--force-confold \ - install -y --no-upgrade $package + install -qq --no-upgrade $package >/tmp/apt-get_info 2>&1 + grep -v "dpkg: warning: files list file for package '.*' missing;" /tmp/apt-get_info done done } @@ -225,7 +226,7 @@ fixup_distro_mirror() fixup_preinstall() { - apt-get install -yf + apt-get install -qqf } fixup_arch_install() @@ -235,7 +236,7 @@ fixup_arch_install() dpkg --add-architecture i386 update - apt -y --allow-remove-essential --fix-broken install -o=APT::Architecture="i386" + apt --allow-remove-essential --fix-broken install -qq -o=APT::Architecture="i386" } pack_benchmark_deps() diff --git a/distro/installer/archlinux b/distro/installer/archlinux index 3be9e7953..81ca82f73 100755 --- a/distro/installer/archlinux +++ b/distro/installer/archlinux @@ -13,4 +13,5 @@ add_aur_helper regular_user=$(grep 1090 /etc/passwd | awk -F':' '{ print $1 }') -sudo -u $regular_user yay -S --needed --noconfirm $* +sudo -u $regular_user yay -Sq --needed --noconfirm $* >/tmp/pacman_info 2>&1 +[ -f /tmp/pacman_info ] && grep -v "warning: could not get file information for *" /tmp/pacman_info diff --git a/distro/installer/centos b/distro/installer/centos index d9ad60038..704d3e3cc 100755 --- a/distro/installer/centos +++ b/distro/installer/centos @@ -1,12 +1,12 @@ #!/bin/bash if ! rpm -q epel-release >/dev/null; then - [ ${DISTRO^^} == "OPENEULER" ] || yum install -y epel-release + [ ${DISTRO^^} == "OPENEULER" ] || yum install -yq epel-release if [ ${DISTRO^^} == "OPENEULER" ]; then - yum makecache timer + yum makecache timer >/dev/null else - yum makecache fast + yum makecache fast >/dev/null fi fi -yum install -y $* +yum install -yq $* diff --git a/distro/installer/debian b/distro/installer/debian index 35b3209bd..8bf166458 100755 --- a/distro/installer/debian +++ b/distro/installer/debian @@ -2,7 +2,8 @@ # Debian package installation export DEBIAN_FRONTEND=noninteractive -dpkg --add-architecture i386 && apt-get update +dpkg --add-architecture i386 && apt-get -qq update apt-get -o Dpkg::Options::="--force-confdef" \ -o Dpkg::Options::="--force-confold" \ - -y install $* >/tmp/apt-get_info 2>&1 + -qq install $* +grep -v "dpkg: warning: files list file for package '.*' missing;" /tmp/apt-get_info diff --git a/distro/installer/fedora b/distro/installer/fedora index 1da708c2a..61176b361 100755 --- a/distro/installer/fedora +++ b/distro/installer/fedora @@ -1,3 +1,3 @@ #!/bin/sh -dnf -y install $* +dnf -yq install $* diff --git a/distro/installer/opensuse b/distro/installer/opensuse index cbdbf4eff..392d77b4a 100755 --- a/distro/installer/opensuse +++ b/distro/installer/opensuse @@ -1,3 +1,3 @@ #!/bin/bash -zypper install -y $* +zypper -q install -y $* diff --git a/distro/installer/oracle b/distro/installer/oracle index 11df716d6..c6881070c 100755 --- a/distro/installer/oracle +++ b/distro/installer/oracle @@ -1,8 +1,8 @@ #!/bin/sh if ! rpm -q epel-release >/dev/null; then - yum install -y epel-release - yum makecache fast + yum install -yq epel-release + yum makecache fast >/dev/null fi -yum install -y $* +yum install -yq $* diff --git a/distro/installer/ubuntu b/distro/installer/ubuntu index 541ba74f6..0ce801a79 100755 --- a/distro/installer/ubuntu +++ b/distro/installer/ubuntu @@ -6,4 +6,4 @@ dpkg --add-architecture i386 && apt-get update apt-get -o Dpkg::Options::="--force-confdef" \ -o Dpkg::Options::="--force-confold" \ -o Dpkg::Options::="--force-overwrite" \ - -y install $* + -qq install $* -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/compare_matrixes.rb: fix rubocop warning
by Lu Weitao 25 Feb '21

25 Feb '21

[Why] /home/lwt/compass-ci/lib/compare_matrixes.rb:586:14: C: Style/StringLiterals: Prefer single-quoted strings when you don't need string interpolation or special symbols. return " < 1% " if stddev_percent < 1 Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d004099..6700126 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -584,7 +584,7 @@ def format_stddev_percent(stddev_percent, average_width) percent_width = SUB_LONG_COLUMN_WIDTH - average_width if stddev_percent if stddev_percent != 0 - return " < 1% " if stddev_percent < 1 + return ' < 1% ' if stddev_percent < 1 percent_str = get_suitable_number_str( stddev_percent.abs, -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/compare_matrixes.rb skip .message|.log metric
by Lu Weitao 25 Feb '21

25 Feb '21

[Why] the metric end with .message|.log, no need to compare Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index ffc6500..d004099 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -163,6 +163,7 @@ def get_matrixes_values(matrixes_list, latest_jobs, options) matrixes_size = get_matrixes_size(matrixes_list) get_matrixes_fields(matrixes_list).each do |field| next if field == 'stats_source' + next if field.end_with?('.message', '.log') success = successful?(field) next unless success || latest_failure?(field, latest_jobs) -- 2.23.0

1 0

[PATCH v5 lkp-tests] jobs: add os-matrix-*.yaml
by Li Ping 25 Feb '21

25 Feb '21

1 0

[PATCH lkp-tests] pkg/mysql: change mysql default port
by Wei Jihui 25 Feb '21

25 Feb '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/mysql/PKGBUILD | 2 +- pkg/mysql/my.cnf | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD index 0f7d8bc8..977ad47e 100644 --- a/pkg/mysql/PKGBUILD +++ b/pkg/mysql/PKGBUILD @@ -9,7 +9,7 @@ source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" "https://github.com/thkukuk/rpcsvc-proto/releases/download/v1.4/rpcsvc-proto…" "my.cnf") -md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '7bb0372ec4e9281813a9afc67c6cfc20') +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '6d545f2b9a5ef6989c675063a328e407') prepare() { diff --git a/pkg/mysql/my.cnf b/pkg/mysql/my.cnf index 9732af40..acca75ae 100644 --- a/pkg/mysql/my.cnf +++ b/pkg/mysql/my.cnf @@ -13,7 +13,7 @@ socket=/data/mysql/run/mysql.sock tmpdir=/data/mysql/tmp datadir=/data/mysql/data default_authentication_plugin=mysql_native_password -port=3309 +port=3306 user=mysql memlock=on bind-address=0.0.0.0 -- 2.23.0

3 2

[PATCH lkp-tests] daemon: add mysql
by Wei Jihui 25 Feb '21

25 Feb '21

daemon/mysql is for start mysql server Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/mysql | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100755 daemon/mysql diff --git a/daemon/mysql b/daemon/mysql new file mode 100755 index 00000000..0291d3e9 --- /dev/null +++ b/daemon/mysql @@ -0,0 +1,104 @@ +#!/bin/sh +# - nic +# - device +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=$server}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=mysql_password}" + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$device" + mount /dev/"$device" /data + + mkdir -p /data/mysql/data /data/mysql/share /data/mysql/tmp /data/mysql/run /data/mysql/log + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +start_mysql_server() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "$mysql is not installed!" >&2 && exit 1 + fi + + "$mysqld" --user=mysql --basedir=/data/mysql --datadir=/data/mysql/data/ --initialize + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +disable_irqbalance() +{ + systemctl stop firewalld + iptables -F + pkill -9 irqbalance + systemctl start tuned +} + +intrrupt_nic_binding() +{ + local core_ranges='55-79 93-95' + ethtool -L $nic combined 28 + for core_range in $core_ranges + do + $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic + done + ethtool -l $nic +} + +net_io_cache_setting() +{ + echo deadline > /sys/block/"$device"/queue/scheduler + echo 2048 >/sys/block/"$device"/queue/nr_requests +} + +getcpu_process_bind_core() +{ + cd /getcpu || exit + insmod getcpu-hi.ko + numactl -C 0-92 -i 0-3 "mysqld" --defaults-file=/etc/my.cnf & + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + create_basedir + start_mysql_server + create_database + disable_irqbalance + intrrupt_nic_binding + net_io_cache_setting + getcpu_process_bind_core +} +main -- 2.23.0

2 1

[PATCH lkp-tests 2/3] tests/sysbench-mysql: skip clean up data
by Wei Jihui 25 Feb '21

25 Feb '21

comment out unnecessary opration: cleanup data Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/sysbench-mysql | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 1d424aaa..5dd8686f 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -85,9 +85,13 @@ run_sysbench_step() run_sysbench_mysql() { - run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + echo "data is being prepared, this will take some time..." + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare > /dev/null + run_sysbench_step oltp.lua ${args2[@]} run - run_sysbench_step oltp.lua ${args2[@]} cleanup + + # clean up test data will take some time, this step can be skipped + # run_sysbench_step oltp.lua ${args2[@]} cleanup > /dev/null } stop_firewalld -- 2.23.0

2 1

[PATCH lkp-tests 1/3] jobs/sysbench-mysql.yaml: udpate test args
by Wei Jihui 25 Feb '21

25 Feb '21

1. change cluster to cs-s1-a102-c1 2. add kernel args 3. add mysql args 4. add sysctl for performance optimized 5. add server nic and device, this can be sured by cluster Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/sysbench-mysql.yaml | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml index 83795a18..607acf12 100644 --- a/jobs/sysbench-mysql.yaml +++ b/jobs/sysbench-mysql.yaml @@ -1,10 +1,35 @@ suite: sysbench-mysql category: benchmark -cluster: cs-lkp-hsw-ep5 +cluster: cs-s1-a102-c1 +kernel_version: 4.19.90-mysql +kernel_custom_params: sched_steal_node_limit=8 + +mysql_user: +mysql_host: +mysql_port: +mysql_db: +mysql_password: if role server: + sysctl: + sc_numa_balancing: 0 + sc_sched_autogroup_enabled: 0 + sc_sched_min_granularity_ns: 10000000 + sc_sched_wakeup_granularity_ns: 15000000 + sc_threads-max: 52130 + sc_tcp_max_syn_backlog: 360000 + sc_dirty_ratio: 5 + sc_somaxconn: 1024 + sc_rmem_max: 16777216 + sc_wmem_max: 16777216 + sc_swappiness: 1 + sc_ipv4_tcp_rmem: 4096 87380 16777216 + sc_ipv4_tcp_wmem: 4096 87380 16777216 + sc_sched_features: "NO_WAKEUP_PREEMPTION STEAL" mysql: + nic: enp189s0f1 + device: nvme0n1 if role client: sysbench-mysql: @@ -17,6 +42,6 @@ if role client: runtime: 600 rand_type: special rand_spec_pct: 100 - oltp_read_only: off + oltp_read_only: false report_interval: 1 events: 2000000000 -- 2.23.0

2 1

[PATCH v3 compass-ci] doc/stats-rule.md
by Wang Chenglong 25 Feb '21

25 Feb '21

[Why] The rules to follow when writing the stats script. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/stats-rule.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 doc/stats-rule.md diff --git a/doc/stats-rule.md b/doc/stats-rule.md new file mode 100644 index 0000000..eaa2586 --- /dev/null +++ b/doc/stats-rule.md @@ -0,0 +1,47 @@ +${LKP_SERVER}/stats/${script} stat�� +========================================== + +## ��es�� + +1.��error_id�� + - original: {"changed": false, "failures": ["No package yum-plugin-priorities available."], "msg": "Failed to + install some of the specified packages", "rc": 1, "results": []} + + - error_id: error.No-package-yum-plugin-priorities-available: 1 + +2.�� + - ��error_id�� + # �� + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + end + + - ��jobs��'��'��es��error_id.��stats/${script}�� + +3.��error message�� + # original error message + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + end + +4.��error_id�� + - ��'-'�� + - ��'#'. + - ��'-',��. + + def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/-+/, '-') # ��'-', �� + line.gsub!(/\b-[a-z][0-9]\.[0-9]\.[0-9]+/, "#") # dgraph-linux-amd64-v0.8.2.tar.gz + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/{%/, ' ') + line.gsub!(/%}/, ' ') + line + end -- 2.23.0

2 1

[PATCH compass-ci] container/dracut-initrd: add mke2fs to module 98dracut-systemd
by Xu Xijian 25 Feb '21

25 Feb '21

1. align cmd "inst /usr/bin/awk" when insert it into module 40network/module-setup.sh. 2. add mke2fs to module 98dracut-systemd, it's used to make filesystem type. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 8f03fa8..849480d 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -16,4 +16,5 @@ rm -rf /var/lib/apt/lists/* cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh -sed -i "/install() {/ainst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh +sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh +sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh -- 2.23.0

1 0

[PATCH v3 lkp-tests] jobs: add os-matrix-arch.yaml
by Li Ping 25 Feb '21

25 Feb '21

1 0

[PATCH v5 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 25 Feb '21

25 Feb '21

[How] ansible_stats_openeuler < ${job_output} wcl@crystal ~/lkp-tests/stats% ./ansible_test < /srv/result/ansible_test/2021-02-04/ vm-2p16g--wcl/openeuler-20.03-aarch64/jgeusebroek-ansible-role-docker/crystal.974042/ansible_test error.Failed-to-download-metadata-for-repo-dockerrepo: 1 error.Failed-to-download-metadata-for-repo-dockerrepo.message: {"ansible_loop_var": "item", "changed": false, "item": "docker-engine", "msg": "Failed to download metadata for repo 'dockerrepo': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 138 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 127 insertions(+), 11 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..67e312f47 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,131 @@ #!/usr/bin/env ruby -while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line +require 'json' + +def output_error(error_msg) + error_id = common_error_id error_msg + puts "error.#{error_id}: 1" + puts "error.#{error_id}.message: #{@ansible_failed_info}" +end + +def parse_msg_1(ansible_failed_json) + case ansible_failed_json.chomp + + # When running some plug-in, the file was not found + when /(No file was found when using (.+))\./ + output_error $1 + + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + + # The task includes an option with an undefined variable + # {"msg": "The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined + # The error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, + # but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: + # Define mysql_packages. here"} + when /(.+ is undefined)/ + output_error $1 + + # {"changed": true, "cmd": ["packaging/installer/install-required-packages.sh", "--non-interactive", "netdata"], + # "msg": "non-zero return code", "rc": 1} + when /(non-zero return code)/ + output_error $1 + + # Some service could not find the requested host + when /(Could not find the requested service .+): host/ + output_error $1 + + # Unable to start some services + when /(Unable to start service .+):/ + output_error $1 + end +end + +def parse_msg_2(ansible_failed_json) + case ansible_failed_json.chomp + + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + + when /(Failure downloading .+) / + output_error $1 + + # Failed to download the yum repodata + # {"changed": false, "msg": "Failed to download metadata for repo 'docker-ce-stable': Cannot download repomd.xml", + # "rc": 1, "results": []} + when /(Failed to download metadata for repo.+)':/ + output_error $1 + + # Some role in not sipported this system + # {"changed": false, "msg": "Distribution openEuler is not supported by this role!"} + when /(.+ is not supported .+)/ + output_error $1 + + # failed to create temporary content file: The read operation timed out. + when /(.+ The read operation timed out)/ + output_error $1 + + # {"msg": "The conditional check 'java_version is defined' failed. The error was: error while evaluating conditional + # (java_version is defined): {{ java_default_version }}: {{ _java_default_version[ansible_distribution ~ '-' ~ + # ansible_distribution_major_version] | default(_java_default_version[ansible_distribution]) }}: 'dict object' has + # no attribute 'openEuler'\n\nThe error appears to be in '/root/.ansible/roles/ansible-role-java/tasks/assert.yml': + # line 19, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line + # appears to be:\n\n\n- name: test if java_version is set correctly\n ^ here\n"} + when /(The conditional check .+ failed)/ + output_error $1 + end +end + +def parse_failures(ansible_failed_json) + ansible_failed_json.each do |items| + case items.chomp + # Failed to install some of the specified packages. + # {"changed": false, "failures": ["No package erlang available."], "msg": + # "Failed to install some of the specified packages", "rc": 1, "results": []} + when /(No package .+ available)/ + output_error $1 + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json.chomp + + # Could not find or access ansible yaml + # {"ansible_facts": {}, "ansible_included_var_files": [], "changed": false, "message": + # "Could not find or access 'openEuler.yml' Searched in: + # /root/.ansible/roles/ansible-unzip/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/openEuler.yml i + # /root/.ansible/roles/ansible-unzip/tasks/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/tasks/openEuler.yml + # /root/.ansible/vars/openEuler.yml /root/.ansible/openEuler.yml on the Ansible Controller. + # If you are using a module and expect the file to exist on the remote, see the remote_src option"} + when /(Could not find or access .+')/ + output_error $1 end end + +def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line +end + +while (line = STDIN.gets) + next unless line =~ /(FAILED!|failed:).*=>(.*)/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.parse @ansible_failed_info + + parse_msg_1 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_msg_2 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_failures ansible_failed_json['failures'] unless ansible_failed_json['failures'].nil? + parse_message ansible_failed_json['message'] unless ansible_failed_json['message'].nil? +end -- 2.23.0

2 4

[PATCH v4 lkp-tests] jobs: add os-multi-dimension-*.yaml
by Li Ping 24 Feb '21

24 Feb '21

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: fix continue error
by Liu Yinsi 24 Feb '21

24 Feb '21

[error] ./buildall: line 59: continue: only meaningful in a `for', `while', or `until' loop [how] before: [ "$container_name" == 'ssh-r' ] && continue after: [ "$container_name" == 'ssh-r' ] && exit [why] 'continue' can not use independently without `for', `while', or `until', so use 'exit' to fix. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 96c7385..bb77a1f 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,7 +56,7 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && continue + [ "$container_name" == 'ssh-r' ] && exit [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/es/start: call check_es_ready()
by Liu Yinsi 24 Feb '21

24 Feb '21

call container/defconfig.sh:check_es_ready() to accelerate es/logging-es startup time. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/start | 7 +++---- container/logging-es/start | 5 ++--- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/container/es/start b/container/es/start index d84ad33..a1b81f9 100755 --- a/container/es/start +++ b/container/es/start @@ -20,7 +20,6 @@ cmd=( "${cmd[@]}" -echo "Please wait 60s for setting index" -sleep 60 - -set_es_indices +# set index +check_es_ready 9200 +find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; diff --git a/container/logging-es/start b/container/logging-es/start index 66464e6..13bf25d 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -20,9 +20,8 @@ cmd=( "${cmd[@]}" -echo "Please wait 60s for setting index" -sleep 60 - +# set index +check_es_ready 9202 curl -sSH "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' { "index": { -- 2.23.0

1 0

[PATCH lkp-tests] setup/simply-ci: fix git clone error
by Liu Yinsi 24 Feb '21

24 Feb '21

[how] before: git clone https://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git after: git clone https://gitee.com/wu_fengguang/compass-ci.git [why] refer code: git_ci(): ... if [ -n "$GIT_SERVER" ]; then cat >> /etc/gitconfig <<-EOF [url "git://$GIT_SERVER/gitee.com"] insteadOf=https://gitee.com ... if $GIT_SERVER exists, "http://gitee.com" will be instead of "git://$GIT_SERVER/gitee.com", so the old code is wrong. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index 076c357d9..c8f8a5a53 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -14,7 +14,7 @@ git_ci() EOF fi - git clone https://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git /c/compass-ci + git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci } dev_env() -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] delimiter: use secrets field to submit process job
by Cao Xueliang 24 Feb '21

24 Feb '21

Before: Search account info from ES. After: Use secrets field to get the my_token. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.rb | 8 ++++++++ src/delimiter/utils.rb | 18 ++++-------------- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/src/delimiter.rb b/src/delimiter.rb index 1ea5de5..cb43cb4 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -5,8 +5,16 @@ require_relative './delimiter/delimiter' require_relative '../lib/config_account' +def config_secrets_yaml + account = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/defaults/account.yaml" + lab = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/include/lab/#{account['lab']}.yaml" + secrets = Hash["secrets" => lab] + File.open("#{ENV['HOME']}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + begin config_yaml('delimiter') + config_secrets_yaml delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 0930dd7..dbc42ef 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -125,30 +125,20 @@ module Utils File.open(commit_jobs, 'a+') { |f| f.puts content } end - def get_account_info - ESQuery.new(index: 'accounts').query_by_id(DELIMITER_EMAIL) - end - def init_job_content(job_id) job_yaml = AssistResult.new.get_job_yaml(job_id) raise "get job yaml failed, job id: #{job_id}" unless job_yaml job = JSON.parse job_yaml - - account_info = get_account_info - raise "query delimiter account info failed: #{DELIMITER_EMAIL}" unless account_info - record_jobs(job['id'], job['upstream_commit']) job['suite'] = 'bisect' - job['my_name'] = account_info['my_name'] - job['my_email'] = account_info['my_email'] - job['my_token'] = account_info['my_token'] + job['my_name'] = ENV['my_name'] + job['my_email'] = ENV['my_email'] + job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id + job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" - job.delete('tboxgroup') - job.delete('subqueue') - job.delete('queue') job.delete('id') return job -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] sched: response ipxe_response value if ipxe_response field in job yaml
by Cao Xueliang 24 Feb '21

24 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9e2898e..e8d590b 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,6 +160,8 @@ class Sched end private def get_boot_ipxe(job : Job) + return job["ipxe_response"] if job["ipxe_response"]? + response = "#!ipxe\n\n" _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] delimiter: use secrets field to submit process job
by Cao Xueliang 24 Feb '21

24 Feb '21

Before: Search account info from ES. After: Use secrets field to get the my_token. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.rb | 8 ++++++++ src/delimiter/utils.rb | 10 ++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/src/delimiter.rb b/src/delimiter.rb index 1ea5de5..cb43cb4 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -5,8 +5,16 @@ require_relative './delimiter/delimiter' require_relative '../lib/config_account' +def config_secrets_yaml + account = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/defaults/account.yaml" + lab = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/include/lab/#{account['lab']}.yaml" + secrets = Hash["secrets" => lab] + File.open("#{ENV['HOME']}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + begin config_yaml('delimiter') + config_secrets_yaml delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 0930dd7..cd026f9 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -141,14 +141,12 @@ module Utils record_jobs(job['id'], job['upstream_commit']) job['suite'] = 'bisect' - job['my_name'] = account_info['my_name'] - job['my_email'] = account_info['my_email'] - job['my_token'] = account_info['my_token'] + job['my_name'] = ENV['my_name'] + job['my_email'] = ENV['my_email'] + job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id + job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" - job.delete('tboxgroup') - job.delete('subqueue') - job.delete('queue') job.delete('id') return job -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] lib/git_bisect.rb: fix: syntax error, unexpected string literal, expecting']'
by Cao Xueliang 24 Feb '21

24 Feb '21

need add ',' after 'upstream_url' => @upstream_url Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index dd0b782..6aedda5 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -113,7 +113,7 @@ class GitBisect 'work_dir' => @work_dir, 'bisect_error' => bisect_error, 'all_errors' => all_errors, - 'upstream_url' => @upstream_url + 'upstream_url' => @upstream_url, 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end -- 2.23.0

1 0

[PATCH compass-ci 3/3] delimiter: use secrets field to submit process job
by Cao Xueliang 24 Feb '21

24 Feb '21

Before: Search account info from ES. After: Use secrets field to get the my_token. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.rb | 8 ++++++++ src/delimiter/utils.rb | 10 ++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/src/delimiter.rb b/src/delimiter.rb index 1ea5de5..cb43cb4 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -5,8 +5,16 @@ require_relative './delimiter/delimiter' require_relative '../lib/config_account' +def config_secrets_yaml + account = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/defaults/account.yaml" + lab = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/include/lab/#{account['lab']}.yaml" + secrets = Hash["secrets" => lab] + File.open("#{ENV['HOME']}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + begin config_yaml('delimiter') + config_secrets_yaml delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 0930dd7..cd026f9 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -141,14 +141,12 @@ module Utils record_jobs(job['id'], job['upstream_commit']) job['suite'] = 'bisect' - job['my_name'] = account_info['my_name'] - job['my_email'] = account_info['my_email'] - job['my_token'] = account_info['my_token'] + job['my_name'] = ENV['my_name'] + job['my_email'] = ENV['my_email'] + job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id + job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" - job.delete('tboxgroup') - job.delete('subqueue') - job.delete('queue') job.delete('id') return job -- 2.23.0

1 0

[PATCH compass-ci 2/3] lib/git_bisect.rb: fix: syntax error, unexpected string literal, expecting']'
by Cao Xueliang 24 Feb '21

24 Feb '21

need add ',' after 'upstream_url' => @upstream_url Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index dd0b782..6aedda5 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -113,7 +113,7 @@ class GitBisect 'work_dir' => @work_dir, 'bisect_error' => bisect_error, 'all_errors' => all_errors, - 'upstream_url' => @upstream_url + 'upstream_url' => @upstream_url, 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/stats-rule.md
by Wang Chenglong 24 Feb '21

24 Feb '21

[Why] The rules to follow when writing the stats script. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/stats-rule.md | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 doc/stats-rule.md diff --git a/doc/stats-rule.md b/doc/stats-rule.md new file mode 100644 index 0000000..0fb557a --- /dev/null +++ b/doc/stats-rule.md @@ -0,0 +1,42 @@ +${LKP_SERVER}/stats/${script} stat�� +========================================== + +## ��es�� + +1.��error_id�� + - original: {"changed": false, "failures": ["No package yum-plugin-priorities available."], "msg": "Failed to + install some of the specified packages", "rc": 1, "results": []} + + - error_id: error.No-package-yum-plugin-priorities-available: 1 + +2.�� + - ��error_id�� + # �� + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + end + + - ��jobs��'��'��es��error_id.��stats/${script}�� + +3.��error message�� + # original error message + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + end + +4.��error_id�� + - �� + def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/{%/, ' ') + line.gsub!(/%}/, ' ') + line + end -- 2.23.0

2 2

[PATCH lkp-tests 3/3] tests/sysbench-mysql: fix mysql_port, mysql_password default value
by Wei Jihui 24 Feb '21

24 Feb '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/sysbench-mysql | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 5dd8686f..d6f5dcda 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -20,9 +20,9 @@ : "${mysql_user:=root}" : "${mysql_host:=$server}" -: "${mysql_port:=$mysql_port}" +: "${mysql_port:=3306}" : "${mysql_db:=sysbench_1}" -: "${mysql_password:=$mysql_password}" +: "${mysql_password:=mysql_password}" : "${db_driver:=mysql}" : "${oltp_test_mode:=complex}" : "${oltp_tables_count:=1000}" -- 2.23.0

1 0

[PATCH v3 lkp-tests] jobs: add os-multi-dimension-arch.yaml
by Li Ping 24 Feb '21

24 Feb '21

1 0

[PATCH v2 compass-ci] container/assign-account: optimize config login permission
by Luan Shengde 24 Feb '21

24 Feb '21

optimize account login permission setting: before: set enable login: has key: enable_login key value: true set disable login has key: enable_login key value: false or has no key: enable_login now: set enable login: has key: enable_login key value: true set disable login: has key: enable_login key value: false keep the default login permission: has no key: enable_login Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 7e0a5a7..228d468 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -136,7 +136,7 @@ class AccountStorage def permit_login_config(login_name) if @data.key?('enable_login') && @data['enable_login'] %x(usermod -s /usr/bin/zsh #{login_name}) - else + elsif @data.key?('enable_login') && ! @data['enable_login'] %x(usermod -s /sbin/nologin #{login_name}) end end -- 2.23.0

1 0

[PATCH lkp-tests] add tests script for openeuler os migration
by Bai Jing 24 Feb '21

24 Feb '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/openeuler_docker | 114 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100755 tests/openeuler_docker diff --git a/tests/openeuler_docker b/tests/openeuler_docker new file mode 100755 index 000000000..f79610b34 --- /dev/null +++ b/tests/openeuler_docker @@ -0,0 +1,114 @@ +#!/bin/sh +# - dockerfile_url + +. $LKP_SRC/lib/debug.sh + +[ -n "$dockerfile_url" ] || die "dockerfile_url is empty" + +cd $BENCHMARK_ROOT + +fit_repo() +{ + yum reinstall -y git + git clone "$dockerfile_url" || exit 0 + file_dir=$(find ./ -type f -name "*Dockerfile*" | xargs grep -E '^FROM centos:(centos|)[78]') + [ -n "$file_dir" ] || exit 0 +} + +openeuler_ready() +{ + docker load < $BENCHMARK_ROOT/openeuler_docker/openeuler-20.03-lts-v1.tar +} + +centos_docker() +{ + flag=0 + for file in $(find ./ -type f -name "*Dockerfile*" | xargs grep -E '^FROM centos:(centos|)[78]' | awk -F: '{print $1}') + do + image=$(echo "$file" | awk -F '/' '{print $3}' | tr 'A-Z' 'a-z') + if [ -f "build.sh" ] && [ -x "build.sh" ]; then + ./build.sh > /dev/null 2> /dev/null && handle_dockerfile + else + docker_build > /dev/null 2> /dev/null && handle_dockerfile + fi + done +} + +handle_FROM() +{ + sed -i "s/^FROM .*/FROM openeuler-20.03-lts:v1/g" "$file" + +} + +handle_epel() +{ + grep -qw "epel-release" "$file" && { + sed -i "s/epel-release/bash/g" "$file" + } +} + +handle_epelrpm() +{ + + grep -qw "https://.*epel-release.*rpm" "$file" && { + sed -i "s|https://.*epel-release.*rpm|bash|g" "$file" + + } +} + +handle_groupadd() +{ + grep -qw "groupadd*" "$file" && { + sed -i '/groupadd*/i\ yum -y install shadow && \\' "$file" + } + +} + +handle_epel_repo() +{ + grep -qwE '.* wget .*(epel|CentOS-Base)\.repo' "$file" && { + sed 's|^|#|g' "$file" + } +} + +handle_dockerfile() +{ + flag=1 + handle_FROM + handle_epel + handle_groupadd + docker_build "-openeuler" + [ $? -ne 0 ] || docker_run "-openeuler" +} + +docker_build() +{ + docker build -f "$file" -t "${image}$1" . + [ $? -ne 0 ] && { + [ "$flag" == "1" ] && echo "${image}$1.build.fail: 1" + return 1 + } + [ "$flag" == "1" ] && echo "${image}$1.build.success: 1" + flag=0 +} + +docker_run() +{ + docker run -d --name "${image}" "${image}$1" + docker_stats=$(docker inspect ${image} | grep Running | awk -F ' ' '{print $2}' | tr -d ',') + [ $docker_stats == false ] && { + echo "${image}.run.fail: 1" + return 1 + } + + echo "${image}.run.success: 1" +} + +docker_test() +{ + fit_repo + openeuler_ready + centos_docker +} + +docker_test -- 2.23.0

1 1

[PATCH v2 lkp-tests] jobs/: add os-multi-dimension-arch.yaml
by Li Ping 24 Feb '21

24 Feb '21

3 7

[PATCH v3 compass-ci] answerback-email: add store lab to ES
by Luan Shengde 24 Feb '21

24 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 8dceee1..f573f0f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -40,7 +40,8 @@ my_info = { 'my_commit_url' => nil, 'my_token' => nil, 'my_login_name' => nil, - 'my_ssh_pubkey' => [] + 'my_ssh_pubkey' => [], + 'lab' => LAB } # stdin_info is used to store infos added with option: @@ -179,7 +180,6 @@ def apply_account(my_info, conf_info) apply_info = {} apply_info.update my_info apply_info.update conf_info - apply_info['lab'] = LAB assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" account_info_str = RestClient.post assign_account_url, apply_info.to_json -- 2.23.0

1 0

[PATCH v6 compass-ci] doc: optimize os_mount.md
by Yu Chuan 24 Feb '21

24 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 255 +++++++++++++++++++++++++++++++------------- 1 file changed, 178 insertions(+), 77 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..f4b28bb4b4c7 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,9 +1,9 @@ # Summary -========= +--------- -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: +The below types are supported: - nfs - cifs - initramfs @@ -11,84 +11,185 @@ It has the following optional values: - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. - -2. scheduler return the ipxe_str to testbox + + ```shell + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +# os_mount=local +---------------- + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. + +So if you want to run your job in disk file system, you need to specify `os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + + +## temporary rootfs data by default + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follows: + ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot from this lv, just use it as a source data, and take a snapshot of it to get the real boot logical volume. + - src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + - ${timestamp} is just a version number, which is the latest rootfs version on the nfs server in cluster. + ${boot_lv} -- boot logical volume: + - will boot from this lv. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it. + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. ``` -3. dracut step of boot +## persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follows: + + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the + rootfs data of job-20210218.yaml so that it can be used by the subsequent + jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs + data of job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, `save_root_partition` and `use_root_partition` must be the same, and unique. + - you must `save_root_partition` firstly, then you can `use_root_partition`. otherwise, your job will fail. + +## work flow + +1. user submit job + + job fields: + ```yaml + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` + +2. scheduler return the ipxe_response to testbox + + ``` + #!ipxe + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ``` + +3. dracut step of boot + + ```bash + analyze_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analyze_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main + ``` 4. execute the job -- 2.23.0

1 0

[PATCH v5 compass-ci] doc: optimize os_mount.md
by Yu Chuan 24 Feb '21

24 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 264 ++++++++++++++++++++++++++++++-------------- 1 file changed, 180 insertions(+), 84 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..64e86b0ecb5f 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,94 +1,190 @@ -# Summary -========= +Summary +======= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: - - nfs - - cifs - - initramfs - - container - - local +The below types are supported: + - nfs + - cifs + - initramfs + - container + - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +os_mount=local +============== + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. +So if you want to run your job in disk file system, you need to specify +`os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + + +## temporary rootfs data by default + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs +will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot from this lv, just use it as a source data, and take a + snapshot of it to get the real boot logical volume. + - src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + - ${timestamp} is just a version number, which is the latest rootfs version + on the nfs server in cluster. + ${boot_lv} -- boot logical volume: + - will boot from this lv. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +## persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), +two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + job-20210218.yaml so that it can be used by the subsequent jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, ${save_root_partition} and ${use_root_partition} must be the same, and unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox - ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot - ``` + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ``` 3. dracut step of boot - ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi - ``` + ``` + analyze_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analyze_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main + ``` 4. execute the job -- 2.23.0

2 2

[PATCH compass-ci] doc/manual/submit-job.md: fix link error
by Wei Jihui 24 Feb '21

24 Feb '21

the links file name has changed to english. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- doc/manual/submit-job.en.md | 2 +- doc/manual/submit-job.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/submit-job.en.md b/doc/manual/submit-job.en.md index 2748fbe..de73f8f 100644 --- a/doc/manual/submit-job.en.md +++ b/doc/manual/submit-job.en.md @@ -2,7 +2,7 @@ ### Prerequisites -The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/本地…. +The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/in…. ### Purpose diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 92faa5c..77e9566 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -2,7 +2,7 @@ ### 前提条件 -已经按照[本地安装compass-ci客户端.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/本地安装compass-ci客户端.md )完成安装 +已经按照[本地安装compass-ci客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/install-cci-client.md )完成安装 ### 概述 -- 2.23.0

1 0

[PATCH compass-ci] doc/manual/submit-job.md: fix link error
by Wei Jihui 24 Feb '21

24 Feb '21

the links file name has changed to english. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- doc/manual/submit-job.en.md | 2 +- doc/manual/submit-job.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/submit-job.en.md b/doc/manual/submit-job.en.md index 2748fbe..de73f8f 100644 --- a/doc/manual/submit-job.en.md +++ b/doc/manual/submit-job.en.md @@ -2,7 +2,7 @@ ### Prerequisites -The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��…. +The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/in…. ### Purpose diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 92faa5c..77e9566 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -2,7 +2,7 @@ ### �� -��[��compass-ci��.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��compass-ci��.md )�� +��[��compass-ci��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/install-cci-client.md )�� ### �� -- 2.23.0

1 0

[PATCH compass-ci] doc/stats-rule.md
by Wang Chenglong 24 Feb '21

24 Feb '21

[Why] The rules to follow when writing the stats script. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/stats-rule.md | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 doc/stats-rule.md diff --git a/doc/stats-rule.md b/doc/stats-rule.md new file mode 100644 index 0000000..94affa7 --- /dev/null +++ b/doc/stats-rule.md @@ -0,0 +1,37 @@ +${LKP_SERVER}/stats/${script} stat�� +========================================== + +## ��es�� + +1.��error_id�� + - original: {"changed": false, "failures": ["No package yum-plugin-priorities available."], "msg": "Failed to + install some of the specified packages", "rc": 1, "results": []} + + - error_id: error.No-package-yum-plugin-priorities-available: 1 + +2.�� + - catch all kind errors + ��error_id�� + ��jobs��'��'��es��error_id.��stats/${script}�� + +3.��error message�� + # original error message + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + end + +4.��error_id�� + - �� + def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/{%/, ' ') + line.gsub!(/%}/, ' ') + line + end -- 2.23.0

2 2

[PATCH lab-z9] hosts/*: add default hdd_partitions
by Cui Yili 24 Feb '21

24 Feb '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- hosts/vm-1p1g | 1 + hosts/vm-2p16g | 1 + hosts/vm-2p1g | 1 + hosts/vm-2p4g | 1 + hosts/vm-2p8g | 1 + 5 files changed, 5 insertions(+) diff --git a/hosts/vm-1p1g b/hosts/vm-1p1g index 3af3646..b278e80 100644 --- a/hosts/vm-1p1g +++ b/hosts/vm-1p1g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 1 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p16g b/hosts/vm-2p16g index 10847a7..cd49afc 100644 --- a/hosts/vm-2p16g +++ b/hosts/vm-2p16g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 16G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p1g b/hosts/vm-2p1g index 0293ce9..4fffd2a 100644 --- a/hosts/vm-2p1g +++ b/hosts/vm-2p1g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 1G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p4g b/hosts/vm-2p4g index 219c4c9..41d0a3b 100644 --- a/hosts/vm-2p4g +++ b/hosts/vm-2p4g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 4G +hdd_partitions: /dev/vda diff --git a/hosts/vm-2p8g b/hosts/vm-2p8g index 6249322..8bc5cad 100644 --- a/hosts/vm-2p8g +++ b/hosts/vm-2p8g @@ -4,3 +4,4 @@ template: kvm nr_node: 1 nr_cpu: 2 memory: 8G +hdd_partitions: /dev/vda -- 2.23.0

2 1

[PATCH compass-ci] doc/manual/submit-job.md: fix link error
by Wei Jihui 24 Feb '21

24 Feb '21

the links file name has changed to english. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- doc/manual/submit-job.en.md | 2 +- doc/manual/submit-job.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/manual/submit-job.en.md b/doc/manual/submit-job.en.md index 2748fbe..de73f8f 100644 --- a/doc/manual/submit-job.en.md +++ b/doc/manual/submit-job.en.md @@ -2,7 +2,7 @@ ### Prerequisites -The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��…. +The Compass-CI client has been installed. For details, see [Installing the Local Compass-CI Client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/in…. ### Purpose diff --git a/doc/manual/submit-job.zh.md b/doc/manual/submit-job.zh.md index 92faa5c..77e9566 100644 --- a/doc/manual/submit-job.zh.md +++ b/doc/manual/submit-job.zh.md @@ -2,7 +2,7 @@ ### �� -��[��compass-ci��.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/��compass-ci��.md )�� +��[��compass-ci��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/install-cci-client.md )�� ### �� -- 2.23.0

1 0

[PATCH v4 compass-ci] doc: optimize os_mount.md
by Yu Chuan 24 Feb '21

24 Feb '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/job/os_mount.md | 265 ++++++++++++++++++++++++++++++-------------- 1 file changed, 181 insertions(+), 84 deletions(-) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index 226251a8bace..1e6b349a136d 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -1,94 +1,191 @@ -# Summary -========= +Summary +======= -`os_mount` defines the mount method of testbox's root partition. +`os_mount` defines the rootfs type. -It has the following optional values: - - nfs - - cifs - - initramfs - - container - - local +The below types are supported: + - nfs + - cifs + - initramfs + - container + - local Usage example: - - submit iperf.yaml testbox=vm-2p8g os_mount=nfs - - submit iperf.yaml testbox=vm-2p8g os_mount=cifs - - submit iperf.yaml testbox=vm-2p8g os_mount=initramfs - - submit iperf.yaml testbox=dc-8g os_mount=container - - submit iperf.yaml testbox=vm-2p8g os_mount=local - -# Optional Values -================= - -## Work flow when os_mount=local - -1. user submit job with os_mount: local - - optional {kernel_append_root}: - - save_root_partition: give an iconic string, then the root partition data used by this job will be retained. - and you can use the root partition data of this job in the futher job by {use_root_partition}. - - use_root_partition : give an iconic string, then the root partition data generated by the - pervious job which has the {save_root_partition} field will be used. - - root_partition_size: specify the size of root partition, *default is 10G*. - - - tips: - - value of {save_root_partition} and {use_root_partition} need to be iconic enough, uuid is recommended. - - value of {save_root_partition} and {use_root_partition} will be the suffix of logical volume. + ``` + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=nfs + submit -m -c borrow-1h.yaml testbox=vm-2p8g os_mount=cifs + submit -m -c borrow-1h.yaml testbox=vm-2p16g os_mount=initramfs + submit -m -c borrow-1h.yaml testbox=dc-8g os_mount=container + submit -m -c borrow-1h.yaml testbox=taishan200-2280-2s48p-256g os_mount=local + ``` + +os_mount=local +============== + +`nfs`, `cifs`, `initramfs`, `container`, rootfs of them all are in memory. +So if you want to run your job in disk file system, you need to specify +`os_mount=local` in your job.yaml. + +BTW, the job's rootfs will be placed on a logical volume. + +## description + +### temporary rootfs data *Defaultly* + +If you only add `os_mount: local` in your job.yaml, the logical volume of rootfs +will be deleted and re-created in initrd stage before every job executes. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - will never boot from this lv, just use it as a source data, and take a + snapshot of it to get the real boot logical volume. + - src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + - ${timestamp} is just a version number, which is the latest rootfs version + on the nfs server in cluster. + ${boot_lv} -- boot logical volume: + - will boot from this lv. + - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +### persistent rootfs data + +When you need to persist the rootfs data of a job, and use it in the subsequent job(s), +two fields in `kernel_append_root` will help you: `save_root_partition`, `use_root_partition`. + +The brief flow is as follows: + ``` + 1. boot and request scheduler for job. + 2. initrd stage: + - firstly, we need two logical volume: + ${src_lv} -- src logical volume: + - if have ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${use_root_partition} + - if no ${use_root_partition}, src_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${timestamp} + ${boot_lv} -- boot logical volume: + - if have ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version}_${save_root_partition} + - if no ${save_root_partition}, boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - if ${src_lv} not exists: + - if have ${use_root_partition}, exit 1. + - if no ${use_root_partition}, create ${src_lv}, and rsync the rootfs from cluster nfs server. + - if ${boot_lv} != ${src_lv}: + - if ${boot_lv} exists: delete it + - create ${boot_lv} as the snapshot of ${src_lv}. + - switch root to ${boot_lv}. + 3. boot the rootfs and execute the job. + ``` + +Demo usage: + ``` + - in 20210218, you submit a job-20210218.yaml, and you want to persist the rootfs data of + job-20210218.yaml so that it can be used by the subsequent jobs. + Then you need add the follow field in your job-20210218.yaml: + kernel_append_root: save_root_partition=zhangsan_local_for_iperf_20210218 + + - in 20210219, you submit a job-20210219.yaml, and you want to use the rootfs data of + job-20210218.yaml. + Then you need add the follow field in your job-20210219.yaml: + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 + ``` + +Notes: + - obviously, ${save_root_partition} and ${use_root_partition} must be the same, and unique. + - you must 'save_root_partition' firstly, then you can 'use_root_partition'. otherwise, your job will fail. + +## work flow + +1. user submit job + job fields: + ``` + os: openeuler + os_arch: aarch64 + os_version: 20.03 + os_mount: local + kernel_append_root: use_root_partition=zhangsan_local_for_iperf_20210218 save_root_partition=zhangsan_local_for_iperf_20210219 + ``` 2. scheduler return the ipxe_str to testbox - ``` - dhcp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/initrd.lkp - initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/{timestamp}/boot/vmlinuz - imgargs vmlinux root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src={nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} initrd=initrd.lkp {kernel_append_root} - boot - ``` + ``` + dhcp + initrd http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/initrd.lkp + kernel http://${http_server_ip}:${http_server_port}/os/openeuler/aarch64/20.03-iso-snapshots/${timestamp}/boot/vmlinuz root=/dev/mapper/os-openeuler_aarch64_20.03 rootfs_src=${nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/${timestamp} initrd=initrd.lkp ${kernel_append_root} + boot + ``` 3. dracut step of boot - ``` - if have {use_root_partition}; then - base_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || exit 1 - - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - - if {save_root_partition} == {use_root_partition}; then - boot from /dev/mappaer/os-{boot_lv_name} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - else - boot_lv_name=openeuler_aarch64_20.03_{value_of_use_root_partition} - boot from /dev/mappaer/os-{boot_lv_name} - fi - - else: - base_lv_name=openeuler_aarch64_20.03_{timestamp} - lvdisplay /dev/mappaer/os-{base_lv_name} > /dev/null || { - # create logical volume - lvcreate -L {root_partition_size} -n {base_lv_name} os || exit 1 - - # rsync nfsroot to lvm:/dev/mapper/os-{base_lv_name} - mount -t nfs {nfs_server_ip}:os/openeuler/aarch64/20.03-iso-snapshots/{timestamp} /mnt - mkdir /mnt1 && mount /dev/mapper/os-{base_lv_name} /mnt1 - cp -a /mnt/. /mnt1/ - umount /mnt /mnt1 - - # change premission of lvm:/dev/mapper/os-{base_lv_name} to readonly - lvchange -p r /dev/mapper/os-{base_lv_name} - } - - boot_lv_name=openeuler_aarch64_20.03 - if have {save_root_partition}; then - boot_lv_name=openeuler_aarch64_20.03_{value_of_save_root_partition} - fi - - lvremove -f /dev/mappaer/os-{boot_lv_name} - lvcreate /dev/mappaer/os-{boot_lv_name} from /dev/mappaer/os-{base_lv_name} || exit 1 - boot from /dev/mappaer/os-{boot_lv_name} - fi - ``` + ``` + analysis_kernel_cmdline_params() + { + for i in $(cat /proc/cmdline) + do + [ "$i" =~ "rootfs_src=" ] && export ROOTFS_SRC=${i#rootfs_src=} + done + + [ -z "${ROOTFS_SRC}" ] && die "cannot find var in kernel cmdline params: rootfs_src" + + export TIMESTAMP=$(basename ${ROOTFS_SRC}) + } + + rsync_src_lv() + { + local src_lv=$1 + + lvdisplay ${src_lv} > /dev/null && return + + # create logical volume + lvcreate --size 10G --name $(basename ${src_lv}) os || exit + + # rsync nfsroot to ${src_lv} + mount -t nfs ${ROOTFS_SRC} /mnt + mkdir /mnt1 && mount ${src_lv} /mnt1 + cp -a /mnt/. /mnt1/ + umount /mnt /mnt1 + + # change permission of ${src_lv} to readonly + lvchange --permission r ${src_lv} + } + + snapshot_boot_lv() + { + local src_lv=$1 + local boot_lv=$2 + + [ "$src_lv" == "$boot_lv" ] && return + + lvremove --force ${boot_lv} + lvcreate --size 10G --name $(basename ${boot_lv}) --snapshot ${src_lv} || exit + } + + main() + { + analysis_kernel_cmdline_params + + if [ -z "${use_root_partition}" ]; then + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${TIMESTAMP}" + rsync_src_lv ${src_lv} + else + src_lv="/dev/mapper/os-openeuler_aarch64_20.03_${use_root_partition}" + lvdisplay ${src_lv} > /dev/null || exit + fi + + if [ -z "${save_root_partition}" ]; then + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03" + else + boot_lv="/dev/mapper/os-openeuler_aarch64_20.03_${save_root_partition}" + fi + + snapshot_boot_lv ${src_lv} ${boot_lv} + boot_from_boot_lv + } + + main() + ``` 4. execute the job -- 2.23.0

2 2

[PATCH lkp-tests] jobs/: add os-multi-dimension-arch.yaml
by Li Ping 23 Feb '21

23 Feb '21

2 2

[PATCH lkp-tests] lib: export secrets.yaml in job_init
by Cao Xueliang 23 Feb '21

23 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/job-init.sh | 2 ++ lib/job.sh | 18 +++++++++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-) diff --git a/lib/job-init.sh b/lib/job-init.sh index 1fb898872..27d3c6d0b 100755 --- a/lib/job-init.sh +++ b/lib/job-init.sh @@ -1,4 +1,5 @@ . $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/job.sh mount_cgroup() { @@ -270,4 +271,5 @@ job_init() job_redirect_stdout_stderr job_env + read_secret_vars } diff --git a/lib/job.sh b/lib/job.sh index dc65f68d8..87ca7ee67 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -1,8 +1,20 @@ #!/bin/sh +read_secret_vars() +{ + read_yaml_vars "$(dirname $job_file)/secrets.yaml" secrets_ +} + read_env_vars() { - [ -f "$TMP/env.yaml" ] || return 0 + read_yaml_vars "$TMP/env.yaml" +} + +read_yaml_vars() +{ + local file=$1 + local prefix=$2 + [ -f "$file" ] || return 0 local key local val @@ -11,8 +23,8 @@ read_env_vars() do [ "${key%[a-zA-Z0-9_]:}" != "$key" ] || continue key=${key%:} - export "$key=$val" - done < $TMP/env.yaml + export "$prefix$key=$val" + done < $file return 0 } -- 2.23.0

1 0

find latest github git pushes
by Wu Fengguang 23 Feb '21

23 Feb '21

Hi Yuanchao, I researched github and have 2 finding 1) a registered user can watch a number of repos and get notifications. however the notification type does not include "git push/new commit". There is only "new pull request", which looks much less useful for us. 2) we can still take advantage of its search API to find latest updated repos, like this https://github.com/search?l=&o=desc&q=pushed%3A%3E2021-02-21+filename%3AREA… 7,543 repository results clean-code-craft-tcq-1/modular-c-AnPra7 modular-c-AnPra7 created by GitHub Classroom C MIT license Updated 3 minutes ago trpo2021/chessviz-Jamesss1337 chessviz-Jamesss1337 created by GitHub Classroom C Updated 3 minutes ago mubu43/ExploringC C Updated 3 minutes ago BlankyShi/gitTest gitTest C Updated 4 minutes ago aabelque/malloc C Updated 4 minutes ago ythan88/r-card C Updated 4 minutes ago koipkoi/pokeemerald-kr A korean translation project for Pokémon Emerald. C Updated 4 minutes ago OpenMPT/openmpt Official read-only git mirror of the OpenMPT and libopenmpt subversion repository at https://source.openmpt.org/. Thi… libopenmpt tracker audio music demoscene music-creation protracker impulsetracker screamtracker fasttracker 223 C Updated 4 minutes ago silabs-DenverL/ADCxSyncCapture ADC0 and ADC1 Synchronization Capture C MIT license Updated 4 minutes ago gopro/gopro-lib-node.gl OpenGL engine for building and rendering graph-based scenes 32 C Apache-2.0 license Updated 4 minutes ago Thanks, Fengguang

1 2