February 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/es_jobs: remove unnecessary null in raw.stats
by Wang Chenglong 26 Feb '21

26 Feb '21

before: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: [null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null, null,null,null,null,null,null,null,null,null,"{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found\"}", null,null,null,null,null,null,null,null,null,null,null] after: raw.stats.ansible_test.error.No-file-was-found-when-using-first_found.message: ["{\"msg\": \"An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped ifno files are found\"}"] Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/es_jobs.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index 2d2f3bf..c5c0f0d 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -104,7 +104,7 @@ class ESJobs def set_default_value(result, stats, metrics) left_metrics = metrics - stats.keys - left_metrics.each { |metric| result['raw.stats'][metric] << nil } + left_metrics.each { |metric| result['raw.stats'][metric] } stats.each do |key, value| result['raw.stats'][key] << value -- 2.23.0

1 0

[PATCH compass-ci] job.cr: add has_key? function in Job class
by Cao Xueliang 26 Feb '21

26 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 4 ++++ src/scheduler/find_job_boot.cr | 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..52965bc 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -332,6 +332,10 @@ class Job @hash.[key]? end + def has_key?(key : String) + @hash.has_key?(key) + end + def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" raise "Should not use []= update #{key}, use update_#{key}" diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 88db69e..8dd22a8 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,7 +160,7 @@ class Sched end private def get_boot_ipxe(job : Job) - return job["ipxe_response"] if job["suite"] == "install-iso" && job["ipxe_response"]? + return job["ipxe_response"] if job["suite"] == "install-iso" && job.has_key?("ipxe_response") response = "#!ipxe\n\n" -- 2.23.0

1 0

[PATCH v8 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 26 Feb '21

26 Feb '21

add time limit for borrowing machine. the max runtime is limited to no more than 10 days. case runtime/sleep beyond the limit or both of them was set empty value. it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..91d28fa 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -405,6 +406,29 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time(runtime/sleep) cannot exceed 10 days. Consider re-borrow.\n" + + if @hash["pp"]["sleep"].as_i? + sleep_run_time = @hash["pp"]["sleep"] + elsif @hash["pp"]["sleep"].as_h? + sleep_run_time = @hash["pp"]["sleep"]["runtime"] + else + notice_msg = "\nPlease set runtime/sleep first for the job yaml and retry." + raise notice_msg + end + + raise error_msg if sleep_run_time.as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH lkp-tests] bin/run-lkp: reduce unnecessary ucode output
by Wang Yong 25 Feb '21

25 Feb '21

reduce output when $ucode not exist Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- bin/run-lkp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/run-lkp b/bin/run-lkp index b422a7eb4..ff15b7c62 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -29,7 +29,7 @@ setup_result_root || { job_done; exit 1; } echo run-job $job set_job_state 'running' -echo "target ucode: $ucode" +[ -n "${ucode}" ] && echo "target ucode: $ucode" if [ -n "$ucode" ] && ! is_same_as_current_ucode_version "$ucode"; then echo "microcode is not matched, skip testing..." set_job_state 'microcode_is_not_matched' -- 2.23.0

2 2

git count
by Wu Fengguang 25 Feb '21

25 Feb '21

Yinsi is our code star, thanks Yinsi! wfg@crystal ~/todo/bin% ./git-count 25 41 Liu Yinsi 26 Luan Shengde 23 Cao Xueliang 17 Wei Jihui 17 Yu Chuan 16 Wang Yong 14 Li Yuanchao 13 Lu Weitao 11 Xu Xijian 10 Bai Jing 9 Xiao Shenwei 9 Cui Yili 9 Li Ping 8 Lin Jiaxin 6 Wu Zhende 5 Liu Shaofei 5 Wu Fengguang 3 Wang Chenglong 2 Zhang Dewan 0 Tong Qunfeng 0 Hu Xuejiao 0 Du Kaitian 0 Sun Yukui 0 Zhang Yale 0 Zhang Yu 0 Ren Wen 0 Lu Kaiyi 0 Shi Zhichao

1 0

review star
by Wu Fengguang 25 Feb '21

25 Feb '21

Jihui is our review star, thanks Jihui! Review email records in the last 7 days total review paragraph point people 84 40 125 44 Wu Fengguang 58 12 15 17 Wei Jihui 72 8 13 15 Cao Xueliang 33 5 18 7 Luan Shengde 7 2 8 6 Wang Yong 44 4 11 4 Yu Chuan 9 1 4 3 Xu Xijian 16 0 0 2 Li Yuanchao 17 1 0 1 Bai Jing 23 2 2 1 Li Ping 6 0 0 0 Liu Yinsi 3 1 1 0 Lin Jiaxin 2 0 0 0 Zhang Dewan 15 0 0 0 Cui Yili 6 0 0 0 Lu Weitao 22 1 2 0 Wang Chenglong

1 0

[PATCH compass-ci 1/3] sched: response ipxe_response value if ipxe_response filed in job yaml
by Cao Xueliang 25 Feb '21

25 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9e2898e..e8d590b 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,6 +160,8 @@ class Sched end private def get_boot_ipxe(job : Job) + return job["ipxe_response"] if job["ipxe_response"]? + response = "#!ipxe\n\n" _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } -- 2.23.0

2 3

[PATCH v7 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 25 Feb '21

25 Feb '21

add limit for borrowing machine, the max runtime is limited to no more than 10 days. case the runtime beyond the limit, it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..db1af2b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -405,6 +406,20 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time cannot exceed 10 days. Consider re-borrow.\n" + + raise error_msg if @hash["pp"]["sleep"]["runtime"].as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

3 4

[PATCH lkp-tests] distro/*: reduce Package Manager output
by Wang Yong 25 Feb '21

25 Feb '21

reduce unnecessary output to get effective information ref: commit 16cef539c517d0e6bd3495a1e3079214a8b4bd24 yum/dnf: -q, --quiet ref: https://linux.die.net/man/8/yum ref: https://www.man7.org/linux/man-pages/man8/dnf.8.html apt-get: -q, --quiet Quiet. Produces output suitable for logging, omitting progress indicators. More q's will produce more quiet up to a maximum of two. Note that quiet level 2 implies -y. ref: https://linux.die.net/man/8/apt-get pacman: -q, --quiet ref: https://archlinux.org/pacman/pacman.8.html zypper: --quiet, -q ref: https://en.opensuse.org/SDB:Zypper_usage Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/aliyun | 24 ++++++++++++------------ distro/archlinux | 11 ++++++----- distro/debian | 15 ++++++++------- distro/installer/archlinux | 3 ++- distro/installer/centos | 8 ++++---- distro/installer/debian | 5 +++-- distro/installer/fedora | 2 +- distro/installer/opensuse | 2 +- distro/installer/oracle | 6 +++--- distro/installer/ubuntu | 2 +- 10 files changed, 41 insertions(+), 37 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 83727ce32..83f058c33 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -9,17 +9,17 @@ shopt -s nullglob support_repo_epel() { if ! rpm -q epel-release >/dev/null; then - yum install -y epel-release + yum install -yq epel-release fi } update_yum_into_dnf() { if ! rpm -q dnf >/dev/null; then - yum install -y dnf + yum install -yq dnf ln -sf /usr/bin/dnf /usr/bin/yum else - dnf makecache timer + dnf makecache timer >/dev/null fi } @@ -46,7 +46,7 @@ distro_install_depends() for pkg in $packages do - rpm --quiet --query $pkg || yum install -y $pkg + rpm --quiet --query $pkg || yum install -yq $pkg done done } @@ -73,8 +73,8 @@ yum_download() [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -y --downloadonly $*" - yum install -y --downloadonly "$@" + echo "yum install -yq --downloadonly $*" + yum install -yq --downloadonly "$@" } yum_pack() @@ -137,9 +137,9 @@ install_gcc7() wget -O /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm https://rpmfind.net/linux/centos/7/extras/x86_64/Packages/centos-release-sc… rpm -ivh --ignoresize /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm else - yum install -y centos-release-scl-rh + yum install -yq centos-release-scl-rh fi - yum install -y "devtoolset-7-gcc*" + yum install -yq "devtoolset-7-gcc*" source /opt/rh/devtoolset-7/enable } @@ -148,7 +148,7 @@ download_ocfs2_tools() local pkgs=$1 mkdir -p /opt/rpms - yum install -y wget + yum install -yq wget wget -nc -P /opt/rpms https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/ocfs2-… @@ -160,7 +160,7 @@ download_ocfs2_tools() fi # Try to download the dependences of ocfs2-tools/ocfs2-tools-devel - yum install -y --downloadonly "$download_pkgs" + yum install -yq --downloadonly "$download_pkgs" } install_ocfs2_tools() @@ -169,8 +169,8 @@ install_ocfs2_tools() download_ocfs2_tools "$pkgs" - yum install -y /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm + yum install -yq /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm echo "$pkgs" | grep -q 'ocfs2-tools-devel' && \ - yum install -y /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm + yum install -yq /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm } diff --git a/distro/archlinux b/distro/archlinux index 59c8da1d6..dd4b2331c 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -43,8 +43,8 @@ find_pacman_package() pacman_download() { - echo "pacman -Sw --noconfirm $*" - pacman -Sw --noconfirm "$@" + echo "pacman -Swq --noconfirm $*" + pacman -Swq --noconfirm "$@" } pacman_pack() @@ -88,7 +88,8 @@ distro_install_depends() for pkg in $packages do - pacman -Qq --noconfirm $pkg > /dev/null || pacman -Sy --noconfirm $pkg + pacman -Qq --noconfirm $pkg > /dev/null || pacman -Syq --noconfirm --needed $pkg >/tmp/pacman_info 2>&1 + [ -f /tmp/pacman_info ] && grep -v "warning: could not get file information for *" /tmp/pacman_info done done } @@ -115,13 +116,13 @@ add_aur_helper() { echo "lkp ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers - pacman -Q yay 2>/dev/null && return + pacman -Qq yay >/dev/null && return local user=lkp local dir=/home/${user} mkdir -p ${dir} && chown ${user}:${user} ${dir} - pacman -S --needed --noconfirm git base-devel 2>/dev/null + pacman -Sq --needed --noconfirm git base-devel 2>/dev/null su - ${user} -c "git clone https://aur.archlinux.org/yay.git" su - ${user} -c "cd yay && makepkg -si --needed --noconfirm" } diff --git a/distro/debian b/distro/debian index 21dafe21b..a1642cc1f 100755 --- a/distro/debian +++ b/distro/debian @@ -8,7 +8,7 @@ shopt -s nullglob update() { - apt-get update + apt-get -qq update } add_i386_package() @@ -18,8 +18,8 @@ add_i386_package() download() { - echo apt-get download $* - apt-get download $* + echo apt-get -qq download $* + apt-get -qq download $* } hardlink_to_symlink() @@ -128,7 +128,7 @@ resolve_depends() local deps=$* local command - command="apt-get --simulate --no-install-recommends install $deps" + command="apt-get --simulate --no-install-recommends install -qq $deps" # using "bash -c" to support wildcard in apt-get. # if use wildcard in the package version variable, # apt-get will treat the wildcard as a normal character. @@ -193,7 +193,8 @@ distro_install_depends() do $chronic apt-get -o Dpkg::Options::=--force-confdef \ -o Dpkg::Options::=--force-confold \ - install -y --no-upgrade $package + install -qq --no-upgrade $package >/tmp/apt-get_info 2>&1 + grep -v "dpkg: warning: files list file for package '.*' missing;" /tmp/apt-get_info done done } @@ -225,7 +226,7 @@ fixup_distro_mirror() fixup_preinstall() { - apt-get install -yf + apt-get install -qqf } fixup_arch_install() @@ -235,7 +236,7 @@ fixup_arch_install() dpkg --add-architecture i386 update - apt -y --allow-remove-essential --fix-broken install -o=APT::Architecture="i386" + apt --allow-remove-essential --fix-broken install -qq -o=APT::Architecture="i386" } pack_benchmark_deps() diff --git a/distro/installer/archlinux b/distro/installer/archlinux index 3be9e7953..81ca82f73 100755 --- a/distro/installer/archlinux +++ b/distro/installer/archlinux @@ -13,4 +13,5 @@ add_aur_helper regular_user=$(grep 1090 /etc/passwd | awk -F':' '{ print $1 }') -sudo -u $regular_user yay -S --needed --noconfirm $* +sudo -u $regular_user yay -Sq --needed --noconfirm $* >/tmp/pacman_info 2>&1 +[ -f /tmp/pacman_info ] && grep -v "warning: could not get file information for *" /tmp/pacman_info diff --git a/distro/installer/centos b/distro/installer/centos index d9ad60038..704d3e3cc 100755 --- a/distro/installer/centos +++ b/distro/installer/centos @@ -1,12 +1,12 @@ #!/bin/bash if ! rpm -q epel-release >/dev/null; then - [ ${DISTRO^^} == "OPENEULER" ] || yum install -y epel-release + [ ${DISTRO^^} == "OPENEULER" ] || yum install -yq epel-release if [ ${DISTRO^^} == "OPENEULER" ]; then - yum makecache timer + yum makecache timer >/dev/null else - yum makecache fast + yum makecache fast >/dev/null fi fi -yum install -y $* +yum install -yq $* diff --git a/distro/installer/debian b/distro/installer/debian index 35b3209bd..8bf166458 100755 --- a/distro/installer/debian +++ b/distro/installer/debian @@ -2,7 +2,8 @@ # Debian package installation export DEBIAN_FRONTEND=noninteractive -dpkg --add-architecture i386 && apt-get update +dpkg --add-architecture i386 && apt-get -qq update apt-get -o Dpkg::Options::="--force-confdef" \ -o Dpkg::Options::="--force-confold" \ - -y install $* >/tmp/apt-get_info 2>&1 + -qq install $* +grep -v "dpkg: warning: files list file for package '.*' missing;" /tmp/apt-get_info diff --git a/distro/installer/fedora b/distro/installer/fedora index 1da708c2a..61176b361 100755 --- a/distro/installer/fedora +++ b/distro/installer/fedora @@ -1,3 +1,3 @@ #!/bin/sh -dnf -y install $* +dnf -yq install $* diff --git a/distro/installer/opensuse b/distro/installer/opensuse index cbdbf4eff..392d77b4a 100755 --- a/distro/installer/opensuse +++ b/distro/installer/opensuse @@ -1,3 +1,3 @@ #!/bin/bash -zypper install -y $* +zypper -q install -y $* diff --git a/distro/installer/oracle b/distro/installer/oracle index 11df716d6..c6881070c 100755 --- a/distro/installer/oracle +++ b/distro/installer/oracle @@ -1,8 +1,8 @@ #!/bin/sh if ! rpm -q epel-release >/dev/null; then - yum install -y epel-release - yum makecache fast + yum install -yq epel-release + yum makecache fast >/dev/null fi -yum install -y $* +yum install -yq $* diff --git a/distro/installer/ubuntu b/distro/installer/ubuntu index 541ba74f6..0ce801a79 100755 --- a/distro/installer/ubuntu +++ b/distro/installer/ubuntu @@ -6,4 +6,4 @@ dpkg --add-architecture i386 && apt-get update apt-get -o Dpkg::Options::="--force-confdef" \ -o Dpkg::Options::="--force-confold" \ -o Dpkg::Options::="--force-overwrite" \ - -y install $* + -qq install $* -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/compare_matrixes.rb: fix rubocop warning
by Lu Weitao 25 Feb '21

25 Feb '21

[Why] /home/lwt/compass-ci/lib/compare_matrixes.rb:586:14: C: Style/StringLiterals: Prefer single-quoted strings when you don't need string interpolation or special symbols. return " < 1% " if stddev_percent < 1 Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d004099..6700126 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -584,7 +584,7 @@ def format_stddev_percent(stddev_percent, average_width) percent_width = SUB_LONG_COLUMN_WIDTH - average_width if stddev_percent if stddev_percent != 0 - return " < 1% " if stddev_percent < 1 + return ' < 1% ' if stddev_percent < 1 percent_str = get_suitable_number_str( stddev_percent.abs, -- 2.23.0

1 0