- Compass-ci - mailweb.openeuler.org

[PATCH v8 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 26 Feb '21

26 Feb '21

add time limit for borrowing machine. the max runtime is limited to no more than 10 days. case runtime/sleep beyond the limit or both of them was set empty value. it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..91d28fa 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -405,6 +406,29 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time(runtime/sleep) cannot exceed 10 days. Consider re-borrow.\n" + + if @hash["pp"]["sleep"].as_i? + sleep_run_time = @hash["pp"]["sleep"] + elsif @hash["pp"]["sleep"].as_h? + sleep_run_time = @hash["pp"]["sleep"]["runtime"] + else + notice_msg = "\nPlease set runtime/sleep first for the job yaml and retry." + raise notice_msg + end + + raise error_msg if sleep_run_time.as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

1 0

[PATCH lkp-tests] bin/run-lkp: reduce unnecessary ucode output
by Wang Yong 25 Feb '21

25 Feb '21

reduce output when $ucode not exist Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- bin/run-lkp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/run-lkp b/bin/run-lkp index b422a7eb4..ff15b7c62 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -29,7 +29,7 @@ setup_result_root || { job_done; exit 1; } echo run-job $job set_job_state 'running' -echo "target ucode: $ucode" +[ -n "${ucode}" ] && echo "target ucode: $ucode" if [ -n "$ucode" ] && ! is_same_as_current_ucode_version "$ucode"; then echo "microcode is not matched, skip testing..." set_job_state 'microcode_is_not_matched' -- 2.23.0

2 2

git count
by Wu Fengguang 25 Feb '21

25 Feb '21

Yinsi is our code star, thanks Yinsi! wfg@crystal ~/todo/bin% ./git-count 25 41 Liu Yinsi 26 Luan Shengde 23 Cao Xueliang 17 Wei Jihui 17 Yu Chuan 16 Wang Yong 14 Li Yuanchao 13 Lu Weitao 11 Xu Xijian 10 Bai Jing 9 Xiao Shenwei 9 Cui Yili 9 Li Ping 8 Lin Jiaxin 6 Wu Zhende 5 Liu Shaofei 5 Wu Fengguang 3 Wang Chenglong 2 Zhang Dewan 0 Tong Qunfeng 0 Hu Xuejiao 0 Du Kaitian 0 Sun Yukui 0 Zhang Yale 0 Zhang Yu 0 Ren Wen 0 Lu Kaiyi 0 Shi Zhichao

1 0

review star
by Wu Fengguang 25 Feb '21

25 Feb '21

Jihui is our review star, thanks Jihui! Review email records in the last 7 days total review paragraph point people 84 40 125 44 Wu Fengguang 58 12 15 17 Wei Jihui 72 8 13 15 Cao Xueliang 33 5 18 7 Luan Shengde 7 2 8 6 Wang Yong 44 4 11 4 Yu Chuan 9 1 4 3 Xu Xijian 16 0 0 2 Li Yuanchao 17 1 0 1 Bai Jing 23 2 2 1 Li Ping 6 0 0 0 Liu Yinsi 3 1 1 0 Lin Jiaxin 2 0 0 0 Zhang Dewan 15 0 0 0 Cui Yili 6 0 0 0 Lu Weitao 22 1 2 0 Wang Chenglong

1 0

[PATCH compass-ci 1/3] sched: response ipxe_response value if ipxe_response filed in job yaml
by Cao Xueliang 25 Feb '21

25 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9e2898e..e8d590b 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,6 +160,8 @@ class Sched end private def get_boot_ipxe(job : Job) + return job["ipxe_response"] if job["ipxe_response"]? + response = "#!ipxe\n\n" _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } -- 2.23.0

2 3

[PATCH v7 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 25 Feb '21

25 Feb '21

add limit for borrowing machine, the max runtime is limited to no more than 10 days. case the runtime beyond the limit, it will throw error message and prevent the submit for borrowing machine. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index c3eeaf4..db1af2b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -145,6 +145,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -405,6 +406,20 @@ class Job @hash.delete("my_token") end + private def check_run_time + # only job.yaml for borrowing machine has the key: ssh_pub_key + return unless @hash.has_key?("ssh_pub_key") + + # the maxmum borrowing time is limited no more than 10 days. + # case the runtime/sleep value count beyond the limit, + # it will throw error message and prevent the submit for borrowing machine. + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + error_msg = "\nMachine borrow time cannot exceed 10 days. Consider re-borrow.\n" + + raise error_msg if @hash["pp"]["sleep"]["runtime"].as_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

3 4

[PATCH lkp-tests] distro/*: reduce Package Manager output
by Wang Yong 25 Feb '21

25 Feb '21

reduce unnecessary output to get effective information ref: commit 16cef539c517d0e6bd3495a1e3079214a8b4bd24 yum/dnf: -q, --quiet ref: https://linux.die.net/man/8/yum ref: https://www.man7.org/linux/man-pages/man8/dnf.8.html apt-get: -q, --quiet Quiet. Produces output suitable for logging, omitting progress indicators. More q's will produce more quiet up to a maximum of two. Note that quiet level 2 implies -y. ref: https://linux.die.net/man/8/apt-get pacman: -q, --quiet ref: https://archlinux.org/pacman/pacman.8.html zypper: --quiet, -q ref: https://en.opensuse.org/SDB:Zypper_usage Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/aliyun | 24 ++++++++++++------------ distro/archlinux | 11 ++++++----- distro/debian | 15 ++++++++------- distro/installer/archlinux | 3 ++- distro/installer/centos | 8 ++++---- distro/installer/debian | 5 +++-- distro/installer/fedora | 2 +- distro/installer/opensuse | 2 +- distro/installer/oracle | 6 +++--- distro/installer/ubuntu | 2 +- 10 files changed, 41 insertions(+), 37 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 83727ce32..83f058c33 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -9,17 +9,17 @@ shopt -s nullglob support_repo_epel() { if ! rpm -q epel-release >/dev/null; then - yum install -y epel-release + yum install -yq epel-release fi } update_yum_into_dnf() { if ! rpm -q dnf >/dev/null; then - yum install -y dnf + yum install -yq dnf ln -sf /usr/bin/dnf /usr/bin/yum else - dnf makecache timer + dnf makecache timer >/dev/null fi } @@ -46,7 +46,7 @@ distro_install_depends() for pkg in $packages do - rpm --quiet --query $pkg || yum install -y $pkg + rpm --quiet --query $pkg || yum install -yq $pkg done done } @@ -73,8 +73,8 @@ yum_download() [ ${DISTRO^^} == "CENTOS" ] && support_repo_epel [ ${DISTRO^^} == "CENTOS" ] && update_yum_into_dnf - echo "yum install -y --downloadonly $*" - yum install -y --downloadonly "$@" + echo "yum install -yq --downloadonly $*" + yum install -yq --downloadonly "$@" } yum_pack() @@ -137,9 +137,9 @@ install_gcc7() wget -O /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm https://rpmfind.net/linux/centos/7/extras/x86_64/Packages/centos-release-sc… rpm -ivh --ignoresize /tmp/centos-release-scl-rh-2-3.el7.centos.noarch.rpm else - yum install -y centos-release-scl-rh + yum install -yq centos-release-scl-rh fi - yum install -y "devtoolset-7-gcc*" + yum install -yq "devtoolset-7-gcc*" source /opt/rh/devtoolset-7/enable } @@ -148,7 +148,7 @@ download_ocfs2_tools() local pkgs=$1 mkdir -p /opt/rpms - yum install -y wget + yum install -yq wget wget -nc -P /opt/rpms https://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/getPackage/ocfs2-… @@ -160,7 +160,7 @@ download_ocfs2_tools() fi # Try to download the dependences of ocfs2-tools/ocfs2-tools-devel - yum install -y --downloadonly "$download_pkgs" + yum install -yq --downloadonly "$download_pkgs" } install_ocfs2_tools() @@ -169,8 +169,8 @@ install_ocfs2_tools() download_ocfs2_tools "$pkgs" - yum install -y /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm + yum install -yq /opt/rpms/ocfs2-tools-1.8.6-11.el7.x86_64.rpm echo "$pkgs" | grep -q 'ocfs2-tools-devel' && \ - yum install -y /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm + yum install -yq /opt/rpms/ocfs2-tools-devel-1.8.6-11.el7.x86_64.rpm } diff --git a/distro/archlinux b/distro/archlinux index 59c8da1d6..dd4b2331c 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -43,8 +43,8 @@ find_pacman_package() pacman_download() { - echo "pacman -Sw --noconfirm $*" - pacman -Sw --noconfirm "$@" + echo "pacman -Swq --noconfirm $*" + pacman -Swq --noconfirm "$@" } pacman_pack() @@ -88,7 +88,8 @@ distro_install_depends() for pkg in $packages do - pacman -Qq --noconfirm $pkg > /dev/null || pacman -Sy --noconfirm $pkg + pacman -Qq --noconfirm $pkg > /dev/null || pacman -Syq --noconfirm --needed $pkg >/tmp/pacman_info 2>&1 + [ -f /tmp/pacman_info ] && grep -v "warning: could not get file information for *" /tmp/pacman_info done done } @@ -115,13 +116,13 @@ add_aur_helper() { echo "lkp ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers - pacman -Q yay 2>/dev/null && return + pacman -Qq yay >/dev/null && return local user=lkp local dir=/home/${user} mkdir -p ${dir} && chown ${user}:${user} ${dir} - pacman -S --needed --noconfirm git base-devel 2>/dev/null + pacman -Sq --needed --noconfirm git base-devel 2>/dev/null su - ${user} -c "git clone https://aur.archlinux.org/yay.git" su - ${user} -c "cd yay && makepkg -si --needed --noconfirm" } diff --git a/distro/debian b/distro/debian index 21dafe21b..a1642cc1f 100755 --- a/distro/debian +++ b/distro/debian @@ -8,7 +8,7 @@ shopt -s nullglob update() { - apt-get update + apt-get -qq update } add_i386_package() @@ -18,8 +18,8 @@ add_i386_package() download() { - echo apt-get download $* - apt-get download $* + echo apt-get -qq download $* + apt-get -qq download $* } hardlink_to_symlink() @@ -128,7 +128,7 @@ resolve_depends() local deps=$* local command - command="apt-get --simulate --no-install-recommends install $deps" + command="apt-get --simulate --no-install-recommends install -qq $deps" # using "bash -c" to support wildcard in apt-get. # if use wildcard in the package version variable, # apt-get will treat the wildcard as a normal character. @@ -193,7 +193,8 @@ distro_install_depends() do $chronic apt-get -o Dpkg::Options::=--force-confdef \ -o Dpkg::Options::=--force-confold \ - install -y --no-upgrade $package + install -qq --no-upgrade $package >/tmp/apt-get_info 2>&1 + grep -v "dpkg: warning: files list file for package '.*' missing;" /tmp/apt-get_info done done } @@ -225,7 +226,7 @@ fixup_distro_mirror() fixup_preinstall() { - apt-get install -yf + apt-get install -qqf } fixup_arch_install() @@ -235,7 +236,7 @@ fixup_arch_install() dpkg --add-architecture i386 update - apt -y --allow-remove-essential --fix-broken install -o=APT::Architecture="i386" + apt --allow-remove-essential --fix-broken install -qq -o=APT::Architecture="i386" } pack_benchmark_deps() diff --git a/distro/installer/archlinux b/distro/installer/archlinux index 3be9e7953..81ca82f73 100755 --- a/distro/installer/archlinux +++ b/distro/installer/archlinux @@ -13,4 +13,5 @@ add_aur_helper regular_user=$(grep 1090 /etc/passwd | awk -F':' '{ print $1 }') -sudo -u $regular_user yay -S --needed --noconfirm $* +sudo -u $regular_user yay -Sq --needed --noconfirm $* >/tmp/pacman_info 2>&1 +[ -f /tmp/pacman_info ] && grep -v "warning: could not get file information for *" /tmp/pacman_info diff --git a/distro/installer/centos b/distro/installer/centos index d9ad60038..704d3e3cc 100755 --- a/distro/installer/centos +++ b/distro/installer/centos @@ -1,12 +1,12 @@ #!/bin/bash if ! rpm -q epel-release >/dev/null; then - [ ${DISTRO^^} == "OPENEULER" ] || yum install -y epel-release + [ ${DISTRO^^} == "OPENEULER" ] || yum install -yq epel-release if [ ${DISTRO^^} == "OPENEULER" ]; then - yum makecache timer + yum makecache timer >/dev/null else - yum makecache fast + yum makecache fast >/dev/null fi fi -yum install -y $* +yum install -yq $* diff --git a/distro/installer/debian b/distro/installer/debian index 35b3209bd..8bf166458 100755 --- a/distro/installer/debian +++ b/distro/installer/debian @@ -2,7 +2,8 @@ # Debian package installation export DEBIAN_FRONTEND=noninteractive -dpkg --add-architecture i386 && apt-get update +dpkg --add-architecture i386 && apt-get -qq update apt-get -o Dpkg::Options::="--force-confdef" \ -o Dpkg::Options::="--force-confold" \ - -y install $* >/tmp/apt-get_info 2>&1 + -qq install $* +grep -v "dpkg: warning: files list file for package '.*' missing;" /tmp/apt-get_info diff --git a/distro/installer/fedora b/distro/installer/fedora index 1da708c2a..61176b361 100755 --- a/distro/installer/fedora +++ b/distro/installer/fedora @@ -1,3 +1,3 @@ #!/bin/sh -dnf -y install $* +dnf -yq install $* diff --git a/distro/installer/opensuse b/distro/installer/opensuse index cbdbf4eff..392d77b4a 100755 --- a/distro/installer/opensuse +++ b/distro/installer/opensuse @@ -1,3 +1,3 @@ #!/bin/bash -zypper install -y $* +zypper -q install -y $* diff --git a/distro/installer/oracle b/distro/installer/oracle index 11df716d6..c6881070c 100755 --- a/distro/installer/oracle +++ b/distro/installer/oracle @@ -1,8 +1,8 @@ #!/bin/sh if ! rpm -q epel-release >/dev/null; then - yum install -y epel-release - yum makecache fast + yum install -yq epel-release + yum makecache fast >/dev/null fi -yum install -y $* +yum install -yq $* diff --git a/distro/installer/ubuntu b/distro/installer/ubuntu index 541ba74f6..0ce801a79 100755 --- a/distro/installer/ubuntu +++ b/distro/installer/ubuntu @@ -6,4 +6,4 @@ dpkg --add-architecture i386 && apt-get update apt-get -o Dpkg::Options::="--force-confdef" \ -o Dpkg::Options::="--force-confold" \ -o Dpkg::Options::="--force-overwrite" \ - -y install $* + -qq install $* -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/compare_matrixes.rb: fix rubocop warning
by Lu Weitao 25 Feb '21

25 Feb '21

[Why] /home/lwt/compass-ci/lib/compare_matrixes.rb:586:14: C: Style/StringLiterals: Prefer single-quoted strings when you don't need string interpolation or special symbols. return " < 1% " if stddev_percent < 1 Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d004099..6700126 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -584,7 +584,7 @@ def format_stddev_percent(stddev_percent, average_width) percent_width = SUB_LONG_COLUMN_WIDTH - average_width if stddev_percent if stddev_percent != 0 - return " < 1% " if stddev_percent < 1 + return ' < 1% ' if stddev_percent < 1 percent_str = get_suitable_number_str( stddev_percent.abs, -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/compare_matrixes.rb skip .message|.log metric
by Lu Weitao 25 Feb '21

25 Feb '21

[Why] the metric end with .message|.log, no need to compare Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index ffc6500..d004099 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -163,6 +163,7 @@ def get_matrixes_values(matrixes_list, latest_jobs, options) matrixes_size = get_matrixes_size(matrixes_list) get_matrixes_fields(matrixes_list).each do |field| next if field == 'stats_source' + next if field.end_with?('.message', '.log') success = successful?(field) next unless success || latest_failure?(field, latest_jobs) -- 2.23.0

1 0

[PATCH v5 lkp-tests] jobs: add os-matrix-*.yaml
by Li Ping 25 Feb '21

25 Feb '21

1 0

[PATCH lkp-tests] pkg/mysql: change mysql default port
by Wei Jihui 25 Feb '21

25 Feb '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/mysql/PKGBUILD | 2 +- pkg/mysql/my.cnf | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD index 0f7d8bc8..977ad47e 100644 --- a/pkg/mysql/PKGBUILD +++ b/pkg/mysql/PKGBUILD @@ -9,7 +9,7 @@ source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" "https://github.com/thkukuk/rpcsvc-proto/releases/download/v1.4/rpcsvc-proto…" "my.cnf") -md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '7bb0372ec4e9281813a9afc67c6cfc20') +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '6d545f2b9a5ef6989c675063a328e407') prepare() { diff --git a/pkg/mysql/my.cnf b/pkg/mysql/my.cnf index 9732af40..acca75ae 100644 --- a/pkg/mysql/my.cnf +++ b/pkg/mysql/my.cnf @@ -13,7 +13,7 @@ socket=/data/mysql/run/mysql.sock tmpdir=/data/mysql/tmp datadir=/data/mysql/data default_authentication_plugin=mysql_native_password -port=3309 +port=3306 user=mysql memlock=on bind-address=0.0.0.0 -- 2.23.0

3 2

[PATCH lkp-tests] daemon: add mysql
by Wei Jihui 25 Feb '21

25 Feb '21

daemon/mysql is for start mysql server Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/mysql | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100755 daemon/mysql diff --git a/daemon/mysql b/daemon/mysql new file mode 100755 index 00000000..0291d3e9 --- /dev/null +++ b/daemon/mysql @@ -0,0 +1,104 @@ +#!/bin/sh +# - nic +# - device +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=$server}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=mysql_password}" + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$device" + mount /dev/"$device" /data + + mkdir -p /data/mysql/data /data/mysql/share /data/mysql/tmp /data/mysql/run /data/mysql/log + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +start_mysql_server() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "$mysql is not installed!" >&2 && exit 1 + fi + + "$mysqld" --user=mysql --basedir=/data/mysql --datadir=/data/mysql/data/ --initialize + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +disable_irqbalance() +{ + systemctl stop firewalld + iptables -F + pkill -9 irqbalance + systemctl start tuned +} + +intrrupt_nic_binding() +{ + local core_ranges='55-79 93-95' + ethtool -L $nic combined 28 + for core_range in $core_ranges + do + $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic + done + ethtool -l $nic +} + +net_io_cache_setting() +{ + echo deadline > /sys/block/"$device"/queue/scheduler + echo 2048 >/sys/block/"$device"/queue/nr_requests +} + +getcpu_process_bind_core() +{ + cd /getcpu || exit + insmod getcpu-hi.ko + numactl -C 0-92 -i 0-3 "mysqld" --defaults-file=/etc/my.cnf & + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + create_basedir + start_mysql_server + create_database + disable_irqbalance + intrrupt_nic_binding + net_io_cache_setting + getcpu_process_bind_core +} +main -- 2.23.0

2 1

[PATCH lkp-tests 2/3] tests/sysbench-mysql: skip clean up data
by Wei Jihui 25 Feb '21

25 Feb '21

comment out unnecessary opration: cleanup data Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/sysbench-mysql | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 1d424aaa..5dd8686f 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -85,9 +85,13 @@ run_sysbench_step() run_sysbench_mysql() { - run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + echo "data is being prepared, this will take some time..." + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare > /dev/null + run_sysbench_step oltp.lua ${args2[@]} run - run_sysbench_step oltp.lua ${args2[@]} cleanup + + # clean up test data will take some time, this step can be skipped + # run_sysbench_step oltp.lua ${args2[@]} cleanup > /dev/null } stop_firewalld -- 2.23.0

2 1

[PATCH lkp-tests 1/3] jobs/sysbench-mysql.yaml: udpate test args
by Wei Jihui 25 Feb '21

25 Feb '21

1. change cluster to cs-s1-a102-c1 2. add kernel args 3. add mysql args 4. add sysctl for performance optimized 5. add server nic and device, this can be sured by cluster Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/sysbench-mysql.yaml | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml index 83795a18..607acf12 100644 --- a/jobs/sysbench-mysql.yaml +++ b/jobs/sysbench-mysql.yaml @@ -1,10 +1,35 @@ suite: sysbench-mysql category: benchmark -cluster: cs-lkp-hsw-ep5 +cluster: cs-s1-a102-c1 +kernel_version: 4.19.90-mysql +kernel_custom_params: sched_steal_node_limit=8 + +mysql_user: +mysql_host: +mysql_port: +mysql_db: +mysql_password: if role server: + sysctl: + sc_numa_balancing: 0 + sc_sched_autogroup_enabled: 0 + sc_sched_min_granularity_ns: 10000000 + sc_sched_wakeup_granularity_ns: 15000000 + sc_threads-max: 52130 + sc_tcp_max_syn_backlog: 360000 + sc_dirty_ratio: 5 + sc_somaxconn: 1024 + sc_rmem_max: 16777216 + sc_wmem_max: 16777216 + sc_swappiness: 1 + sc_ipv4_tcp_rmem: 4096 87380 16777216 + sc_ipv4_tcp_wmem: 4096 87380 16777216 + sc_sched_features: "NO_WAKEUP_PREEMPTION STEAL" mysql: + nic: enp189s0f1 + device: nvme0n1 if role client: sysbench-mysql: @@ -17,6 +42,6 @@ if role client: runtime: 600 rand_type: special rand_spec_pct: 100 - oltp_read_only: off + oltp_read_only: false report_interval: 1 events: 2000000000 -- 2.23.0

2 1

[PATCH v3 compass-ci] doc/stats-rule.md
by Wang Chenglong 25 Feb '21

25 Feb '21

[Why] The rules to follow when writing the stats script. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/stats-rule.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 doc/stats-rule.md diff --git a/doc/stats-rule.md b/doc/stats-rule.md new file mode 100644 index 0000000..eaa2586 --- /dev/null +++ b/doc/stats-rule.md @@ -0,0 +1,47 @@ +${LKP_SERVER}/stats/${script} stat�� +========================================== + +## ��es�� + +1.��error_id�� + - original: {"changed": false, "failures": ["No package yum-plugin-priorities available."], "msg": "Failed to + install some of the specified packages", "rc": 1, "results": []} + + - error_id: error.No-package-yum-plugin-priorities-available: 1 + +2.�� + - ��error_id�� + # �� + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + end + + - ��jobs��'��'��es��error_id.��stats/${script}�� + +3.��error message�� + # original error message + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + end + +4.��error_id�� + - ��'-'�� + - ��'#'. + - ��'-',��. + + def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/-+/, '-') # ��'-', �� + line.gsub!(/\b-[a-z][0-9]\.[0-9]\.[0-9]+/, "#") # dgraph-linux-amd64-v0.8.2.tar.gz + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line.gsub!(/{%/, ' ') + line.gsub!(/%}/, ' ') + line + end -- 2.23.0

2 1

[PATCH compass-ci] container/dracut-initrd: add mke2fs to module 98dracut-systemd
by Xu Xijian 25 Feb '21

25 Feb '21

1. align cmd "inst /usr/bin/awk" when insert it into module 40network/module-setup.sh. 2. add mke2fs to module 98dracut-systemd, it's used to make filesystem type. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 8f03fa8..849480d 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -16,4 +16,5 @@ rm -rf /var/lib/apt/lists/* cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh -sed -i "/install() {/ainst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh +sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh +sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh -- 2.23.0

1 0

[PATCH v3 lkp-tests] jobs: add os-matrix-arch.yaml
by Li Ping 25 Feb '21

25 Feb '21

1 0

[PATCH v5 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 25 Feb '21

25 Feb '21

[How] ansible_stats_openeuler < ${job_output} wcl@crystal ~/lkp-tests/stats% ./ansible_test < /srv/result/ansible_test/2021-02-04/ vm-2p16g--wcl/openeuler-20.03-aarch64/jgeusebroek-ansible-role-docker/crystal.974042/ansible_test error.Failed-to-download-metadata-for-repo-dockerrepo: 1 error.Failed-to-download-metadata-for-repo-dockerrepo.message: {"ansible_loop_var": "item", "changed": false, "item": "docker-engine", "msg": "Failed to download metadata for repo 'dockerrepo': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 138 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 127 insertions(+), 11 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..67e312f47 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,131 @@ #!/usr/bin/env ruby -while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line +require 'json' + +def output_error(error_msg) + error_id = common_error_id error_msg + puts "error.#{error_id}: 1" + puts "error.#{error_id}.message: #{@ansible_failed_info}" +end + +def parse_msg_1(ansible_failed_json) + case ansible_failed_json.chomp + + # When running some plug-in, the file was not found + when /(No file was found when using (.+))\./ + output_error $1 + + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + + # The task includes an option with an undefined variable + # {"msg": "The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined + # The error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, + # but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: + # Define mysql_packages. here"} + when /(.+ is undefined)/ + output_error $1 + + # {"changed": true, "cmd": ["packaging/installer/install-required-packages.sh", "--non-interactive", "netdata"], + # "msg": "non-zero return code", "rc": 1} + when /(non-zero return code)/ + output_error $1 + + # Some service could not find the requested host + when /(Could not find the requested service .+): host/ + output_error $1 + + # Unable to start some services + when /(Unable to start service .+):/ + output_error $1 + end +end + +def parse_msg_2(ansible_failed_json) + case ansible_failed_json.chomp + + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + + when /(Failure downloading .+) / + output_error $1 + + # Failed to download the yum repodata + # {"changed": false, "msg": "Failed to download metadata for repo 'docker-ce-stable': Cannot download repomd.xml", + # "rc": 1, "results": []} + when /(Failed to download metadata for repo.+)':/ + output_error $1 + + # Some role in not sipported this system + # {"changed": false, "msg": "Distribution openEuler is not supported by this role!"} + when /(.+ is not supported .+)/ + output_error $1 + + # failed to create temporary content file: The read operation timed out. + when /(.+ The read operation timed out)/ + output_error $1 + + # {"msg": "The conditional check 'java_version is defined' failed. The error was: error while evaluating conditional + # (java_version is defined): {{ java_default_version }}: {{ _java_default_version[ansible_distribution ~ '-' ~ + # ansible_distribution_major_version] | default(_java_default_version[ansible_distribution]) }}: 'dict object' has + # no attribute 'openEuler'\n\nThe error appears to be in '/root/.ansible/roles/ansible-role-java/tasks/assert.yml': + # line 19, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line + # appears to be:\n\n\n- name: test if java_version is set correctly\n ^ here\n"} + when /(The conditional check .+ failed)/ + output_error $1 + end +end + +def parse_failures(ansible_failed_json) + ansible_failed_json.each do |items| + case items.chomp + # Failed to install some of the specified packages. + # {"changed": false, "failures": ["No package erlang available."], "msg": + # "Failed to install some of the specified packages", "rc": 1, "results": []} + when /(No package .+ available)/ + output_error $1 + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json.chomp + + # Could not find or access ansible yaml + # {"ansible_facts": {}, "ansible_included_var_files": [], "changed": false, "message": + # "Could not find or access 'openEuler.yml' Searched in: + # /root/.ansible/roles/ansible-unzip/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/openEuler.yml i + # /root/.ansible/roles/ansible-unzip/tasks/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/tasks/openEuler.yml + # /root/.ansible/vars/openEuler.yml /root/.ansible/openEuler.yml on the Ansible Controller. + # If you are using a module and expect the file to exist on the remote, see the remote_src option"} + when /(Could not find or access .+')/ + output_error $1 end end + +def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line +end + +while (line = STDIN.gets) + next unless line =~ /(FAILED!|failed:).*=>(.*)/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.parse @ansible_failed_info + + parse_msg_1 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_msg_2 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_failures ansible_failed_json['failures'] unless ansible_failed_json['failures'].nil? + parse_message ansible_failed_json['message'] unless ansible_failed_json['message'].nil? +end -- 2.23.0

2 4

[PATCH v4 lkp-tests] jobs: add os-multi-dimension-*.yaml
by Li Ping 24 Feb '21

24 Feb '21

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: fix continue error
by Liu Yinsi 24 Feb '21

24 Feb '21

[error] ./buildall: line 59: continue: only meaningful in a `for', `while', or `until' loop [how] before: [ "$container_name" == 'ssh-r' ] && continue after: [ "$container_name" == 'ssh-r' ] && exit [why] 'continue' can not use independently without `for', `while', or `until', so use 'exit' to fix. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 96c7385..bb77a1f 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,7 +56,7 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && continue + [ "$container_name" == 'ssh-r' ] && exit [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/es/start: call check_es_ready()
by Liu Yinsi 24 Feb '21

24 Feb '21

call container/defconfig.sh:check_es_ready() to accelerate es/logging-es startup time. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/start | 7 +++---- container/logging-es/start | 5 ++--- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/container/es/start b/container/es/start index d84ad33..a1b81f9 100755 --- a/container/es/start +++ b/container/es/start @@ -20,7 +20,6 @@ cmd=( "${cmd[@]}" -echo "Please wait 60s for setting index" -sleep 60 - -set_es_indices +# set index +check_es_ready 9200 +find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; diff --git a/container/logging-es/start b/container/logging-es/start index 66464e6..13bf25d 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -20,9 +20,8 @@ cmd=( "${cmd[@]}" -echo "Please wait 60s for setting index" -sleep 60 - +# set index +check_es_ready 9202 curl -sSH "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' { "index": { -- 2.23.0

1 0

[PATCH lkp-tests] setup/simply-ci: fix git clone error
by Liu Yinsi 24 Feb '21

24 Feb '21

[how] before: git clone https://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git after: git clone https://gitee.com/wu_fengguang/compass-ci.git [why] refer code: git_ci(): ... if [ -n "$GIT_SERVER" ]; then cat >> /etc/gitconfig <<-EOF [url "git://$GIT_SERVER/gitee.com"] insteadOf=https://gitee.com ... if $GIT_SERVER exists, "http://gitee.com" will be instead of "git://$GIT_SERVER/gitee.com", so the old code is wrong. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index 076c357d9..c8f8a5a53 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -14,7 +14,7 @@ git_ci() EOF fi - git clone https://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git /c/compass-ci + git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci } dev_env() -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] delimiter: use secrets field to submit process job
by Cao Xueliang 24 Feb '21

24 Feb '21

Before: Search account info from ES. After: Use secrets field to get the my_token. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.rb | 8 ++++++++ src/delimiter/utils.rb | 18 ++++-------------- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/src/delimiter.rb b/src/delimiter.rb index 1ea5de5..cb43cb4 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -5,8 +5,16 @@ require_relative './delimiter/delimiter' require_relative '../lib/config_account' +def config_secrets_yaml + account = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/defaults/account.yaml" + lab = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/include/lab/#{account['lab']}.yaml" + secrets = Hash["secrets" => lab] + File.open("#{ENV['HOME']}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + begin config_yaml('delimiter') + config_secrets_yaml delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 0930dd7..dbc42ef 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -125,30 +125,20 @@ module Utils File.open(commit_jobs, 'a+') { |f| f.puts content } end - def get_account_info - ESQuery.new(index: 'accounts').query_by_id(DELIMITER_EMAIL) - end - def init_job_content(job_id) job_yaml = AssistResult.new.get_job_yaml(job_id) raise "get job yaml failed, job id: #{job_id}" unless job_yaml job = JSON.parse job_yaml - - account_info = get_account_info - raise "query delimiter account info failed: #{DELIMITER_EMAIL}" unless account_info - record_jobs(job['id'], job['upstream_commit']) job['suite'] = 'bisect' - job['my_name'] = account_info['my_name'] - job['my_email'] = account_info['my_email'] - job['my_token'] = account_info['my_token'] + job['my_name'] = ENV['my_name'] + job['my_email'] = ENV['my_email'] + job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id + job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" - job.delete('tboxgroup') - job.delete('subqueue') - job.delete('queue') job.delete('id') return job -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] sched: response ipxe_response value if ipxe_response field in job yaml
by Cao Xueliang 24 Feb '21

24 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9e2898e..e8d590b 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,6 +160,8 @@ class Sched end private def get_boot_ipxe(job : Job) + return job["ipxe_response"] if job["ipxe_response"]? + response = "#!ipxe\n\n" _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] delimiter: use secrets field to submit process job
by Cao Xueliang 24 Feb '21

24 Feb '21

Before: Search account info from ES. After: Use secrets field to get the my_token. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.rb | 8 ++++++++ src/delimiter/utils.rb | 10 ++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/src/delimiter.rb b/src/delimiter.rb index 1ea5de5..cb43cb4 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -5,8 +5,16 @@ require_relative './delimiter/delimiter' require_relative '../lib/config_account' +def config_secrets_yaml + account = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/defaults/account.yaml" + lab = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/include/lab/#{account['lab']}.yaml" + secrets = Hash["secrets" => lab] + File.open("#{ENV['HOME']}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + begin config_yaml('delimiter') + config_secrets_yaml delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 0930dd7..cd026f9 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -141,14 +141,12 @@ module Utils record_jobs(job['id'], job['upstream_commit']) job['suite'] = 'bisect' - job['my_name'] = account_info['my_name'] - job['my_email'] = account_info['my_email'] - job['my_token'] = account_info['my_token'] + job['my_name'] = ENV['my_name'] + job['my_email'] = ENV['my_email'] + job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id + job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" - job.delete('tboxgroup') - job.delete('subqueue') - job.delete('queue') job.delete('id') return job -- 2.23.0

1 0