April 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] extract-stats: fix no new_error_id
by Cao Xueliang 26 Apr '21

26 Apr '21

regression_client need add auth info. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/regression_client.cr | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/regression_client.cr b/src/extract-stats/regression_client.cr index 586ea20..4ff8ab7 100644 --- a/src/extract-stats/regression_client.cr +++ b/src/extract-stats/regression_client.cr @@ -5,7 +5,13 @@ class RegressionClient HOST = (ENV.has_key?("ES_HOST") ? ENV["ES_HOST"] : JOB_ES_HOST) PORT = (ENV.has_key?("ES_PORT") ? ENV["ES_PORT"] : JOB_ES_PORT).to_i32 - def initialize(host = HOST, port = PORT) + def initialize(host = HOST, port = PORT, auth = true) + if auth + user = ENV["ES_USER"]? + password = ENV["ES_PASSWORD"]? + host = "#{user}:#{URI.encode_www_form(password)}@#{host}" if user && password + end + @client = Elasticsearch::API::Client.new({:host => host, :port => port}) end -- 2.23.0

1 0

[PATCH v2 lkp-tests] etc/latency: added
by Lu Weitao 26 Apr '21

26 Apr '21

[Why] compass-ci/lib/compare_matrixes.rb need lkp-test/etc/latency Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- etc/latency | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 etc/latency diff --git a/etc/latency b/etc/latency new file mode 100644 index 000000000..25dab098e --- /dev/null +++ b/etc/latency @@ -0,0 +1,13 @@ +dbench.max_latency +dd.kill_time +dd.startup_time +ebizzy.throughput.per_thread.stddev_percent +fileio.latency_95th_ms +fileio.latency_max_ms +iostat\..*\.(await|r_await|w_await|svctm|avgqu-sz)\.max +latency_stats\..* +oltp.latency_95th_ms +oltp.latency_max_ms +oltp.latency_min_ms +tbench.max_latency +vm-scalability.stddev -- 2.23.0

1 0

[PATCH lkp-tests] etc/latency: added
by Lu Weitao 26 Apr '21

26 Apr '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- etc/latency | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 etc/latency diff --git a/etc/latency b/etc/latency new file mode 100644 index 000000000..25dab098e --- /dev/null +++ b/etc/latency @@ -0,0 +1,13 @@ +dbench.max_latency +dd.kill_time +dd.startup_time +ebizzy.throughput.per_thread.stddev_percent +fileio.latency_95th_ms +fileio.latency_max_ms +iostat\..*\.(await|r_await|w_await|svctm|avgqu-sz)\.max +latency_stats\..* +oltp.latency_95th_ms +oltp.latency_max_ms +oltp.latency_min_ms +tbench.max_latency +vm-scalability.stddev -- 2.23.0

2 2

[PATCH v2 lkp-tests 2/2] lib/rpm.sh: add add_repo function
by Wang Yong 26 Apr '21

26 Apr '21

add function for rpmbuild-pkg and install-rpm Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/rpm.sh | 20 ++++++++++++++++++++ tests/install-rpm | 22 +--------------------- tests/rpmbuild-pkg | 6 ++++++ 3 files changed, 27 insertions(+), 21 deletions(-) create mode 100755 lib/rpm.sh diff --git a/lib/rpm.sh b/lib/rpm.sh new file mode 100755 index 000000000..91f234da4 --- /dev/null +++ b/lib/rpm.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +add_repo() +{ + custom_repo_name=($custom_repo_name) + custom_repo_addr=($custom_repo_addr) + + for i in ${!custom_repo_name[@]} + do + cat <<EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" +[${custom_repo_name[$i]}] +name=${custom_repo_name[$i]} +baseurl=${custom_repo_addr[$i]} +enabled=1 +gpgcheck=0 +priority=100 + +EOF + done +} \ No newline at end of file diff --git a/tests/install-rpm b/tests/install-rpm index d69d2d4c1..7449b3bd1 100755 --- a/tests/install-rpm +++ b/tests/install-rpm @@ -9,6 +9,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh . $LKP_SRC/lib/log.sh +. $LKP_SRC/lib/rpm.sh : "${compat_os:=budding-openeuler}" : "${DISTRO:=openeuler}" @@ -19,29 +20,8 @@ [ -n "${custom_repo_name}" ] || die "custom_repo_name is empty" [ -n "${custom_repo_addr}" ] || die "custom_repo_addr is empty" -custom_repo_name=($custom_repo_name) -custom_repo_addr=($custom_repo_addr) rpm_name=($rpm_name) -add_repo() -{ - custom_repo_name=($custom_repo_name) - custom_repo_addr=($custom_repo_addr) - - for i in ${!custom_repo_name[@]} - do - cat <<EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" -[${custom_repo_name[$i]}] -name=${custom_repo_name[$i]} -baseurl=${custom_repo_addr[$i]} -enabled=1 -gpgcheck=0 -priority=100 - -EOF - done -} - install_rpm() { log_info "Starting install test: yum install -y ${rpm_name[@]}" diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 312582d16..5c694b9d4 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -3,10 +3,15 @@ # - compat_os # - repo_name # - repo_addr +# - custom_repo_name +# - custom_repo_addr # Git repo jobs only use the first two parameters, SRPMs jobs use all +# user can add custom_repo_name and custom_repo_addr to setup +# local repo, value can be an array . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/rpm.sh : "${compat_os:=budding-openeuler}" @@ -133,6 +138,7 @@ update_repo_mq() -d "{\"upload_rpms\": ["${full_list}"]}" } +add_repo [ -n "$repo_name" ] && from_srpm || from_git build_rpm upload_rpm_pkg -- 2.23.0

1 0

[PATCH lkp-tests 1/2] tests/install-rpm: add suite install-rpm
by Wang Yong 26 Apr '21

26 Apr '21

add install-rpm to install/remove and ldd test for rpm packages Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/install-rpm | 1 + jobs/install-rpm.yaml | 4 ++ tests/install-rpm | 79 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 84 insertions(+) create mode 100644 distro/depends/install-rpm create mode 100755 jobs/install-rpm.yaml create mode 100755 tests/install-rpm diff --git a/distro/depends/install-rpm b/distro/depends/install-rpm new file mode 100644 index 000000000..602479c9a --- /dev/null +++ b/distro/depends/install-rpm @@ -0,0 +1 @@ +dnf-plugins-core diff --git a/jobs/install-rpm.yaml b/jobs/install-rpm.yaml new file mode 100755 index 000000000..9afa9d992 --- /dev/null +++ b/jobs/install-rpm.yaml @@ -0,0 +1,4 @@ +suite: install-rpm +category: functional + +install-rpm: diff --git a/tests/install-rpm b/tests/install-rpm new file mode 100755 index 000000000..428ca38d6 --- /dev/null +++ b/tests/install-rpm @@ -0,0 +1,79 @@ +#!/bin/bash +# - compat_os +# - rpm_name +# - custom_repo_name +# - custom_repo_addr +# user can add custom_repo_name and custom_repo_addr to setup +# local repo, value can be an array + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/log.sh + +: "${compat_os:=budding-openeuler}" +: "${DISTRO:=openeuler}" +: "${basearch:=$(arch)}" +. $LKP_SRC/distro/${DISTRO} + +[ -n "${rpm_name}" ] || die "rpm_name is empty" +[ -n "${custom_repo_name}" ] || die "custom_repo_name is empty" +[ -n "${custom_repo_addr}" ] || die "custom_repo_addr is empty" + +custom_repo_name=($custom_repo_name) +custom_repo_addr=($custom_repo_addr) +rpm_name=($rpm_name) + +add_repo() +{ + custom_repo_name=($custom_repo_name) + custom_repo_addr=($custom_repo_addr) + + for i in ${!custom_repo_name[@]} + do + cat <<EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" +[${custom_repo_name[$i]}] +name=${custom_repo_name[$i]} +baseurl=${custom_repo_addr[$i]} +enabled=1 +gpgcheck=0 +priority=100 + +EOF + done +} + +install_rpm() +{ + log_info "Starting install test: yum install -y ${rpm_name[@]}" + yum install -y ${rpm_name[@]} || die "install rpm failed" +} + +ldd_test() +{ + log_info "Starting ldd test" + for i in $(rpm -qa ${rpm_name[@]}) + do + ldd_list=$(rpm -ql $i | awk -F'/' '$3 ~ /bin|sbin|lib|lib64/') + [ -z "${ldd_list}" ] || { + for i in ${ldd_list[@]} + do + [[ -f "$i" && -x "$i" ]] && { + echo "ldd for $i" + ldd -d -r $i + } + done + } + done +} + +remove_rpm() +{ + log_info "Starting remove test: yum remove -y ${rpm_name[@]}" + yum remove -y ${rpm_name[@]} || die "uninstall rpm failed" +} + +distro_install_depends install-rpm +add_repo +install_rpm +ldd_test +remove_rpm -- 2.23.0

1 1

[PATCH v3 lkp-tests] lib/bootstrap.sh: fix the output error when ping not exist
by Wang Chenglong 26 Apr '21

26 Apr '21

[Why] If the ping command exists, then run test_ping, if not, then skip. [Error] [ 179.335889] Kernel tests: Boot OK! [ 180.180309] LKP: waiting for network... [ 182.991732] /lkp/lkp/src/lib/bootstrap.sh: line 90: ping: command not found [ 183.010815] /lkp/lkp/src/lib/bootstrap.sh: line 90: ping: command not found Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/bootstrap.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 070dabf46..5b82c2a6b 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -88,6 +88,10 @@ net_devices_link() test_ping() { + has_cmd ping || { + echo "no command ping, skip ping test." + return 0 + } ping -c 1 -w 1 $LKP_SERVER > /dev/null } -- 2.23.0

2 2

[PATCH v4 compass-ci] sbin/download-rootfs: download osimage
by Liu Yinsi 26 Apr '21

26 Apr '21

the default os_mount=initramfs, so download osimage to run qemu kvm. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 39 ++++++++++++++++++++++++++------------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/sbin/download-rootfs b/sbin/download-rootfs index 68a68039..cea416c0 100755 --- a/sbin/download-rootfs +++ b/sbin/download-rootfs @@ -19,22 +19,35 @@ os_version=${3:-20.03} load_cci_defaults rootfs_dir=/srv/os/${os}/${os_arch}/${os_version} +osimage_dir=/srv/initrd/osimage/${os}/${os_arch}/${os_version} -download_os_rootfs() -{ - mkdir -p $rootfs_dir - cd $rootfs_dir +mkdir -p $rootfs_dir $osimage_dir - download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" - wget $download_url +wget_download() +{ + local download_url=$1 + wget $download_url + + [ "$?" -ne 0 ] && { + log_error "failed to download os rootfs: $download_url" + exit 1 + } +} - [ "$?" -ne 0 ] && { - log_error "failed to download os rootfs: $download_url" - exit 1 - } +download_os_rootfs() +{ + cd $rootfs_dir + wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" + + gzip -dc ${os_version}.cgz | cpio -id + rm -rf ${os_version}.cgz +} - gzip -dc ${os_version}.cgz | cpio -id - rm -rf ${os_version}.cgz +download_osimage() +{ + cd $osimage_dir + wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/osimage/…" + wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/deps/nfs…" } is_virtual() @@ -42,4 +55,4 @@ is_virtual() dmidecode -s system-product-name | grep -iq "virtual" } -is_virtual || download_os_rootfs +is_virtual || ( download_os_rootfs && download_osimage ) -- 2.23.0

3 4

[PATCH compass-ci] sparrow/3-code/dev-env: rename some variables
by Liu Yinsi 26 Apr '21

26 Apr '21

[why] prevent to use variables value from z9 in testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index a478fffe..ba213e3a 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -41,16 +41,16 @@ server_name=$(hostname | cut -f1 -d.) : ${git_server:=$server_ip} : ${SCHED_HTTPS_HOST:=$server_ip} : ${SCHED_HTTPS_PORT:=20002} -: ${DOCKER_PROXY_HOST:=$server_ip} +: ${docker_proxy_host:=$server_ip} : ${DOCKER_PROXY_PORT:=3128} -: ${JUMPER_HOST:=$server_ip} +: ${jumper_host:=$server_ip} : ${JUMPER_PORT:=10000} -: ${MONITOR_HOST:=$server_ip} +: ${monitor_host:=$server_ip} : ${MONITOR_PORT:=20001} : ${FETCH_MAIL_DIR:=/srv/cci/Maildir} : ${FETCH_MAIL_BOX:=.compass-ci} : ${GIT_DAEMON_HOST:=$server_ip} -: ${GIT_MIRROR_HOST:=$server_ip} +: ${git_mirror_host:=$server_ip} : ${REDIS_PORT:=6379} : ${MAILDIR:=/srv/cci/Maildir/.compass-ci} : ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} @@ -122,16 +122,16 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS -DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_HOST: $docker_proxy_host DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT -JUMPER_HOST: $JUMPER_HOST +JUMPER_HOST: $jumper_host JUMPER_PORT: $JUMPER_PORT -MONITOR_HOST: $MONITOR_HOST +MONITOR_HOST: $monitor_host MONITOR_PORT: $MONITOR_PORT FETCH_MAIL_DIR: $FETCH_MAIL_DIR FETCH_MAIL_BOX: $FETCH_MAIL_BOX GIT_DAEMON_HOST: $GIT_DAEMON_HOST -GIT_MIRROR_HOST: $GIT_MIRROR_HOST +GIT_MIRROR_HOST: $git_mirror_host REDIS_PORT: $REDIS_PORT MAILDIR: $MAILDIR SENT_MAILDIR: $SENT_MAILDIR -- 2.23.0

2 3

[PATCH compass-ci] sparrow: set default value $CCI_SRC
by Liu Yinsi 26 Apr '21

26 Apr '21

if $CCI_SRC is null, can't find check_public_network.sh script. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 2 ++ sparrow/0-package/os/openEuler | 2 ++ sparrow/5-build/os-ready | 2 ++ 3 files changed, 6 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index eb86b242..82c09ff5 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci + . $CCI_SRC/lib/check_public_network.sh public_network_ok && { diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 8cdadd52..5829a123 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci + . $CCI_SRC/lib/check_public_network.sh pkgs=( diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready index 5e1acc48..91323379 100755 --- a/sparrow/5-build/os-ready +++ b/sparrow/5-build/os-ready @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci + . $CCI_SRC/lib/check_public_network.sh public_network_ok && $CCI_SRC/sbin/download-rootfs -- 2.23.0

2 1

[PATCH v2 lkp-tests] lib/bootstrap.sh: fix the output error when ping not exist
by Wang Chenglong 26 Apr '21

26 Apr '21

[Why] If the ping command exists, then run test_ping, if not, then skip. [Error] [ 179.335889] Kernel tests: Boot OK! [ 180.180309] LKP: waiting for network... [ 182.991732] /lkp/lkp/src/lib/bootstrap.sh: line 90: ping: command not found [ 183.010815] /lkp/lkp/src/lib/bootstrap.sh: line 90: ping: command not found Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/bootstrap.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 070dabf46..c699b40e4 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -88,6 +88,10 @@ net_devices_link() test_ping() { + [ $(command -v ping) ] || { + echo "no command ping, skip ping test." + return 0 + } ping -c 1 -w 1 $LKP_SERVER > /dev/null } -- 2.23.0

2 2