- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/2] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 21 Dec '20

21 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..f1b2c78 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,10 +76,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| matrix[key] = [] unless matrix[key] matrix[key] << value -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/monitor: get host from job
by Wu Zhende 21 Dec '20

21 Dec '20

If @job is exist and monitor_url is not configured, set monitor_url based on SCHED_HOST Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index f65cfccf1..283cd788d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -24,7 +24,6 @@ class Monitor # 1 means timeout exit @exit_status_code = 0 @defaults = {} - load_default @result = [] @stop_query = {} @reason = nil @@ -33,6 +32,10 @@ class Monitor def load_default return unless @monitor_url == '' + if host = @job["SCHED_HOST"] + return @monitor_url = "ws://#{host}:11310/filter" + end + Dir.glob(['/etc/compass-ci/monitor/*.yaml', "#{ENV['HOME']}/.config/compass-ci/monitor/*.yaml"]).each do |file| next unless File.exist? file @@ -98,6 +101,7 @@ class Monitor def run(timeout = nil) merge_overrides + load_default field_check @query.each do |k, v| -- 2.23.0

2 1

Re: [PATCH v2 compass-ci] lib/matrix2.rb remove the *timestamp* from matrix
by Ren Wen 21 Dec '20

21 Dec '20

Why remove ':' after 'lib/matrix2.rb'? Thanks, RenWen Reply-To: In-Reply-To: <20201221022051.3690974-1-luweitaobe(a)163.com> On Mon, Dec 21, 2020 at 10:20:51AM +0800, Lu Weitao wrote: >[Why] >In compare results: >the value of *timestamp*(metric) doesn't mean the number of occurrences, >it may confuse user like bellow: > > openeuler centos archLinux metric >-------------------- ------------------------------ ------------------------------ ------------------------------ > fails:runs change fails:runs change fails:runs > | | | | | > 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last > 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last > 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail > >[How] >remove the metric that include "timestamp" from matrix > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > lib/matrix2.rb | 2 ++ > 1 file changed, 2 insertions(+) > >diff --git a/lib/matrix2.rb b/lib/matrix2.rb >index c712e32..2769f7c 100644 >--- a/lib/matrix2.rb >+++ b/lib/matrix2.rb >@@ -87,6 +87,8 @@ def create_matrix(job_list) > next unless stats > > stats.each do |key, value| >+ next if key.include?('timestamp') >+ > matrix[key] = [] unless matrix[key] > matrix[key] << value > end >-- >2.23.0 >

2 1

[PATCH lkp-tests 2/4] lib/env.sh: move functions to env.sh
by Wei Jihui 21 Dec '20

21 Dec '20

move functions:write_shell_profile and write_host to lib/env.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/env.sh | 30 ++++++++++++++++++++++++++++++ sbin/set-env.sh | 39 --------------------------------------- 2 files changed, 30 insertions(+), 39 deletions(-) delete mode 100755 sbin/set-env.sh diff --git a/lib/env.sh b/lib/env.sh index 8ac42a83..d360f163 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -125,3 +125,33 @@ shell_profile() [ -f "$profile" ] || touch $profile echo $profile } + +write_shell_profile() +{ + shell_profile_file=$(shell_profile) + if [ $# -gt 0 ]; then + echo $@ >> $shell_profile_file + fi + + source $shell_profile_file +} + +write_host() +{ + if is_system "Linux"; then + nr_cpu=$(nproc) + memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') + else + nr_cpu=$(sysctl -n hw.logicalcpu) + memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') + fi + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + + cat > hosts/$(hostname) <<-EOF + nr_cpu: $nr_cpu + memory: $memory + hdd_partitions: + ssd_partitions: + EOF +} diff --git a/sbin/set-env.sh b/sbin/set-env.sh deleted file mode 100755 index cc7f9050..00000000 --- a/sbin/set-env.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/bin/sh - -. $LKP_SRC/lib/env.sh - -write_shell_profile() -{ - shell_profile_file=$(shell_profile) - if [ $# -gt 0 ]; then - echo $@ >> $shell_profile_file - else - echo "export LKP_SRC=$PWD" >> $shell_profile_file - echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file - fi - - source $shell_profile_file -} - -write_host() -{ - if is_system "Linux"; then - nr_cpu=$(nproc) - memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - else - nr_cpu=$(sysctl -n hw.logicalcpu) - memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') - fi - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') - memory=$(echo $memory_new | awk '{print int($0)+1}')G - - cat > hosts/$(hostname) <<-EOF - nr_cpu: $nr_cpu - memory: $memory - hdd_partitions: - ssd_partitions: - EOF -} - -write_shell_profile -write_host -- 2.23.0

2 1

Re: [PATCH compass-ci 2/2] lib/matrix2.rb remove the *timestamp* form matrix
by Ren Wen 21 Dec '20

21 Dec '20

spell: form -> from Thanks, RenWen Reply-To: In-Reply-To: <20201219133430.1143483-2-luweitaobe(a)163.com> On Sat, Dec 19, 2020 at 09:34:30PM +0800, Lu Weitao wrote: >[Why] >In compare results: >the value of *timestamp*(metric) doesn't mean the number of occurrences, >it may confuse user like bellow: > > openeuler centos archLinux metric >-------------------- ------------------------------ ------------------------------ ------------------------------ > fails:runs change fails:runs change fails:runs > | | | | | > 3585:13 +6913.7% 344:1 -2.6e+4% 11:1 dmesg.timestamp:last > 2350:13 +1.6e+4% 344:1 -1.7e+4% 10:1 kmsg.timestamp:last > 2:13 -15.4% 0:1 +84.6% 1:1 last_state.exit_fail > >[How] >remove the metric that include "timestamp" from matrix > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > lib/matrix2.rb | 2 ++ > 1 file changed, 2 insertions(+) > >diff --git a/lib/matrix2.rb b/lib/matrix2.rb >index c712e32..2769f7c 100644 >--- a/lib/matrix2.rb >+++ b/lib/matrix2.rb >@@ -87,6 +87,8 @@ def create_matrix(job_list) > next unless stats > > stats.each do |key, value| >+ next if key.include?('timestamp') >+ > matrix[key] = [] unless matrix[key] > matrix[key] << value > end >-- >2.23.0 >

2 1

[PATCH v2 compass-ci] providers/docker: set the private key of the testbox
by Xiao Shenwei 21 Dec '20

21 Dec '20

[why] login the docker testbox without password [how] Add ssh private key for docker, for automatically establish the ssh reverse tunnel between sut and port map container. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/bin/entrypoint.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/providers/docker/bin/entrypoint.sh b/providers/docker/bin/entrypoint.sh index 63cb605..feb5edf 100755 --- a/providers/docker/bin/entrypoint.sh +++ b/providers/docker/bin/entrypoint.sh @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +mkdir -p /root/.ssh +cp /lkp/lkp/src/rootfs/addon/root/.ssh/id_rsa /root/.ssh/ sh /lkp/lkp/src/rootfs/addon/etc/init.d/lkp-bootstrap -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/matrix2.rb: fix rubocop warning
by Lu Weitao 21 Dec '20

21 Dec '20

[Why] lib/matrix2.rb:75:1: C: Metrics/MethodLength: Method has too many lines. [16/15] def create_matrix(job_list) ... Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..c712e32 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -64,6 +64,12 @@ def samples_fill_missing_zeros(value, size) value.concat([0] * (size - value.size)) end +def matrix_fill_miss_zeros(matrix, col_size) + matrix.each_value do |value| + samples_fill_missing_zeros(value, col_size) + end +end + # input: job_list # return: matrix of Hash(String, Array(Number)) # Eg: matrix: { @@ -86,9 +92,7 @@ def create_matrix(job_list) end end col_size = job_list.size - matrix.each_value do |value| - samples_fill_missing_zeros(value, col_size) - end + matrix_fill_miss_zeros(matrix, col_size) return matrix, suites end -- 2.23.0

2 1

Re: [PATCH compass-ci 2/3] assign-account-email: remove .bash_profile usage
by Luan Shengde 21 Dec '20

21 Dec '20

On Sat, Dec 19, 2020 at 11:23:43AM +0800, Li Ping wrote: >On Sat, Dec 19, 2020 at 11:09:54AM +0800, Wu Fengguang wrote: >>Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> >>--- >> container/mail-robot/lib/assign-account-email.rb | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> >>diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb >>index 77ed4e8..ee3b574 100755 >>--- a/container/mail-robot/lib/assign-account-email.rb >>+++ b/container/mail-robot/lib/assign-account-email.rb >>@@ -37,7 +37,7 @@ def build_apply_account_email(my_info) >> git clone https://gitee.com/wu_fengguang/lkp-tests.git >> cd lkp-tests >> make install >>- source ${HOME}/.bashrc && source ${HOME}/.bash_profile >>+ source ${HOME}/.${SHELL##*/}rc > > use ${HOME} will become root in email Use \\${HOME} will resolve this problem. Thanks, Luan Shengde > >> >> 3) submit job >> reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md >>-- >>2.23.0 >>

1 0

[PATCH lkp-tests] tests/build-pkg: add config file support
by Wang Yong 21 Dec '20

21 Dec '20

[Why] config file in some software need storage and recall now we can add config value in yaml and cp ${config} to ${cci_dir}/${benchmark}/ in PKGBUILD so that config file can be uploaded correctly $ cat test.yaml ... config: ${config} ... Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index addff9f7b..0a143813f 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -5,6 +5,7 @@ # - git_tag # - pkg_ver # upstream_url +# - config # We'll use the below global env vars, but not treat them as parameter to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ @@ -20,6 +21,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/distro/common +. $LKP_SRC/lib/http.sh check_vars() { @@ -31,6 +33,29 @@ check_vars() [ -n "$os_mount" ] || die "os_mount is empty" } +check_config_var() +{ + [ -n "$config" ] || return +} + +get_config() +{ + local cci_http_host=${INITRD_HTTP_HOST} + local cci_http_port=${INITRD_HTTP_PORT:-8800} + export cci_dir=/cci/config + local remote_dir="http://${cci_http_host}:${cci_http_port}${cci_dir}/${benchmark}" + + mkdir -p ${cci_dir} + + check_config_var + http_get_file "${remote_dir}/${config}" "${cci_dir}/${benchmark}/${config}" +} + +upload_config() +{ + upload_to_target_dir "${cci_dir}/${benchmark}" +} + mount_dest() { # the same image is mounted to cifs and nfs, the generated cgz files @@ -71,6 +96,13 @@ create_cgz_name() [ -n "$git_tag" ] && { cgz_name="${cgz_path}/${git_tag}.cgz" + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + export benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" } [ -n "$pkg_ver" ] && { @@ -131,7 +163,11 @@ build_source_pkg() PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf - [ "$?" == 0 ] || exit 1 + local build_state="$?" + + upload_config + + [ "${build_state}" == "0" ] || exit 1 create_softlink upload_to_target_dir ${PKG_MNT}/${pack_to}/${pkgname} } @@ -139,4 +175,5 @@ build_source_pkg() check_vars mount_dest request_pkg +get_config build_source_pkg -- 2.23.0

2 2

[PATCH compass-ci] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 19 Dec '20

19 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..f1b2c78 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,10 +76,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| matrix[key] = [] unless matrix[key] matrix[key] << value -- 2.23.0

2 2