- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] container/sub-fluentd: fix heartbeat error
by Wu Zhende 05 Nov '20

05 Nov '20

[Error] fluent.warn: Unknown heartbeat response received from 172.17.0.1:24224. It may service out heartbeat_type default is tcp, delete the "heartbeat_type udp" to use tcp Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/sub-fluentd/docker-fluentd.conf | 1 - 1 file changed, 1 deletion(-) diff --git a/container/sub-fluentd/docker-fluentd.conf b/container/sub-fluentd/docker-fluentd.conf index c674fb1..09fa461 100644 --- a/container/sub-fluentd/docker-fluentd.conf +++ b/container/sub-fluentd/docker-fluentd.conf @@ -56,7 +56,6 @@ @type forward flush_interval 0 send_timeout 60 - heartbeat_type udp heartbeat_interval 1 recover_wait 10 hard_timeout 60 -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/3] lib/job.rb: no need puts the a warning
by Li Ping 04 Nov '20

04 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts --- lib/job.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..86cf3fa1 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -340,10 +340,8 @@ class Job lab_hosts_file = get_lab_hosts_file if lab_hosts_file hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file + elsif File.file?(hosts_file) + hosts_file else puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") nil -- 2.23.0

1 1

[PATCH v2 lkp-tests 3/3] lib/job2sh.rb: fix bug: can not find the file: pkg/$program/PKGBUILD
by Li Ping 04 Nov '20

04 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 15 ++++++--------- lib/job2sh.rb | 1 + 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5632ad88..e45ae682 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,18 +334,15 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + hosts_file = lab_hosts_file + return hosts_file if hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.file?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - elsif File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") end def include_files diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..7e6a5d7c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -350,6 +350,7 @@ class Job2sh < Job out_line out_line "\texport define_files='#{define_files.keys.join ' '}'" define_files.each do |file, val| + out_line "\tmkdir -p $LKP_SRC/$(dirname #{file})" out_line "\tcat > $LKP_SRC/#{file} <<'EOF'" out_line val out_line "EOF" -- 2.23.0

1 1

[PATCH v2 lkp-tests 2/3] lib/job.rb: fix bug: add_define_files not work
by Li Ping 04 Nov '20

04 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths actually don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 86cf3fa1..5632ad88 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -624,6 +624,7 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + program = $1 if program =~ /^(.*)-\d+?/ # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

1 0

[PATCH v4 lkp-tests] jobs/hackbench-git.yaml: fix datasize and threads filed
by Zhang Yale 04 Nov '20

04 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So fix datasize to loops, fix threads to thread. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/hackbench-git.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/jobs/hackbench-git.yaml b/jobs/hackbench-git.yaml index 12943c04..e2320621 100644 --- a/jobs/hackbench-git.yaml +++ b/jobs/hackbench-git.yaml @@ -1,4 +1,4 @@ -suite: hackbench +suite: hackbench-git category: benchmark disable_latency_stats: 1 @@ -11,13 +11,13 @@ nr_threads: - 10 - 20 iterations: 10 -datasize: 1000 +loops: 1000 runtime: 300 -hackbench: +hackbench-git: mode: - process - - threads + - thread ipc: - pipe - socket -- 2.23.0

3 4

[PATCH lkp-tests] depends: add sysbench-mysql-dev
by Zhang Yu 04 Nov '20

04 Nov '20

[why] Dependency for code compilation. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/sysbench-mysql-dev | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 distro/depends/sysbench-mysql-dev diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev new file mode 100644 index 00000000..ca5b03d8 --- /dev/null +++ b/distro/depends/sysbench-mysql-dev @@ -0,0 +1,3 @@ +pkg-config +libmysqlclient-dev +mysql-server -- 2.23.0

1 0

[PATCH compass-ci] lkp-initrd: send the two vars necessary for rsync to rsync script
by Yu Chuan 04 Nov '20

04 Nov '20

[Why] When we rsync result lkp.cgz, we need the follow two vars: - lkp_initrd_user - ARCH Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/lkp-initrd/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/lkp-initrd/run b/container/lkp-initrd/run index a0c246f6ea31..34fe330727b0 100755 --- a/container/lkp-initrd/run +++ b/container/lkp-initrd/run @@ -27,5 +27,5 @@ echo "result: /srv/initrd/lkp/${lkp_initrd_user:-latest}/lkp-${ARCH}.cgz" # rsync lkp-${ARCH}.cgz if needed if [ -x /usr/local/bin/rsync-lkp-cgz ]; then - /usr/local/bin/rsync-lkp-cgz + /usr/local/bin/rsync-lkp-cgz "${lkp_initrd_user:-latest}" "${ARCH}" fi -- 2.23.0

1 0

[PATCH lkp-tests] pkg: add PKGBUILD of sysbench-mysql
by Zhang Yu 04 Nov '20

04 Nov '20

[why] The sysbench 1.0.19 version is required in test. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- pkg/sysbench-mysql | 1 - pkg/sysbench-mysql/PKGBUILD | 23 +++++++++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) delete mode 120000 pkg/sysbench-mysql create mode 100644 pkg/sysbench-mysql/PKGBUILD diff --git a/pkg/sysbench-mysql b/pkg/sysbench-mysql deleted file mode 120000 index a6f8719d..00000000 --- a/pkg/sysbench-mysql +++ /dev/null @@ -1 +0,0 @@ -sysbench \ No newline at end of file diff --git a/pkg/sysbench-mysql/PKGBUILD b/pkg/sysbench-mysql/PKGBUILD new file mode 100644 index 00000000..d9f342a5 --- /dev/null +++ b/pkg/sysbench-mysql/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench +pkgver=1.0 +pkgrel=19 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="https://github.com/akopytov/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("https://github.com/akopytov/sysbench/archive/$pkgver.$pkgrel.tar.gz") +md5sums=('2912bfe7238cac7351459019a84e2557') + +build() { + cd "$pkgname-$pkgver.$pkgrel" + ./autogen.sh + ./configure --with-mysql-includes=/usr/local/mysql/include/ --with-mysql-libs=/usr/local/mysql/lib/ --disable-dependency-tracking + make -j4 +} + +package() { + cd "$pkgname-$pkgver.$pkgrel" + make DESTDIR=$pkgdir install + [[ $os == openeuler ]] && mkdir -p $pkgdir/usr/lib64 + cp -rpf /usr/local/mysql/lib/* $pkgdir/usr/lib64/ +} -- 2.23.0

1 0

[PATCH compass-ci] sbin: fix git-bisect script get incomplete error_id info
by Cao Xueliang 04 Nov '20

04 Nov '20

[why] Before: use error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" get error_id="build-pkg.report-uri" After: use error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" get error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/git-bisect | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/sbin/git-bisect b/sbin/git-bisect index 4744a02..86f79d4 100755 --- a/sbin/git-bisect +++ b/sbin/git-bisect @@ -12,10 +12,9 @@ require_relative '../lib/mail_bisect_result' def parse_argv items = {} ARGV.each do |item| - key, value = item.split('=') + key, value = item.split('=', 2) if key && value - value_list = value.split(',') - items[key] = value_list.length > 1 ? value_list : value + items[key] = value end end items -- 2.23.0

2 1

[PATCH lkp-tests] fio: fix error when test with write mode
by Wei Jihui 04 Nov '20

04 Nov '20

[Why] when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. [How] add allow_mounted_write when write mode Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..f90963aa 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -116,6 +116,12 @@ $create_task ioscheduler=$ioscheduler" fi +if [[ "$rw" =~ "write" ]]; then + create_task="\ +$create_task +allow_mounted_write=1" +fi + if parse_bool -q "$time_based"; then create_task=" $create_task -- 2.23.0

2 1

[PATCH v3 lkp-tests] (job2sh.rb) add on_fail to job.sh
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh export_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 6e00b088..7d2c2d5c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -367,9 +367,7 @@ class Job2sh < Job @programs = available_programs(:workload_elements) - if !job - job = (@jobx || @job).clone - end + job ||= (@jobx || @job).clone @cur_func = :on_fail -- 2.23.0

2 1

[PATCH v3 lkp-tests] pkg: add PKGBUILD for libmicro
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/libmicro/PKGBUILD | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 pkg/libmicro/PKGBUILD diff --git a/pkg/libmicro/PKGBUILD b/pkg/libmicro/PKGBUILD new file mode 100644 index 00000000..8efc8b19 --- /dev/null +++ b/pkg/libmicro/PKGBUILD @@ -0,0 +1,19 @@ +pkgname=libMicro +pkgver=0.4 +pkgrel=0 +pkgdesc="A portable set of microbenchmarks focused on the syscall and system library interfaces." +arch=('i386' 'x86_64') +url="https://codeload.github.com/redhat-performance/libMicro" +license=('CDDL') +source=("https://codeload.github.com/redhat-performance/libMicro/zip/0.4.0-rh") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname-$pkgver.$pkgrel-rh" + make +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/libMicro" + cp -r "$srcdir/$pkgname-$pkgver.$pkgrel-rh/"* "$pkgdir/lkp/benchmarks/libMicro" +} -- 2.23.0

2 1

[PATCH v2 compass-ci 1/2] auto group jobs with user-defined template
by Lu Weitao 04 Nov '20

04 Nov '20

auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: auto group jobs_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list) + get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + new_job = get_new_job(job, metrics) + next if new_job.empty? + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + value = find_param_in_job(job, param) + group_key_list << value if value + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +def find_param_in_job(job, param) + return job[param] if job.key?(param) + + job.each_value do |v| + return v[param] if v.is_a?(Hash) && v.key?(param) + end +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job(job, metrics) + return {} unless job['stats'] + + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0

1 1

[PATCH v2 compass-ci 09/10] container/mail-robot: start
by Luan Shengde 04 Nov '20

04 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..b65b002 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v /srv/cci/Maildir/.compass-ci:/srv/cci/Maildir/.compass-ci:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + --log-driver json-file + mail-robot +] + +cmd += ['ruby', 'run.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v2 compass-ci 08/10] container/mail-robot: build
by Luan Shengde 04 Nov '20

04 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci 04/10] container/mail-robot: assign-jumper-account.rb
by Luan Shengde 04 Nov '20

04 Nov '20

AssignJumperAccount apply account with apply_account_info apply_account_info - my_email - my_name - my_uuid - my_ssh_pubkey check account exists Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/assign-jumper-account.rb | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 container/mail-robot/assign-jumper-account.rb diff --git a/container/mail-robot/assign-jumper-account.rb b/container/mail-robot/assign-jumper-account.rb new file mode 100755 index 0000000..0728b89 --- /dev/null +++ b/container/mail-robot/assign-jumper-account.rb @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +# apply jumper account for user +# the enter point is apply_jumper_account +# the apply_account_info hash need include +# my_email, my_name, my_uuid, my_ssh_pubkey +class AssignJumperAccount + def initialize(apply_account_info) + @apply_account_info = apply_account_info + end + + def send_error_email(error_message) + @apply_account_info['error_message'] = error_message + send_err_mail = SendApplyAccountEmail.new(@apply_account_info) + send_err_mail.send_error_email + end + + def apply_jumper_account + account_info_str = %x(curl -XGET "#{ENV['JUMPER_HOST']}:#{ENV['JUMPER_PORT']}/assign_account" \ + -d '#{(a)apply_account_info.to_json}') + account_info = JSON.parse account_info_str + + check_account_info(account_info) + + return account_info + end + + def check_account_info(account_info) + return unless account_info['my_login_name'].nil? + + error_message = 'No more available jumper account.' + send_error_email(error_message) + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] create matrices-groups according to group jobs
by Lu Weitao 04 Nov '20

04 Nov '20

create matrices-groups according to auto group job_list background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: create groups_matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 05f96dc..b38b2fb 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -120,3 +120,27 @@ def combine_group_query_data(query_data, dims) end groups end + +# input: +# 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() +# 2. params from user's template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# return: group_matrix of Hash(String, Hash(String, matrix)) +def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) + job_list = query_data['hits']['hits'] + groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) + groups.each do |group_key, dims| + dims.each do |dim_key, jobs| + groups[group_key][dim_key] = create_matrix(jobs) + end + end + groups +end -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] create matrices-groups according to group jobs
by Lu Weitao 04 Nov '20

04 Nov '20

create matrices-groups according to auto group job_list background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: create groups_matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 05f96dc..b38b2fb 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -120,3 +120,27 @@ def combine_group_query_data(query_data, dims) end groups end + +# input: +# 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() +# 2. params from user's template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# return: group_matrix of Hash(String, Hash(String, matrix)) +def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) + job_list = query_data['hits']['hits'] + groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) + groups.each do |group_key, dims| + dims.each do |dim_key, jobs| + groups[group_key][dim_key] = create_matrix(jobs) + end + end + groups +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] execute on_fail when exit-code isn't 0
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] when a job-task fails(exit_code != 0), the on_fail function needs to be called to keep the program in the waiting state to user adjustment Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- bin/run-lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/run-lkp b/bin/run-lkp index 26669331..c4d43db4 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -97,6 +97,7 @@ elif [ "$run_job_exit_code" != 0 ]; then else set_job_state 'failed' fi + $LKP_DEBUG_PREFIX $job_script on_fail elif [ -f '/var/log/wtmp' ] && has_cmd last && last -n1 | grep -q -E '(pts/|tty)[0-9].*(still logged in)'; then set_job_state 'disturbed' disturbed=1 -- 2.23.0

1 0

[PATCH v3 lkp-tests] (job2sh.rb) add on_fail to job.sh
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh export_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..d128e8ef 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -357,6 +359,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs: add libmicro.yaml
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/libmicro.yaml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 jobs/libmicro.yaml diff --git a/jobs/libmicro.yaml b/jobs/libmicro.yaml new file mode 100644 index 00000000..446ab23b --- /dev/null +++ b/jobs/libmicro.yaml @@ -0,0 +1,4 @@ +suite: libmicro +category: benchmark + +libmicro: -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix bug when program-0 add_define_files not work
by Li Ping 04 Nov '20

04 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 86cf3fa1..67bd6d5e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -624,6 +624,9 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + if program =~ /^(.*)-\d+?/ + program = $1 + end # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

4 6

[PATCH v5 compass-ci 2/3] container/assign-account: get_account_info
by Luan Shengde 04 Nov '20

04 Nov '20

add function: config default yaml file [why]: easier for user to config the default yaml file [how]: write received user infos to the defaults yaml file: ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 51 +++++++++++++------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..b2d2474 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,35 +95,48 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) + setup_authorized_key(login_name, pub_key) unless pub_key.nil? + setup_default_yaml(login_name) + return jumper_account_info end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def setup_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + end + + def setup_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

2 3

[PATCH lkp-tests 2/2] lib/job2sh.rb: fix bug: can not find the file: pkg/$program/PKGBUILD
by Li Ping 04 Nov '20

04 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job2sh.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..7e6a5d7c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -350,6 +350,7 @@ class Job2sh < Job out_line out_line "\texport define_files='#{define_files.keys.join ' '}'" define_files.each do |file, val| + out_line "\tmkdir -p $LKP_SRC/$(dirname #{file})" out_line "\tcat > $LKP_SRC/#{file} <<'EOF'" out_line val out_line "EOF" -- 2.23.0

3 3

[PATCH lkp-tests] jobs: add yaml for libmicro
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/libmicro.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/libmicro.yaml diff --git a/jobs/libmicro.yaml b/jobs/libmicro.yaml new file mode 100644 index 00000000..9a577eb8 --- /dev/null +++ b/jobs/libmicro.yaml @@ -0,0 +1,5 @@ +suite: libmicro +testcase: libmicro +category: benchmark + +libmicro: -- 2.23.0

4 3