- Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests 2/3] lib/job.rb: fix bug: add_define_files not work
by Li Ping 04 Nov '20

04 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths actually don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 86cf3fa1..5632ad88 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -624,6 +624,7 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + program = $1 if program =~ /^(.*)-\d+?/ # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

1 0

[PATCH v4 lkp-tests] jobs/hackbench-git.yaml: fix datasize and threads filed
by Zhang Yale 04 Nov '20

04 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So fix datasize to loops, fix threads to thread. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/hackbench-git.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/jobs/hackbench-git.yaml b/jobs/hackbench-git.yaml index 12943c04..e2320621 100644 --- a/jobs/hackbench-git.yaml +++ b/jobs/hackbench-git.yaml @@ -1,4 +1,4 @@ -suite: hackbench +suite: hackbench-git category: benchmark disable_latency_stats: 1 @@ -11,13 +11,13 @@ nr_threads: - 10 - 20 iterations: 10 -datasize: 1000 +loops: 1000 runtime: 300 -hackbench: +hackbench-git: mode: - process - - threads + - thread ipc: - pipe - socket -- 2.23.0

3 4

[PATCH lkp-tests] depends: add sysbench-mysql-dev
by Zhang Yu 04 Nov '20

04 Nov '20

[why] Dependency for code compilation. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/sysbench-mysql-dev | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 distro/depends/sysbench-mysql-dev diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev new file mode 100644 index 00000000..ca5b03d8 --- /dev/null +++ b/distro/depends/sysbench-mysql-dev @@ -0,0 +1,3 @@ +pkg-config +libmysqlclient-dev +mysql-server -- 2.23.0

1 0

[PATCH compass-ci] lkp-initrd: send the two vars necessary for rsync to rsync script
by Yu Chuan 04 Nov '20

04 Nov '20

[Why] When we rsync result lkp.cgz, we need the follow two vars: - lkp_initrd_user - ARCH Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/lkp-initrd/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/lkp-initrd/run b/container/lkp-initrd/run index a0c246f6ea31..34fe330727b0 100755 --- a/container/lkp-initrd/run +++ b/container/lkp-initrd/run @@ -27,5 +27,5 @@ echo "result: /srv/initrd/lkp/${lkp_initrd_user:-latest}/lkp-${ARCH}.cgz" # rsync lkp-${ARCH}.cgz if needed if [ -x /usr/local/bin/rsync-lkp-cgz ]; then - /usr/local/bin/rsync-lkp-cgz + /usr/local/bin/rsync-lkp-cgz "${lkp_initrd_user:-latest}" "${ARCH}" fi -- 2.23.0

1 0

[PATCH lkp-tests] pkg: add PKGBUILD of sysbench-mysql
by Zhang Yu 04 Nov '20

04 Nov '20

[why] The sysbench 1.0.19 version is required in test. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- pkg/sysbench-mysql | 1 - pkg/sysbench-mysql/PKGBUILD | 23 +++++++++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) delete mode 120000 pkg/sysbench-mysql create mode 100644 pkg/sysbench-mysql/PKGBUILD diff --git a/pkg/sysbench-mysql b/pkg/sysbench-mysql deleted file mode 120000 index a6f8719d..00000000 --- a/pkg/sysbench-mysql +++ /dev/null @@ -1 +0,0 @@ -sysbench \ No newline at end of file diff --git a/pkg/sysbench-mysql/PKGBUILD b/pkg/sysbench-mysql/PKGBUILD new file mode 100644 index 00000000..d9f342a5 --- /dev/null +++ b/pkg/sysbench-mysql/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench +pkgver=1.0 +pkgrel=19 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="https://github.com/akopytov/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("https://github.com/akopytov/sysbench/archive/$pkgver.$pkgrel.tar.gz") +md5sums=('2912bfe7238cac7351459019a84e2557') + +build() { + cd "$pkgname-$pkgver.$pkgrel" + ./autogen.sh + ./configure --with-mysql-includes=/usr/local/mysql/include/ --with-mysql-libs=/usr/local/mysql/lib/ --disable-dependency-tracking + make -j4 +} + +package() { + cd "$pkgname-$pkgver.$pkgrel" + make DESTDIR=$pkgdir install + [[ $os == openeuler ]] && mkdir -p $pkgdir/usr/lib64 + cp -rpf /usr/local/mysql/lib/* $pkgdir/usr/lib64/ +} -- 2.23.0

1 0

[PATCH compass-ci] sbin: fix git-bisect script get incomplete error_id info
by Cao Xueliang 04 Nov '20

04 Nov '20

[why] Before: use error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" get error_id="build-pkg.report-uri" After: use error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" get error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/git-bisect | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/sbin/git-bisect b/sbin/git-bisect index 4744a02..86f79d4 100755 --- a/sbin/git-bisect +++ b/sbin/git-bisect @@ -12,10 +12,9 @@ require_relative '../lib/mail_bisect_result' def parse_argv items = {} ARGV.each do |item| - key, value = item.split('=') + key, value = item.split('=', 2) if key && value - value_list = value.split(',') - items[key] = value_list.length > 1 ? value_list : value + items[key] = value end end items -- 2.23.0

2 1

[PATCH lkp-tests] fio: fix error when test with write mode
by Wei Jihui 04 Nov '20

04 Nov '20

[Why] when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. [How] add allow_mounted_write when write mode Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..f90963aa 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -116,6 +116,12 @@ $create_task ioscheduler=$ioscheduler" fi +if [[ "$rw" =~ "write" ]]; then + create_task="\ +$create_task +allow_mounted_write=1" +fi + if parse_bool -q "$time_based"; then create_task=" $create_task -- 2.23.0

2 1

[PATCH v3 lkp-tests] (job2sh.rb) add on_fail to job.sh
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh export_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 6e00b088..7d2c2d5c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -367,9 +367,7 @@ class Job2sh < Job @programs = available_programs(:workload_elements) - if !job - job = (@jobx || @job).clone - end + job ||= (@jobx || @job).clone @cur_func = :on_fail -- 2.23.0

2 1

[PATCH v3 lkp-tests] pkg: add PKGBUILD for libmicro
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/libmicro/PKGBUILD | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 pkg/libmicro/PKGBUILD diff --git a/pkg/libmicro/PKGBUILD b/pkg/libmicro/PKGBUILD new file mode 100644 index 00000000..8efc8b19 --- /dev/null +++ b/pkg/libmicro/PKGBUILD @@ -0,0 +1,19 @@ +pkgname=libMicro +pkgver=0.4 +pkgrel=0 +pkgdesc="A portable set of microbenchmarks focused on the syscall and system library interfaces." +arch=('i386' 'x86_64') +url="https://codeload.github.com/redhat-performance/libMicro" +license=('CDDL') +source=("https://codeload.github.com/redhat-performance/libMicro/zip/0.4.0-rh") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname-$pkgver.$pkgrel-rh" + make +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/libMicro" + cp -r "$srcdir/$pkgname-$pkgver.$pkgrel-rh/"* "$pkgdir/lkp/benchmarks/libMicro" +} -- 2.23.0

2 1

[PATCH v2 compass-ci 1/2] auto group jobs with user-defined template
by Lu Weitao 04 Nov '20

04 Nov '20

auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: auto group jobs_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list) + get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + new_job = get_new_job(job, metrics) + next if new_job.empty? + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + value = find_param_in_job(job, param) + group_key_list << value if value + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +def find_param_in_job(job, param) + return job[param] if job.key?(param) + + job.each_value do |v| + return v[param] if v.is_a?(Hash) && v.key?(param) + end +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job(job, metrics) + return {} unless job['stats'] + + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0

1 1