- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] container/sub-fluentd: fix heartbeat error
by Wu Zhende 05 Nov '20

05 Nov '20

[Error] fluent.warn: Unknown heartbeat response received from 172.17.0.1:24224. It may service out heartbeat_type default is tcp, delete the "heartbeat_type udp" to use tcp Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/sub-fluentd/docker-fluentd.conf | 1 - 1 file changed, 1 deletion(-) diff --git a/container/sub-fluentd/docker-fluentd.conf b/container/sub-fluentd/docker-fluentd.conf index c674fb1..09fa461 100644 --- a/container/sub-fluentd/docker-fluentd.conf +++ b/container/sub-fluentd/docker-fluentd.conf @@ -56,7 +56,6 @@ @type forward flush_interval 0 send_timeout 60 - heartbeat_type udp heartbeat_interval 1 recover_wait 10 hard_timeout 60 -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/3] lib/job.rb: no need puts the a warning
by Li Ping 04 Nov '20

04 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts --- lib/job.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..86cf3fa1 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -340,10 +340,8 @@ class Job lab_hosts_file = get_lab_hosts_file if lab_hosts_file hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file + elsif File.file?(hosts_file) + hosts_file else puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") nil -- 2.23.0

1 1

[PATCH v2 lkp-tests 3/3] lib/job2sh.rb: fix bug: can not find the file: pkg/$program/PKGBUILD
by Li Ping 04 Nov '20

04 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 15 ++++++--------- lib/job2sh.rb | 1 + 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5632ad88..e45ae682 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,18 +334,15 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + hosts_file = lab_hosts_file + return hosts_file if hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.file?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - elsif File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") end def include_files diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..7e6a5d7c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -350,6 +350,7 @@ class Job2sh < Job out_line out_line "\texport define_files='#{define_files.keys.join ' '}'" define_files.each do |file, val| + out_line "\tmkdir -p $LKP_SRC/$(dirname #{file})" out_line "\tcat > $LKP_SRC/#{file} <<'EOF'" out_line val out_line "EOF" -- 2.23.0

1 1

[PATCH v2 lkp-tests 2/3] lib/job.rb: fix bug: add_define_files not work
by Li Ping 04 Nov '20

04 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths actually don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 86cf3fa1..5632ad88 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -624,6 +624,7 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + program = $1 if program =~ /^(.*)-\d+?/ # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

1 0

[PATCH v4 lkp-tests] jobs/hackbench-git.yaml: fix datasize and threads filed
by Zhang Yale 04 Nov '20

04 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So fix datasize to loops, fix threads to thread. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/hackbench-git.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/jobs/hackbench-git.yaml b/jobs/hackbench-git.yaml index 12943c04..e2320621 100644 --- a/jobs/hackbench-git.yaml +++ b/jobs/hackbench-git.yaml @@ -1,4 +1,4 @@ -suite: hackbench +suite: hackbench-git category: benchmark disable_latency_stats: 1 @@ -11,13 +11,13 @@ nr_threads: - 10 - 20 iterations: 10 -datasize: 1000 +loops: 1000 runtime: 300 -hackbench: +hackbench-git: mode: - process - - threads + - thread ipc: - pipe - socket -- 2.23.0

3 4

[PATCH lkp-tests] depends: add sysbench-mysql-dev
by Zhang Yu 04 Nov '20

04 Nov '20

[why] Dependency for code compilation. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/sysbench-mysql-dev | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 distro/depends/sysbench-mysql-dev diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev new file mode 100644 index 00000000..ca5b03d8 --- /dev/null +++ b/distro/depends/sysbench-mysql-dev @@ -0,0 +1,3 @@ +pkg-config +libmysqlclient-dev +mysql-server -- 2.23.0

1 0

[PATCH compass-ci] lkp-initrd: send the two vars necessary for rsync to rsync script
by Yu Chuan 04 Nov '20

04 Nov '20

[Why] When we rsync result lkp.cgz, we need the follow two vars: - lkp_initrd_user - ARCH Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/lkp-initrd/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/lkp-initrd/run b/container/lkp-initrd/run index a0c246f6ea31..34fe330727b0 100755 --- a/container/lkp-initrd/run +++ b/container/lkp-initrd/run @@ -27,5 +27,5 @@ echo "result: /srv/initrd/lkp/${lkp_initrd_user:-latest}/lkp-${ARCH}.cgz" # rsync lkp-${ARCH}.cgz if needed if [ -x /usr/local/bin/rsync-lkp-cgz ]; then - /usr/local/bin/rsync-lkp-cgz + /usr/local/bin/rsync-lkp-cgz "${lkp_initrd_user:-latest}" "${ARCH}" fi -- 2.23.0

1 0

[PATCH lkp-tests] pkg: add PKGBUILD of sysbench-mysql
by Zhang Yu 04 Nov '20

04 Nov '20

[why] The sysbench 1.0.19 version is required in test. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- pkg/sysbench-mysql | 1 - pkg/sysbench-mysql/PKGBUILD | 23 +++++++++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) delete mode 120000 pkg/sysbench-mysql create mode 100644 pkg/sysbench-mysql/PKGBUILD diff --git a/pkg/sysbench-mysql b/pkg/sysbench-mysql deleted file mode 120000 index a6f8719d..00000000 --- a/pkg/sysbench-mysql +++ /dev/null @@ -1 +0,0 @@ -sysbench \ No newline at end of file diff --git a/pkg/sysbench-mysql/PKGBUILD b/pkg/sysbench-mysql/PKGBUILD new file mode 100644 index 00000000..d9f342a5 --- /dev/null +++ b/pkg/sysbench-mysql/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench +pkgver=1.0 +pkgrel=19 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="https://github.com/akopytov/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("https://github.com/akopytov/sysbench/archive/$pkgver.$pkgrel.tar.gz") +md5sums=('2912bfe7238cac7351459019a84e2557') + +build() { + cd "$pkgname-$pkgver.$pkgrel" + ./autogen.sh + ./configure --with-mysql-includes=/usr/local/mysql/include/ --with-mysql-libs=/usr/local/mysql/lib/ --disable-dependency-tracking + make -j4 +} + +package() { + cd "$pkgname-$pkgver.$pkgrel" + make DESTDIR=$pkgdir install + [[ $os == openeuler ]] && mkdir -p $pkgdir/usr/lib64 + cp -rpf /usr/local/mysql/lib/* $pkgdir/usr/lib64/ +} -- 2.23.0

1 0

[PATCH compass-ci] sbin: fix git-bisect script get incomplete error_id info
by Cao Xueliang 04 Nov '20

04 Nov '20

[why] Before: use error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" get error_id="build-pkg.report-uri" After: use error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" get error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/git-bisect | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/sbin/git-bisect b/sbin/git-bisect index 4744a02..86f79d4 100755 --- a/sbin/git-bisect +++ b/sbin/git-bisect @@ -12,10 +12,9 @@ require_relative '../lib/mail_bisect_result' def parse_argv items = {} ARGV.each do |item| - key, value = item.split('=') + key, value = item.split('=', 2) if key && value - value_list = value.split(',') - items[key] = value_list.length > 1 ? value_list : value + items[key] = value end end items -- 2.23.0

2 1

[PATCH lkp-tests] fio: fix error when test with write mode
by Wei Jihui 04 Nov '20

04 Nov '20

[Why] when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. [How] add allow_mounted_write when write mode Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..f90963aa 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -116,6 +116,12 @@ $create_task ioscheduler=$ioscheduler" fi +if [[ "$rw" =~ "write" ]]; then + create_task="\ +$create_task +allow_mounted_write=1" +fi + if parse_bool -q "$time_based"; then create_task=" $create_task -- 2.23.0

2 1

[PATCH v3 lkp-tests] (job2sh.rb) add on_fail to job.sh
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh export_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 6e00b088..7d2c2d5c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -367,9 +367,7 @@ class Job2sh < Job @programs = available_programs(:workload_elements) - if !job - job = (@jobx || @job).clone - end + job ||= (@jobx || @job).clone @cur_func = :on_fail -- 2.23.0

2 1

[PATCH v3 lkp-tests] pkg: add PKGBUILD for libmicro
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/libmicro/PKGBUILD | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 pkg/libmicro/PKGBUILD diff --git a/pkg/libmicro/PKGBUILD b/pkg/libmicro/PKGBUILD new file mode 100644 index 00000000..8efc8b19 --- /dev/null +++ b/pkg/libmicro/PKGBUILD @@ -0,0 +1,19 @@ +pkgname=libMicro +pkgver=0.4 +pkgrel=0 +pkgdesc="A portable set of microbenchmarks focused on the syscall and system library interfaces." +arch=('i386' 'x86_64') +url="https://codeload.github.com/redhat-performance/libMicro" +license=('CDDL') +source=("https://codeload.github.com/redhat-performance/libMicro/zip/0.4.0-rh") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname-$pkgver.$pkgrel-rh" + make +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/libMicro" + cp -r "$srcdir/$pkgname-$pkgver.$pkgrel-rh/"* "$pkgdir/lkp/benchmarks/libMicro" +} -- 2.23.0

2 1

[PATCH v2 compass-ci 1/2] auto group jobs with user-defined template
by Lu Weitao 04 Nov '20

04 Nov '20

auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: auto group jobs_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list) + get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + new_job = get_new_job(job, metrics) + next if new_job.empty? + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + value = find_param_in_job(job, param) + group_key_list << value if value + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +def find_param_in_job(job, param) + return job[param] if job.key?(param) + + job.each_value do |v| + return v[param] if v.is_a?(Hash) && v.key?(param) + end +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job(job, metrics) + return {} unless job['stats'] + + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0

1 1

[PATCH v2 compass-ci 09/10] container/mail-robot: start
by Luan Shengde 04 Nov '20

04 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..b65b002 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v /srv/cci/Maildir/.compass-ci:/srv/cci/Maildir/.compass-ci:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + --log-driver json-file + mail-robot +] + +cmd += ['ruby', 'run.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v2 compass-ci 08/10] container/mail-robot: build
by Luan Shengde 04 Nov '20

04 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci 04/10] container/mail-robot: assign-jumper-account.rb
by Luan Shengde 04 Nov '20

04 Nov '20

AssignJumperAccount apply account with apply_account_info apply_account_info - my_email - my_name - my_uuid - my_ssh_pubkey check account exists Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/assign-jumper-account.rb | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 container/mail-robot/assign-jumper-account.rb diff --git a/container/mail-robot/assign-jumper-account.rb b/container/mail-robot/assign-jumper-account.rb new file mode 100755 index 0000000..0728b89 --- /dev/null +++ b/container/mail-robot/assign-jumper-account.rb @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +# apply jumper account for user +# the enter point is apply_jumper_account +# the apply_account_info hash need include +# my_email, my_name, my_uuid, my_ssh_pubkey +class AssignJumperAccount + def initialize(apply_account_info) + @apply_account_info = apply_account_info + end + + def send_error_email(error_message) + @apply_account_info['error_message'] = error_message + send_err_mail = SendApplyAccountEmail.new(@apply_account_info) + send_err_mail.send_error_email + end + + def apply_jumper_account + account_info_str = %x(curl -XGET "#{ENV['JUMPER_HOST']}:#{ENV['JUMPER_PORT']}/assign_account" \ + -d '#{(a)apply_account_info.to_json}') + account_info = JSON.parse account_info_str + + check_account_info(account_info) + + return account_info + end + + def check_account_info(account_info) + return unless account_info['my_login_name'].nil? + + error_message = 'No more available jumper account.' + send_error_email(error_message) + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] create matrices-groups according to group jobs
by Lu Weitao 04 Nov '20

04 Nov '20

create matrices-groups according to auto group job_list background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: create groups_matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 05f96dc..b38b2fb 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -120,3 +120,27 @@ def combine_group_query_data(query_data, dims) end groups end + +# input: +# 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() +# 2. params from user's template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# return: group_matrix of Hash(String, Hash(String, matrix)) +def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) + job_list = query_data['hits']['hits'] + groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) + groups.each do |group_key, dims| + dims.each do |dim_key, jobs| + groups[group_key][dim_key] = create_matrix(jobs) + end + end + groups +end -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] create matrices-groups according to group jobs
by Lu Weitao 04 Nov '20

04 Nov '20

create matrices-groups according to auto group job_list background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: create groups_matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 05f96dc..b38b2fb 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -120,3 +120,27 @@ def combine_group_query_data(query_data, dims) end groups end + +# input: +# 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() +# 2. params from user's template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# return: group_matrix of Hash(String, Hash(String, matrix)) +def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) + job_list = query_data['hits']['hits'] + groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) + groups.each do |group_key, dims| + dims.each do |dim_key, jobs| + groups[group_key][dim_key] = create_matrix(jobs) + end + end + groups +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] execute on_fail when exit-code isn't 0
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] when a job-task fails(exit_code != 0), the on_fail function needs to be called to keep the program in the waiting state to user adjustment Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- bin/run-lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/run-lkp b/bin/run-lkp index 26669331..c4d43db4 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -97,6 +97,7 @@ elif [ "$run_job_exit_code" != 0 ]; then else set_job_state 'failed' fi + $LKP_DEBUG_PREFIX $job_script on_fail elif [ -f '/var/log/wtmp' ] && has_cmd last && last -n1 | grep -q -E '(pts/|tty)[0-9].*(still logged in)'; then set_job_state 'disturbed' disturbed=1 -- 2.23.0

1 0

[PATCH v3 lkp-tests] (job2sh.rb) add on_fail to job.sh
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh export_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..d128e8ef 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -357,6 +359,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs: add libmicro.yaml
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/libmicro.yaml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 jobs/libmicro.yaml diff --git a/jobs/libmicro.yaml b/jobs/libmicro.yaml new file mode 100644 index 00000000..446ab23b --- /dev/null +++ b/jobs/libmicro.yaml @@ -0,0 +1,4 @@ +suite: libmicro +category: benchmark + +libmicro: -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix bug when program-0 add_define_files not work
by Li Ping 04 Nov '20

04 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 86cf3fa1..67bd6d5e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -624,6 +624,9 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + if program =~ /^(.*)-\d+?/ + program = $1 + end # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

4 6

[PATCH v5 compass-ci 2/3] container/assign-account: get_account_info
by Luan Shengde 04 Nov '20

04 Nov '20

add function: config default yaml file [why]: easier for user to config the default yaml file [how]: write received user infos to the defaults yaml file: ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 51 +++++++++++++------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..b2d2474 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,35 +95,48 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) + setup_authorized_key(login_name, pub_key) unless pub_key.nil? + setup_default_yaml(login_name) + return jumper_account_info end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def setup_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + end + + def setup_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

2 3

[PATCH lkp-tests 2/2] lib/job2sh.rb: fix bug: can not find the file: pkg/$program/PKGBUILD
by Li Ping 04 Nov '20

04 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job2sh.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..7e6a5d7c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -350,6 +350,7 @@ class Job2sh < Job out_line out_line "\texport define_files='#{define_files.keys.join ' '}'" define_files.each do |file, val| + out_line "\tmkdir -p $LKP_SRC/$(dirname #{file})" out_line "\tcat > $LKP_SRC/#{file} <<'EOF'" out_line val out_line "EOF" -- 2.23.0

3 3

[PATCH lkp-tests] jobs: add yaml for libmicro
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/libmicro.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/libmicro.yaml diff --git a/jobs/libmicro.yaml b/jobs/libmicro.yaml new file mode 100644 index 00000000..9a577eb8 --- /dev/null +++ b/jobs/libmicro.yaml @@ -0,0 +1,5 @@ +suite: libmicro +testcase: libmicro +category: benchmark + +libmicro: -- 2.23.0

4 3

[PATCH v2 lkp-tests] lib/job.rb: no need puts the a warning
by Li Ping 04 Nov '20

04 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts --- lib/job.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..86cf3fa1 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -340,10 +340,8 @@ class Job lab_hosts_file = get_lab_hosts_file if lab_hosts_file hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file + elsif File.file?(hosts_file) + hosts_file else puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") nil -- 2.23.0

2 2

[PATCH v5 compass-ci 3/3] container/assign-account: answerback-email.rb
by Luan Shengde 04 Nov '20

04 Nov '20

1. disable use golbal variable apply_info [why]: rubocop: Style/GlobalVars: Do not introduce global variables. [how]: use local variables instead of global variables 2. fix OptionParser argument: [why]: when write this: opts.on('-e|--email email_address', 'appoint email address') will lead to could not identify the options ./answerback-email.rb --email xxx(a)163.com ./answerback-email.rb:72:in `<main>': invalid option: --email (OptionParser::InvalidOption) [how]: write seprately like: opts.on('-e email_address', '--email email_address', 'appoint email address') 3. add transfer user info when execute apply account command [why]: when applying account, the assign-account service need to write the my info to user's default config file ~/.config/compass-ci/defaults/account.yaml my info: - my_email - my_name - my_uuid [how]: transfer the user info along with the pub_key when sending apply account request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 92 ++++++++++++-------- 1 file changed, 55 insertions(+), 37 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..bda627d 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,58 +12,65 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +apply_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, apply_info) mail_content = Mail.read(email_file) + apply_info['my_email'] = mail_content.from[0] + apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + apply_info end options = OptionParser.new do |opts| - opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key_file] [--raw-email email_file]\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" opts.banner += " -e or -f is required\n" opts.banner += ' -s is optional when use -e' opts.separator '' opts.separator 'options:' - opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + apply_info['my_email'] = email_address + # when apply account with email address, will get no user name + apply_info['my_name'] = '' end - opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ + 'ssh pub_key file, enable password-less login') do |pub_key_file| + apply_info['my_ssh_pubkey'] = File.read(pub_key_file) end - opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + init_info(email_file, apply_info) end - opts.on_tail('-h|--help', 'show this message') do + opts.on_tail('-h', '--help', 'show this message') do puts opts exit end @@ -71,10 +78,10 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: @@ -82,10 +89,10 @@ def build_message(email, acct_infos) You can use the following command to login the jumper server: login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} account password: - #{acct_infos['passwd']} + #{account_info['my_password']} regards compass-ci @@ -94,26 +101,37 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(apply_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(apply_info) message = "No email address specified\n" message += "use -e email_address add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? - - acct_info = account_info($apply_info['my_ssh_pubkey']) - - message = build_message($apply_info['my_email'], acct_info) + raise message if apply_info['my_email'].nil? + + account_info = apply_account(apply_info) + my_info = { + 'my_email' => apply_info['my_email'], + 'my_name' => apply_info['my_name'], + # there is no need to add a commit url for this tool + # the es has the key: my_commit_url + 'my_commit_url' => '', + 'my_login_name' => account_info['my_login_name'], + 'my_uuid' => apply_info['my_uuid'] + } + + store_account_info(my_info) + message = build_message(apply_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(apply_info) -- 2.23.0

2 2

[PATCH compass-ci] kernel_params.cr: add sched_steal_node_limit=8 when test mysql optimization kernel
by Xu Xijian 04 Nov '20

04 Nov '20

[why] When test the kernel with mysql optimization, the parameter "sched_steal_node_limit=8" is required to be added into kernel params. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 54438b5..546f012 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -31,6 +31,9 @@ class Job private def set_kernel_params self["kernel_params"] = " #{kernel_common_params()} #{kernel_append_root} #{kernel_console()}" + if "#{linux_vmlinuz_path}".includes?("mysql") + self["kernel_params"] += " sched_steal_node_limit=8" + end end end -- 2.23.0

3 3

[PATCH compass-ci 3/8] container/mail-robot: apply-account-component.rb
by Luan Shengde 04 Nov '20

04 Nov '20

ParseApplyAccountEmail parse commit url and pub_key check out commit url check commit url exists check base url in upstream-repos check commit available gitee.com clone the repo and check the commit exists non-gitee.com check the feedback of curl command to check the commit url available check out pub_key extract the attachment of the pub_key ApplyAccount apply account with apply_account_info check account SendMail send uuid email build email message send email send error email build error email message send error email Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- .../mail-robot/apply-account-component.rb | 232 ++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100755 container/mail-robot/apply-account-component.rb diff --git a/container/mail-robot/apply-account-component.rb b/container/mail-robot/apply-account-component.rb new file mode 100755 index 0000000..98272e7 --- /dev/null +++ b/container/mail-robot/apply-account-component.rb @@ -0,0 +1,232 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check whether there is a commit url and ssh pub_key in the email +# apply account on junper server +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + @my_email = mail_content.from[0] + @my_name = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + end + + def parse_commit_url_pub_key + commit_url = check_out_commit_url + pub_key = check_out_pub_key + + return commit_url, pub_key + end + + def send_error_email(error_message) + send_error_email_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_info' => '', + 'error_message' => error_message + } + send_err_mail = SendMail.new(send_error_email_info) + send_err_mail.send_error_email + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = 'No matched commit url found' + send_error_email(error_message) + + raise error_message + end + + def check_out_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + check_base_url_in_upstream_repos('/c/upstream-repos', base_url) + check_commit_available(url, base_url) + + return url + end + + def check_base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = 'The url is not in upstream-repo list' + send_error_email(error_message) + + raise error_message + end + + def check_commit_available(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + check_gitee_commit(url, base_url) + else + check_non_gitee_commit(url) + end + end + + def check_non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_email + + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_gitee_commit(url, base_url) + repo_dir = url.split('/')[-3] + repo_url = [base_url, 'git'].join('.') + commit_id = url.split('/')[-1] + + Dir.chdir ENV['ROBOT_TMP_REPOS_DIR'] + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_email + + FileUtils.rm_rf repo_dir + + check_gitee_commit_exist(email_index) + end + + def check_gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_out_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + check_pub_key(pub_key) + + return pub_key + end + + def check_pub_key(pub_key) + return unless pub_key.nil? + + error_message = 'No pub_key found' + send_error_email(error_message) + + raise error_message + end +end + +# apply jumper account for user +class ApplyAccount + def initialize(apply_account_info) + @apply_account_info = apply_account_info + end + + def apply_jumper_account + account_info_str = %x(curl -XGET "#{ENV['JUMPER_HOST']}:#{ENV['JUMPER_PORT']}/assign_account" \ + -d '#{(a)apply_account_info.to_json}') + account_info = JSON.parse account_info_str + + check_account_info(account_info) + + return account_info + end + + def check_account_info(account_info) + return unless account_info['account'].nil? + + error_message = 'No more available jumper account.' + send_error_email(error_message) + + raise error_message + end +end + +# build mail data and send mail +class SendMail + def initialize(send_mail_info) + @send_mail_host = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + @my_email = send_mail_info['my_email'] + @my_name = send_mail_info['my_name'] + @error_message = send_mail_info['error_message'] + @my_info = send_mail_info['my_info'] + end + + def send_uuid_email + email_msg = <<~EMAIL_MESSAGE + To: #{@my_email} + Subject: [compass-ci] Account Ready + + Dear #{@my_name}, + + Thank you for joining us. + You can use the following info to submit jobs: + + 1) setup default config + run the following command to add the below setup to default config file + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/\\${USER}.yaml <<-EOF + my_uuid: #{@my_info['my_uuid']} + my_email: #{@my_info['my_email']} + my_name: #{@my_info['my_name']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + find how to write job yaml + submit jobs/netperf.yaml + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_error_email + email_msg = <<~EMAIL_MESSAGE + To: #{@my_email} + Subject: apply account failed + + Dear #{@my_name} + + Your application for account failed with following errors + + #{@error_message} + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_mail(email_msg) + %x(curl -XPOST "#{@send_mail_host}:#{ENV['SEND_MAIL_PORT']}/send_mail_text" -d "#{email_msg}") + end +end -- 2.23.0

2 8

[PATCH v2 compass-ci] container/sub-fluentd: fix heartbeat error
by Wu Zhende 04 Nov '20

04 Nov '20

[Error] fluent.warn: Unknown heartbeat response received from 172.17.0.1:24224. It may service out heartbeat_type default is tcp, delete the "heartbeat_type udp" to use tcp Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/sub-fluentd/docker-fluentd.conf | 1 - 1 file changed, 1 deletion(-) diff --git a/container/sub-fluentd/docker-fluentd.conf b/container/sub-fluentd/docker-fluentd.conf index c674fb1..09fa461 100644 --- a/container/sub-fluentd/docker-fluentd.conf +++ b/container/sub-fluentd/docker-fluentd.conf @@ -56,7 +56,6 @@ @type forward flush_interval 0 send_timeout 60 - heartbeat_type udp heartbeat_interval 1 recover_wait 10 hard_timeout 60 -- 2.23.0

2 1

[PATCH v9 compass-ci 2/2] kernel_version.md: explain key "kernel_version"
by Xu Xijian 04 Nov '20

04 Nov '20

[why] Explain the meaning of new key "kernel_version" for scheduler, including what's the mapping between its typical values and actual files in the disk and how to set it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 doc/job/kernel_version.md diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md new file mode 100644 index 0000000..f6a5f4d --- /dev/null +++ b/doc/job/kernel_version.md @@ -0,0 +1,39 @@ +# kernel_version + +Meaning: +- Every os has its kernel, however an os can start with different kernels according to different need. +- kernel_version is a key for users to specify a kernel version. +- If kernel_version is not given by users, it will use the default one. +- Here are the kernel versions we support for different os: + openeuler/aarch64/1.0 + - 4.19.90-2001(default) + + openeuler/aarch64/20.03 + - 4.19.90-2003(default) + - 4.19.90-mysql + - 4.19.90-nginx + + debian/aarch64/sid + - 5.4.0-4-arm64(default) + - 5.8.0-1-arm64 + + centos/aarch64/7.6.1810 + - 4.14.0-115.el7a.0.1(default) + + centos/aarch64/7.8.2003 + - 4.18.0-147.8.1.el7(default) + + centos/aarch64/8.1.1911 + - 4.18.0-147.el8(default) + +Related files: +- In initramfs boot process, every kernel version is related with a vmlinuz, module and headers. +- Files like below under $boot_dir, an example $boot_dir can be "/srv/os/openeuler/aarch64/20.03/boot". +├── headers-4.19.90-2003.cgz +├── headers.cgz -> headers-4.19.90-2003.cgz +├── modules-4.19.90-2003.cgz +├── modules.cgz -> modules-4.19.90-2003.cgz +├── vmlinuz-4.19.90-2003 + +Usage example: +- submit iperf.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 -- 2.23.0

4 4

[PATCH compass-ci] container/master-fluentd: get logging-es info from yaml
by Wu Zhende 04 Nov '20

04 Nov '20

[Why] "logging-es" and "master-fluentd" may be on different services and may be performed over the network. In this scenario, the logging-es information needs to be obtained from the configuretion file. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 4 ++-- container/master-fluentd/start | 14 +++++++++++++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index 4660988..a57d4cd 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -40,8 +40,8 @@ <store> @type elasticsearch - host 172.17.0.1 - port 9202 + host "#{ENV['LOGGING_ES_HOST']}" + port "#{ENV['LOGGING_ES_PORT']}" suppress_type_name true flush_interval 1s index_name ${tag} diff --git a/container/master-fluentd/start b/container/master-fluentd/start index b3e3dbb..a05881c 100755 --- a/container/master-fluentd/start +++ b/container/master-fluentd/start @@ -3,9 +3,20 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'set' require_relative '../defconfig.rb' -docker_rm "master-fluentd" +names = Set.new %w[ + LOGGING_ES_HOST + LOGGING_ES_PORT +] + +defaults = relevant_defaults(names) +defaults['LOGGING_ES_HOST'] ||= '172.17.0.1' +defaults['LOGGING_ES_PORT'] ||= '9202' +env = docker_env(defaults) + +docker_rm 'master-fluentd' cmd = %w[ docker run @@ -13,6 +24,7 @@ cmd = %w[ --name master-fluentd -v /etc/localtime:/etc/localtime:ro -d +] + env + %w[ -u 1090:1090 -p 24224:24224/tcp -p 24224:24224/udp -- 2.23.0

2 1

[PATCH v2 lab-z9] hosts: fix taishan200-2280-2s48p-256g--a102 to taishan200-2280-2s48p-512g--a102
by Zhang Yu 04 Nov '20

04 Nov '20

[why] The memory of taishan200-2280-2s48p-256g--a102 is 512g, so need to fix 256g to 512g. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename hosts/{taishan200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} (100%) diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-512g--a102 similarity index 100% rename from hosts/taishan200-2280-2s48p-256g--a102 rename to hosts/taishan200-2280-2s48p-512g--a102 -- 2.23.0

2 2

[PATCH lkp-tests 1/2] lib/job.rb: no need puts the a warning
by Li Ping 04 Nov '20

04 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts --- lib/job.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..86cf3fa1 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -340,10 +340,8 @@ class Job lab_hosts_file = get_lab_hosts_file if lab_hosts_file hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file + elsif File.file?(hosts_file) + hosts_file else puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") nil -- 2.23.0

1 0

[PATCH lkp-tests] libmicro: add PKGBUILD for libmicro
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/libmicro/PKGBUILD | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 pkg/libmicro/PKGBUILD diff --git a/pkg/libmicro/PKGBUILD b/pkg/libmicro/PKGBUILD new file mode 100644 index 00000000..e421c402 --- /dev/null +++ b/pkg/libmicro/PKGBUILD @@ -0,0 +1,20 @@ +pkgname=libMicro +pkgver=0.4 +pkgrel=0 +pkgdesc="A portable set of microbenchmarks focused on the syscall and system library interfaces." +arch=('i386' 'x86_64') +url="https://github.com/rzezeski/libMicro" +license=('CDDL') +source=("https://codeload.github.com/redhat-performance/libMicro/zip/0.4.0-rh") +md5sums=('SKIP') + + +build() { + cd "$srcdir/$pkgname-$pkgver.$pkgrel-rh" + make +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/libMicro" + cp -r "$srcdir/$pkgname-$pkgver.$pkgrel-rh/"* "$pkgdir/lkp/benchmarks/libMicro" +} -- 2.23.0

2 2

[PATCH compass-ci] container/sub-fluentd: fix heartbeat error
by Wu Zhende 04 Nov '20

04 Nov '20

[Error] fluent.warn: Unknown heartbeat response received from 172.17.0.1:24224. It may service out Can't use udp to send heartbeat package. Change to tcp. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/sub-fluentd/docker-fluentd.conf | 1 - 1 file changed, 1 deletion(-) diff --git a/container/sub-fluentd/docker-fluentd.conf b/container/sub-fluentd/docker-fluentd.conf index c674fb1..09fa461 100644 --- a/container/sub-fluentd/docker-fluentd.conf +++ b/container/sub-fluentd/docker-fluentd.conf @@ -56,7 +56,6 @@ @type forward flush_interval 0 send_timeout 60 - heartbeat_type udp heartbeat_interval 1 recover_wait 10 hard_timeout 60 -- 2.23.0

3 2

[PATCH v5 lkp-tests] tests/build-pkg: modify save location for *.cgz file
by Liu Shaofei 04 Nov '20

04 Nov '20

[why] the path of *.cgz file generated by build-pkg script cannot be obtained during test. So the path of cgz file need to be changed. [how] below is example for openeuler: latest.cgz -> f71820d6055373c9b9381615e28a28f307bb2df4.cgz f71820d6055373c9b9381615e28a28f307bb2df4.cgz v1.8.1.cgz -> 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz v1.5.1.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz df0584e46db9e7b412626deab10b923504745b2c.cgz Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..7c494a15 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,15 +23,19 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount $LKP_SERVER:$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } @@ -75,12 +81,22 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 +} + +create_softlink() +{ + [ -n "$upstream_tag" ] || return 0 + [ -e "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" ] && return 0 + + ln -sf "$(basename $(realpath ${cgz_name}))" "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" || return 0 + echo "create soft link: $PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz -> $(basename $(realpath ${cgz_name}))" } check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

4 5

[PATCH v3 compass-ci] sparrow/3-code: add ENV LOGGING_ES_HOST and LOGGING_ES_PORT
by Liu Yinsi 04 Nov '20

04 Nov '20

need LOGGING_ES_HOST and LOGGING_ES_PORT to start logging-es service. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 64f34f8..f4e4fa9 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -13,6 +13,8 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${MASTER_FLUENTD_PORT:=24224} : ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} +: ${LOGGING_ES_HOST:=$server_ip} +: ${LOGGING_ES_PORT:=9202} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -28,9 +30,11 @@ MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT DOCKER_REGISTRY_HOST: $DOCKER_REGISTRY_HOST DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT +LOGGING_ES_HOST: $LOGGING_ES_HOST +LOGGING_ES_PORT: $LOGGING_ES_PORT EOF -cat > /etc/profile.d/compass.sh <<'EOF' +cat > /etc/profile.d/compass.sh <<EOF export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci export REPO_SRC=/c/upstream-repos -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/3-code: add ENV LOGGING_ES_HOST and LOGGING_ES_PORT
by Liu Yinsi 04 Nov '20

04 Nov '20

need LOGGING_ES_HOST and LOGGING_ES_PORT to start logging-es service. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 64f34f8..dc329fe 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -13,6 +13,8 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${MASTER_FLUENTD_PORT:=24224} : ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} +: ${LOGGING_ES_HOST:=$server_ip} +: ${LOGGING_ES_PORT:=9202} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -28,6 +30,8 @@ MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT DOCKER_REGISTRY_HOST: $DOCKER_REGISTRY_HOST DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT +LOGGING_ES_HOST: $LOGGING_ES_HOST +LOGGING_ES_PORT: $LOGGING_ES_PORT EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

2 2

[PATCH v3 compass-ci] monitoring/filter.cr: query value support regularity
by Wu Zhende 04 Nov '20

04 Nov '20

[Why] Enhanced monitoring function. Regular expressions can be used in query's value. query can be {"job_id": "z9.*"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index bc6f901..88702b6 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,13 +56,20 @@ class Filter def match_query(query : Hash(String, JSON::Any), msg : Hash(String, JSON::Any)) query.each do |key, value| - value = value.as_a - if value.includes?(nil) - return false unless msg.has_key?(key) - else - return false unless value.includes?(msg[key]?) - end + return false unless msg.has_key?(key) + + values = value.as_a + next if values.includes?(nil) || values.includes?(msg[key]?) + + return false unless regular_match(values, msg[key]?.to_s) end return true end + + private def regular_match(rules, string) + rules.each do |rule| + return true if string =~ /#{rule}/ + end + return false + end end -- 2.23.0

1 0

[PATCH lkp-tests] job.rb: fix add second program to pp
by Wei Jihui 04 Nov '20

04 Nov '20

[Why] Now we add new field in job yaml like this: on_fail: sleep: 300 and we want add sleep to pp, but at pp() function, it only traverse job's keys--->on_fail, this can not find sleep. it should find sleep at on_fail's keys. [How] when traverse job to find program, in this case, it is sleep, then find it at on_fail's keys, not job's keys. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..3fdf8c79 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -576,10 +576,10 @@ class Job @job[comment_to_symbol('auto generated by add_pp()')] = nil @job["pp"] = Hash.new() init_program_options() - for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, p_args| - k = hash_key_re_string(@job, p_n, @job['pp'].keys) - if k && !(a)job[k].is_a?(Hash) - @job['pp'][k] = @job[k] + for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, _p_args| + k = hash_key_re_string(h, p_n, @job['pp'].keys) + if k && h[k] && !h[k].is_a?(Hash) + @job['pp'][k] = h[k] next end options_hash = @referenced_programs[p_n] -- 2.23.0

1 0

[PATCH lab-z9] hosts: fix taishan200-2280-2s48p-256g--a102 to taishan200-2280-2s48p-512g--a102
by Zhang Yu 04 Nov '20

04 Nov '20

[Why] The memory of taishan200-2280-2s48p-256g--a95 is 512g, so need to fix 256g to 512g. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename hosts/{taishan200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} (100%) diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-512g--a102 similarity index 100% rename from hosts/taishan200-2280-2s48p-256g--a102 rename to hosts/taishan200-2280-2s48p-512g--a102 -- 2.23.0

1 1

[PATCH v4 lkp-tests] tests/build-pkg: modify save location for *.cgz file
by Liu Shaofei 04 Nov '20

04 Nov '20

[why] the path of *.cgz file generated by build-pkg script cannot be obtained during test. So the path of cgz file need to be changed. [how] below is example for openeuler: latest.cgz -> f71820d6055373c9b9381615e28a28f307bb2df4.cgz f71820d6055373c9b9381615e28a28f307bb2df4.cgz v1.8.1.cgz -> 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz v1.5.1.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz df0584e46db9e7b412626deab10b923504745b2c.cgz Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..f9bfa377 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,15 +23,19 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount $LKP_SERVER:$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } @@ -75,12 +81,22 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 } +create_softlink() +{ + [ -n "$upstream_tag" ] || return 0 + [ -e "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" ] && return 0 + + ln -sf "$(basename $(realpath ${cgz_name}))" "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" || return 0 + echo "create soft link: $PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz -> $(basename $(realpath ${cgz_name}))" +} + check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

3 4

[PATCH v2 compass-ci] monitoring/filter.cr: query value support regularity
by Wu Zhende 04 Nov '20

04 Nov '20

[Why] Enhanced monitoring function. Regular expressions can be used in query's value. query can be {"job_id": "z9.*"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index bc6f901..d520e6e 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,13 +56,21 @@ class Filter def match_query(query : Hash(String, JSON::Any), msg : Hash(String, JSON::Any)) query.each do |key, value| + return false unless msg.has_key?(key) + value = value.as_a - if value.includes?(nil) - return false unless msg.has_key?(key) - else - return false unless value.includes?(msg[key]?) - end + next if value.includes?(nil) + next if value.includes?(msg[key]?) + + return false unless regular_match(value, msg[key]?.to_s) end return true end + + private def regular_match(rules, string) + rules.each do |rule| + return true if string =~ /#{rule}/ + end + return false + end end -- 2.23.0

2 1

[PATCH v1 compass-ci] improve multi-qemu code to support queue parameter
by Xiao Shenwei 04 Nov '20

04 Nov '20

Usage: multi-qemu [-n] [-c] [-q] -n, --name HOSTNAME_PREFIX specify used hostname_prefix -c, --count count how much VM do you need -q, --queues queues queues from which tasks are obtained -h, --help show this message example: ./multi-qemu -n vm-2p8g.xsw -c 2 -q vm-2p8g~xsw,vm-2p8g.aarch64 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-qemu | 98 +++++++++++++++++++++++++++++++++----------- 1 file changed, 74 insertions(+), 24 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 23e0451..03564f7 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -4,41 +4,91 @@ # frozen_string_literal: true require 'fileutils' +require 'optparse' +require 'time' -PWD = Dir.pwd +opt = {} +options = OptionParser.new do |opts| + opts.banner = 'Usage: multi-qemu [-n] [-c] [-q]' + + opts.separator '' + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'specify used hostname_prefix') do |name| + opt['hostname_prefix'] = name + end + + opts.on('-c count', '--count count', 'how much VM do you need') do |num| + opt['nr_vm'] = num + end + + opts.on('-q queues', '--queues queues', 'requested queues, use "," to separate more than 2 values') do |queues| + opt['queues'] = queues + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.size.zero? + puts options + exit +end + +options.parse!(ARGV) # Run multiple QEMU in parallel -HOSTNAME = ARGV[0] || "vm-2p8g--#{ENV['USER']}" -NR_VM = ARGV[1] || 1 +PWD = Dir.pwd +HOSTNAME = opt['hostname_prefix'] || "vm-2p8g.#{ENV['USER']}" +NR_VM = opt['nr_vm'] || 1 +QUEUES = opt['queues'] || "vm-2p8g.#{RUBY_PLATFORM.split('-')[0]}" +LOG_DIR = '/srv/cci/serial/logs' -def run(seqno) - loop do - start_time = Time.new - hostname = "#{HOSTNAME}-#{seqno}" - log_file = "/srv/cci/serial/logs/#{hostname}" +def main(hostname) + start_time = Time.new.strftime('%Y-%m-%d %H:%M:%S') + log_file = "#{LOG_DIR}/#{hostname}" + record_runtime(log_file, start_time) + run_vm(hostname) + duration = ((Time.new - Time.parse(start_time)) / 60).round(2) + record_runtime(log_file, duration, is_start: false) +end +def record_runtime(log_file, message, is_start: true) + if is_start File.open(log_file, 'w') do |f| # fluentd refresh time is 1s # let fluentd to monitor this file first sleep(2) - f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting QEMU" + f.puts "\n#{message} starting QEMU" end + return + end + File.open(log_file, 'a') do |f| + f.puts "\nTotal QEMU duration: #{message} minutes" + end +end - pwd_hostname = File.join(PWD, hostname) - FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) - FileUtils.cd(pwd_hostname) - system( - { 'hostname' => hostname }, - ENV['CCI_SRC'] + '/providers/qemu.sh' - ) - - duration = ((Time.new - start_time) / 60).round(2) - File.open(log_file, 'a') do |f| - f.puts "\nTotal QEMU duration: #{duration} minutes" - end +def run_vm(hostname) + pwd_hostname = File.join(PWD, hostname) + FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) + FileUtils.cd(pwd_hostname) + system( + { 'hostname' => hostname, 'queues' => QUEUES }, + ENV['CCI_SRC'] + '/providers/qemu.sh' + ) +end - # sleep 5s is for fluentd to collect it's log - sleep(5) +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.backtrace + # if an exception happend, request the next time after 30 seconds + sleep 25 + ensure + sleep 5 + end end end @@ -55,7 +105,7 @@ def multiqemu pids = [] NR_VM.to_i.times do |i| pid = Process.fork do - run i + loop_main("#{HOSTNAME}-#{i}") end pids << pid end -- 2.23.0

2 3

[PATCH v2 lkp-tests 1/3] (job2sh.rb) add on_fail to job.sh
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh export_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..6e00b088 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors','on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -357,6 +359,30 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + if !job + job = (@jobx || @job).clone + end + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

3 2

[PATCH lkp-tests] add testcase for sysbench-threads-git
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] old version package needs to debug by new testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 tests/sysbench-threads-git diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git new file mode 100755 index 00000000..712b3359 --- /dev/null +++ b/tests/sysbench-threads-git @@ -0,0 +1,17 @@ +#!/bin/sh +# - nr_threads +# - thread_yields +# - thread_locks + +. "$LKP_SRC/lib/reproduce-log.sh" + +: "${nr_threads:=2}" +: "${thread_yields:=100}" +: "${thread_locks:=2}" + +args="\ + --threads=$nr_threads\ + --thread-yields=$thread_yields\ + --thread-locks=$thread_locks\ +" +log_cmd sysbench --test=threads $args run -- 2.23.0

2 1

[PATCH lkp-tests] add PKGBUILD for sysbench-threads-git
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] Adapting sysbench-threads needs to old version to user. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..2779c294 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,21 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=0.4.8 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="http://github.com/dallasmarlow/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() { + cd "$srcdir/$pkgname" + make DESTDIR=$pkgdir install +} -- 2.23.0

1 0

[PATCH lkp-tests] add PKGBUILD for sysbech-threads-git
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] Adapting sysbench-threads needs to old version to user. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..bf47d5a1 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,21 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=0.4.8 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="https://github.com/akopytov/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() { + cd "$srcdir/$pkgname" + make DESTDIR=$pkgdir install +} -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/3] submit on_fail to the scheduler
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] When job2sh.rb defined sh_on_fail, it needs to be invoked by the scheduler. Finally, the task is in the waiting state when the task is failed. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..0bb41f2d 100755 --- a/sbin/submit +++ b/sbin/submit @@ -115,12 +115,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_fail sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

1 0