November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 05/10] container/mail-robot: send-apply-account-email.rb
by Luan Shengde 05 Nov '20

05 Nov '20

SendApplyAccountEmail send uuid email build email message send email send error email build error email message send error email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/send-apply-account-email.rb | 72 +++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100755 container/mail-robot/send-apply-account-email.rb diff --git a/container/mail-robot/send-apply-account-email.rb b/container/mail-robot/send-apply-account-email.rb new file mode 100755 index 0000000..f29ff1e --- /dev/null +++ b/container/mail-robot/send-apply-account-email.rb @@ -0,0 +1,72 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# build mail data and send mail +class SendApplyAccountEmail + def initialize(send_mail_info) + @send_mail_host = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + @send_mail_info = send_mail_info + end + + def send_uuid_email + email_msg = <<~EMAIL_MESSAGE + To: #{@send_mail_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{@send_mail_info['my_name']}, + + Thank you for joining us. + You can use the following info to submit jobs: + + 1) setup default config + run the following command to add the below setup to default config file + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/\\${USER}.yaml <<-EOF + my_email: #{@send_mail_info['my_email']} + my_name: #{@send_mail_info['my_name']} + my_uuid: #{@send_mail_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + find how to write job yaml + submit jobs/netperf.yaml + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_error_email + email_msg = <<~EMAIL_MESSAGE + To: #{@send_mail_info['my_email']} + Subject: apply account failed + + Dear #{@send_mail_info['my_name']} + + Your application for account failed with following errors + + #{@send_mail_info['error_message']} + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_mail(email_msg) + %x(curl -XPOST "#{@send_mail_host}:#{ENV['SEND_MAIL_PORT']}/send_mail_text" -d "#{email_msg}") + end +end -- 2.23.0

3 5

[PATCH v4 lkp-tests] job2sh.rb: define on_fail
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh export_top_envexport_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7e6a5d7c..370c7899 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -358,6 +360,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

1 0

[PATCH compass-ci] qcow2rootfs: modify the location of the vmlinuz soft link of rootfs
by Yu Chuan 05 Nov '20

05 Nov '20

[Why] Due to the changes in the layout of rootfs, so modify the location of the vmlinuz soft link when generating rootfs. [How] Before: ${ROOTFS_DIR} └── vmlinuz -> boot/vmlinuz-xxx After: ${ROOTFS_DIR}/boot/ └── vmlinuz -> vmlinuz-xxx In addition: ROOTFS_VMLINUZ=$(basename "$ROOTFS_VMLINUZ_FILE"). Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/qcow2rootfs/bin/common | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index 0d109e39c3fc..da39f523e2c7 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -174,8 +174,9 @@ create_links_vmlinuz_initrd() { bash -c " cd $EXTRACT_ROOT/rootfs-dir chmod a+r $ROOTFS_INITRD_LKP - ln -sf $ROOTFS_VMLINUZ_FILE vmlinuz ln -sf $ROOTFS_INITRD_LKP initrd.lkp + cd $EXTRACT_ROOT/rootfs-dir/boot + ln -sf $ROOTFS_VMLINUZ vmlinuz " ) if "${cmds[@]}" ; then -- 2.23.0

1 0

[PATCH lkp-tests] fio: fix error when test with write mode
by Wei Jihui 05 Nov '20

05 Nov '20

[Why] fio need test on disk, so mount a disk by: "mount -t xfs -o inode64 /dev/sda /fs/sda" when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 1 + 1 file changed, 1 insertion(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..3ea79fa2 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -95,6 +95,7 @@ fallocate=$fallocate io_size=$io_size file_service_type=$file_service_type random_distribution=$random_distribution +allow_mounted_write=1 group_reporting create_only=1 pre_read=$pre_read -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: change the way get hosts file
by Li Ping 05 Nov '20

05 Nov '20

[why] we can get hosts file from the repo lab-z9 or get hosts file from lkp-tests/hosts before: get hosts file from lab-z9, but the hosts file not exist in lkp-tests/hosts the job will puts the error message: hosts_file not exist ... but there is no need to puts such error message, if we have got the hosts file from lab-z9 --- lib/job.rb | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 995b38bf..254562f7 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,20 +334,14 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + return lab_hosts_file if lab_hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.file?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + puts("hosts_file not exist: #{hosts_file_name}, maybe need check testbox field") end def include_files -- 2.23.0

3 3

[PATCH v2 compass-ci 2/2] add template compare mode
by Lu Weitao 05 Nov '20

05 Nov '20

add template compare mode in sbin/compare(command-line tool) by: - load and parse compare.yaml - calls lib/es_query.rb, query jobs data set from ES - calls lib/matrix.rb, convert jobs data set to group matrices - calls lib/compare_matrixes.rb, use groups matrices to compare background: To support compare with user-defined template feature, the work-flow of user-defined template feature: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results example: compare -t/--template compare_template.yaml, the yaml like: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s output:{ "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "bs|test_size", "tables": { "4k|1G": { "fio.write_iops": { "average": { "openeuler 20.03": 308298.5524761111, "debian sid": 220289.07563 }, "standard_deviation": { "openeuler 20.03": null, "debian sid": null }, "change": { "debian sid vs openeuler 20.03": -28.5 } } }, "16k|1G": {...}, ... } } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++ sbin/compare | 10 +++++++++- 2 files changed, 60 insertions(+), 1 deletion(-) diff --git a/lib/compare.rb b/lib/compare.rb index 2837edc..9edf049 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -6,6 +6,7 @@ require_relative './es_query.rb' require_relative './matrix2.rb' require_relative './compare_matrixes.rb' require_relative './constants.rb' +require 'yaml' # ------------------------------------------------------------------------------------------- # compare_matrices_list @@ -66,3 +67,53 @@ def create_groups_matrices_list(conditions, dims) query_results = es.multi_field_query(conditions) combine_group_query_data(query_results, dims) end + +# ------------------------------------------------------------------------------------------- +# compare with user-defined compare_template.yaml +# the work-flow are: +# load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list +# ---> create groups_matrices ---> compare_values by each metrics ---> format/show results +# + +def compare_by_template(template) + template_params = load_template(template) + groups_matrices = create_groups_matrices(template_params) + compare_values = compare_metrics_values(groups_matrices) + chart_results = format_for_chart(compare_values, template_params) + show_chart_compare_result(chart_results) +end + +def load_template(template) + unless File.file?(template) + warn 'template does not exist' + exit + end + YAML.load_file(template) +end + +# input: template_params: Hash +# eg: +# { +# "compare_metrics"=>["fio.write_iops", "fio.read_iops"], +# "filter"=>[ +# {"suite"=>["fio-bisic"]}, +# {"os_arch"=>["aarch_64"]} +# ], +# "compare_dimensions"=>[ +# {"os"=>"openeuler", "os_version"=>20.03}, +# {"os"=>"centos", "os_version"=>7.6} +# ], +# "x_params"=>["block_size", "package_size"], +# "title"=>"Hackbench Performance Testing", +# "unit"=>"KB/s" +# } +def create_groups_matrices(template_params) + es = ESQuery.new + query_results = es.multi_field_query(template_params['filter']) + combine_group_jobs_list( + query_results, + template_params['x_params'], + template_params['compare_dimensions'], + template_params['compare_metrics'] + ) +end diff --git a/sbin/compare b/sbin/compare index cf3980c..b13682e 100755 --- a/sbin/compare +++ b/sbin/compare @@ -6,6 +6,7 @@ # Usage: # compare "conditions_1" "conditions_2" ... -c "common_conditions" # compare "conditions" -d "dimensions" +# compare -t template.yaml # Eg: # compare "id=6000,6001" "id=7000,7001" # compare "commit=a12d232e" "commit=b3bacc31" @@ -21,6 +22,7 @@ is_group = false dimensions = nil colorful = nil options = {} +template = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: compare "conditions" ... [option]' @@ -45,6 +47,10 @@ opt_parser = OptionParser.new do |opts| colorful = color end + opts.on('-t', '--template template', 'compare with user-defined template') do |t| + template = t + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -60,7 +66,9 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful -if is_group +if template + compare_by_template(template) +elsif is_group compare_group(argv, dimensions, options) else compare_matrices_list(argv, common_conditions, options) -- 2.23.0

3 4

[PATCH lkp-tests] fio: fix error when test with write mode
by Wei Jihui 05 Nov '20

05 Nov '20

[Why] fio need test on disk, so mount a disk by: "mount -t xfs -o inode64 /dev/sda /fs/sda" when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. [How] set allow_mounted_write=1 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 1 + 1 file changed, 1 insertion(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..3ea79fa2 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -95,6 +95,7 @@ fallocate=$fallocate io_size=$io_size file_service_type=$file_service_type random_distribution=$random_distribution +allow_mounted_write=1 group_reporting create_only=1 pre_read=$pre_read -- 2.23.0

2 1

[PATCH compass-ci 4/6] container/webhook: start
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 container/webhook/start diff --git a/container/webhook/start b/container/webhook/start new file mode 100755 index 0000000..50cbf20 --- /dev/null +++ b/container/webhook/start @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "webhook" + +cmd = %W[ + docker run + --restart=always + --name webhook + -d + -p 11301:11301 + node:webhook +] + +cmd += ['sh', '-c', 'node webhook.js'] +system(*cmd) -- 2.23.0

2 2

[PATCH v2 lkp-tests] libmicro: add PKGBUILD for libmicro
by Bai Jing 05 Nov '20

05 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/libmicro/PKGBUILD | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 pkg/libmicro/PKGBUILD diff --git a/pkg/libmicro/PKGBUILD b/pkg/libmicro/PKGBUILD new file mode 100644 index 00000000..61cd32f4 --- /dev/null +++ b/pkg/libmicro/PKGBUILD @@ -0,0 +1,19 @@ +pkgname=libMicro +pkgver=0.4 +pkgrel=0 +pkgdesc="A portable set of microbenchmarks focused on the syscall and system library interfaces." +arch=('i386' 'x86_64') +url="https://github.com/rzezeski/libMicro" +license=('CDDL') +source=("https://codeload.github.com/redhat-performance/libMicro/zip/0.4.0-rh") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname-$pkgver.$pkgrel-rh" + make +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/libMicro" + cp -r "$srcdir/$pkgname-$pkgver.$pkgrel-rh/"* "$pkgdir/lkp/benchmarks/libMicro" +} -- 2.23.0

4 7

[PATCH lkp-tests] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 05 Nov '20

05 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } PKGBUILD-1.1 contains only the changed part. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..dc8471b9 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -888,7 +888,7 @@ update_pkgver() { exit 1 fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" - source_safe "$BUILDFILE" + source_safe "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -1604,10 +1604,14 @@ cd_safe() { source_safe() { shopt -u extglob - if ! source "$@"; then - error "$(gettext "Failed to source %s")" "$1" - exit 1 - fi + local file + for file in "$@" + do + if ! source "$file"; then + error "$(gettext "Failed to source %s")" "$1" + exit 1 + fi + done shopt -s extglob } @@ -1626,7 +1630,6 @@ merge_arch_attrs() { source_buildfile() { source_safe "$@" - if (( !SOURCEONLY )); then merge_arch_attrs fi @@ -3661,6 +3664,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +PKGBUILD_TAG=$PKGBUILD_TAG if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then @@ -3691,7 +3695,7 @@ else if [[ ${BUILDFILE:0:1} != "/" ]]; then BUILDFILE="$startdir/$BUILDFILE" fi - source_buildfile "$BUILDFILE" + source_buildfile "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

4 4

[PATCH lkp-tests] add sysbench-threads-git.yaml
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] job.yaml transfer required parameters to testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/sysbench-threads-git.yaml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 jobs/sysbench-threads-git.yaml diff --git a/jobs/sysbench-threads-git.yaml b/jobs/sysbench-threads-git.yaml new file mode 100644 index 00000000..6b0f2c00 --- /dev/null +++ b/jobs/sysbench-threads-git.yaml @@ -0,0 +1,21 @@ +suite: sysbench-threads-git +category: benchmark + +nr_threads: +- 2 +- 4 +- 8 +- 16 +- 32 +- 64 +- 128 +- 256 +- 512 +- 1024 +- 2048 +- 4096 +- 8192 + +sysbench-threads-git: + thread_yields: 100 + thread_locks: 2 -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 05 Nov '20

05 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..6d24d8d9 --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,36 @@ +#!/bin/sh +# - runtime +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is an old version, It is both a benchmark and a stress test for the Linux kernel scheduler. +## It's main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes) which communicate via either sockets or pipes and time how long +## it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' + +[ -n "$runtime" ] || runtime=600 +[ -n "$iterations" ] || iterations=10 +[ -n "$loops" ] || loops=100 +[ -n "$nr_threads" ] || nr_threads=1 + +run_hackbench() +{ + # args: groups number, mode [process/thread], times + local hackbench_path + if [ -x /lkp/benchmarks/hackbench-git/hackbench/hackbench ]; then + hackbench_path=/lkp/benchmarks/hackbench-git/hackbench/hackbench + else + echo "The hackbench executable file cannot be found." && exit 1 + fi + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

2 3

[PATCH lkp-tests] add sysbench-threads-git.yaml
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] job.yaml transfer required parameters to testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/sysbench-threads-git.yaml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 jobs/sysbench-threads-git.yaml diff --git a/jobs/sysbench-threads-git.yaml b/jobs/sysbench-threads-git.yaml new file mode 100644 index 00000000..fe7552a4 --- /dev/null +++ b/jobs/sysbench-threads-git.yaml @@ -0,0 +1,21 @@ +suite: sysbench-threads-git +category: benchmark + +nr_threads: +- 2 +- 4 +- 8 +- 16 +- 32 +- 64 +- 128 +- 256 +- 512 +- 1024 +- 2048 +- 4096 +- 8192 + +sysbench-threads: + thread_yields: 100 + thread_locks: 2 -- 2.23.0

2 1

[PATCH v2 compass-ci 03/10] container/mail-robot: parse-apply-account-email.rb
by Luan Shengde 05 Nov '20

05 Nov '20

parse email for apply account check out commit url check commit url exists check base url in upstream-repos check commit available hub: gitee.com clone the repo and check the commit exists hub: non-gitee.com check the feedback of curl command to check the commit url available check out pub key extract the attachment of the pub_key Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/parse-apply-account-email.rb | 135 ++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100755 container/mail-robot/parse-apply-account-email.rb diff --git a/container/mail-robot/parse-apply-account-email.rb b/container/mail-robot/parse-apply-account-email.rb new file mode 100755 index 0000000..1325c8f --- /dev/null +++ b/container/mail-robot/parse-apply-account-email.rb @@ -0,0 +1,135 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check whether there is a commit url and ssh pub_key in the email +# apply account on junper server +# the entry point is parse_commit_url_pub_key +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + @my_email = mail_content.from[0] + @my_name = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + end + + def parse_commit_url_pub_key + commit_url = check_out_commit_url + pub_key = check_out_pub_key + + return commit_url, pub_key + end + + def send_error_email(error_message) + send_error_email_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'error_message' => error_message + } + send_err_mail = SendApplyAccountEmail.new(send_error_email_info) + send_err_mail.send_error_email + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = 'No matched commit url found' + send_error_email(error_message) + + raise error_message + end + + def check_out_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + check_base_url_in_upstream_repos('/c/upstream-repos', base_url) + check_commit_available(url, base_url) + + return url + end + + def check_base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = 'The url is not in upstream-repo list' + send_error_email(error_message) + + raise error_message + end + + def check_commit_available(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + check_gitee_commit(url, base_url) + else + check_non_gitee_commit(url) + end + end + + def check_non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_email + + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_gitee_commit(url, base_url) + repo_dir = url.split('/')[-3] + repo_url = [base_url, 'git'].join('.') + commit_id = url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_email + + FileUtils.rm_rf repo_dir + + check_gitee_commit_exist(email_index) + end + + def check_gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_out_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + check_pub_key(pub_key) + + return pub_key + end + + def check_pub_key(pub_key) + return unless pub_key.nil? + + error_message = 'No pub_key found' + send_error_email(error_message) + + raise error_message + end +end -- 2.23.0

4 7

[PATCH v2 compass-ci 02/10] container/mail-robot: apply-account.rb
by Luan Shengde 05 Nov '20

05 Nov '20

check to assign account invoke ParseApplyAccountEmail to parse commit url and pubkey generate uuid and invoke AssignJumperAccount to apply account store account info to es send uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/apply-account.rb | 65 +++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100755 container/mail-robot/apply-account.rb diff --git a/container/mail-robot/apply-account.rb b/container/mail-robot/apply-account.rb new file mode 100755 index 0000000..9a8d706 --- /dev/null +++ b/container/mail-robot/apply-account.rb @@ -0,0 +1,65 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '../../lib/es_client' + +# assign uuid/account +class AssignAccount + def initialize(mail_content) + @mail_content = mail_content + @my_email = mail_content.from[0] + @my_name = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + @send_mail_host = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + end + + def check_to_send_account + my_commit_url, my_ssh_pubkey = check_out_commit_url_pub_key + my_uuid, acct_infos = check_to_apply_account(my_ssh_pubkey) + + my_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_commit_url' => my_commit_url, + 'my_login_name' => acct_infos['my_login_name'], + 'my_uuid' => my_uuid + } + store_account_info(my_info) + my_info['acct_infos'] = acct_infos + send_my_uuid(my_info) + end + + def check_out_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + my_commit_url, my_ssh_pubkey = parse_apply_account_email.parse_commit_url_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def check_to_apply_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + apply_account_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_uuid' => my_uuid, + 'my_ssh_pubkey' => my_ssh_pubkey + } + apply_account = AssignJumperAccount.new(apply_account_info) + acct_infos = apply_account.apply_jumper_account + + return my_uuid, acct_infos + end + + def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) + end + + def send_my_uuid(my_info) + send_uuid = SendApplyAccountEmail.new(my_info) + send_uuid.send_uuid_email + end +end -- 2.23.0

5 8

[PATCH v3 lkp-tests] add PKGBUILD for sysbench-threads-git
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] Adapting sysbench-threads needs old version to user Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..41ed9f46 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=1 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="http://github.com/dallasmarlow/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() +{ + cd "$srcdir/sysbench" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() +{ + cd "$srcdir/sysbench" + make DESTDIR=$pkgdir install +} -- 2.23.0

5 4

[PATCH v4 lkp-tests] add PKGBUILD for sysbench-threads-git
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] Adapting sysbench-threads needs old version to user Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..41ed9f46 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=1 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="http://github.com/dallasmarlow/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() +{ + cd "$srcdir/sysbench" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() +{ + cd "$srcdir/sysbench" + make DESTDIR=$pkgdir install +} -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 05 Nov '20

05 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..799d3c4d --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,36 @@ +#!/bin/sh +# - runtime +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is an old version, It is both a benchmark and a stress test for the Linux kernel scheduler. +## It's main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes) which communicate via either sockets or pipes and time how long +## it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' + +[ -n "$runtime" ] || runtime=600 +[ -n "$iterations" ] || iterations=10 + +run_hackbench() +{ + # args: groups number, mode [process/thread], times + local hackbench_path + if [ -f /lkp/benchmarks/hackbench-git/hackbench/hackbench ]; then + hackbench_path=/lkp/benchmarks/hackbench-git/hackbench/hackbench + elif [ -f /usr/local/bin/hackbench ]; then + hackbench_path=/usr/local/bin/hackbench + else + echo "The hackbench executable file cannot be found." + fi + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

3 3

[PATCH v2 compass-ci] sbin/es-repo-mapping.sh: create script that generates "repo" mapping
by Zhang Yuhang 05 Nov '20

05 Nov '20

[why] We need a script to create "repo" mapping in es. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-repo-mapping.sh | 47 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 sbin/es-repo-mapping.sh diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh new file mode 100755 index 0000000..3a3930e --- /dev/null +++ b/sbin/es-repo-mapping.sh @@ -0,0 +1,47 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Check if "repo" index already exists. +status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) +[ $status_code -eq 200 ] && echo '"repo" index already exists. Exit.' && exit + +# Create "repo" index. +echo 'Start to create "repo" index.' +curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' +{ + "mappings": { + "_doc": { + "properties": { + "git_repo": { + "type": "keyword" + }, + "pkgbuild_repo": { + "type": "keyword" + }, + "url": { + "type": "keyword" + }, + "fetch_time": { + "type": "keyword" + }, + "new_refs_time": { + "type": "keyword" + }, + "offset_fetch": { + "type": "long" + }, + "offset_new_refs": { + "type": "long" + }, + "priority": { + "type": "long" + }, + "queued": { + "type": "boolean" + } + } + } + } +}' +[ $? -ne 0 ] && echo 'Failed to create "repo" index.' -- 2.23.0

2 2

[PATCH v5 lkp-tests 2/2] tests/iozone: modify the way of parsing parameter
by Lu Kaiyi 05 Nov '20

05 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] modify the way of parsing parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 66 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..b6a1e206 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,52 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + # reset IFS to split $test by "," and then restore default + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write") args+=" -i 0" + ;; + "read") args+=" -i 1" + ;; + "rand_rw") args+=" -i 2" + ;; + "backwards") args+=" -i 3" + ;; + "record") args+=" -i 4" + ;; + "stride") args+=" -i 5" + ;; + "fwrite") args+=" -i 6" + ;; + "fread") args+=" -i 7" + ;; + "rand_mix") args+=" -i 8" + ;; + "pwrite") args+=" -i 9" + ;; + "pread") args+=" -i 10" + ;; + "pwritev") args+=" -i 11" + ;; + "preadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

3 3

[PATCH compass-ci 6/6] container/webhook: add scripts for webhook
by Li Yuanchao 05 Nov '20

05 Nov '20

webhook.js will listen to webhook message from 'gitee.com' or 'github.com' if catch 'push' message, call push_hook.rb to send message to git-mirror through message queue Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/root/push_hook.rb | 15 +++++++++++++++ container/webhook/root/webhook.js | 22 ++++++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/webhook/root/push_hook.rb create mode 100755 container/webhook/root/webhook.js diff --git a/container/webhook/root/push_hook.rb b/container/webhook/root/push_hook.rb new file mode 100755 index 0000000..e467ccd --- /dev/null +++ b/container/webhook/root/push_hook.rb @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'bunny' + +connection = Bunny.new('amqp://172.17.0.1:5672') +connection.start +channel = connection.create_channel + +queue = channel.queue('web_hook') +message = ARGV[0] +queue.publish(message) +connection.close diff --git a/container/webhook/root/webhook.js b/container/webhook/root/webhook.js new file mode 100755 index 0000000..a98576c --- /dev/null +++ b/container/webhook/root/webhook.js @@ -0,0 +1,22 @@ +const http = require('http') +const spawn = require('child_process').spawn +const createHandler = require('git-webhook-handler') +// if users want to use webhook, they need to config webhook on their code warehouse('gitee.com' or 'github.com'), +// and use url: http://183.134.196.212:11301/webhook, secret: webhook@git. +const handler = createHandler({ path: '/webhook', secret: 'webhook@git' }) + +handler.on('error', function(err){ + console.error('Error:', err.message) +}) + +handler.on('push', function(event){ + console.log(event.payload.repository.url) + spawn('ruby', ['push_hook.rb', event.payload.repository.url]) +}) + +http.createServer(function(req, res){ + handler(req, res, function(err){ + res.statusCode = 404 + res.end('no such location') + }) +}).listen(11301) -- 2.23.0

2 1

[PATCH compass-ci 3/6] container/webhook: build
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/build | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/webhook/build diff --git a/container/webhook/build b/container/webhook/build new file mode 100755 index 0000000..bb5c0a7 --- /dev/null +++ b/container/webhook/build @@ -0,0 +1,6 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t node:webhook . + -- 2.23.0

2 1

[PATCH compass-ci 2/6] container/webhook: Dockerfile
by Li Yuanchao 05 Nov '20

05 Nov '20

add container webhook Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/Dockerfile | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/webhook/Dockerfile diff --git a/container/webhook/Dockerfile b/container/webhook/Dockerfile new file mode 100644 index 0000000..a9f0356 --- /dev/null +++ b/container/webhook/Dockerfile @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM node:10.21.0-alpine + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +ADD root / + +RUN apk update && \ + apk upgrade && \ + apk add --no-cache git && \ + apk add npm ruby-dev + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc + +RUN npm install git-webhook-handler &&\ + gem install bunny + -- 2.23.0

3 2

[PATCH lkp-tests 2/2] nginx: add nginx PKGBUILD
by Wei Jihui 05 Nov '20

05 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/nginx/PKGBUILD | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 pkg/nginx/PKGBUILD diff --git a/pkg/nginx/PKGBUILD b/pkg/nginx/PKGBUILD new file mode 100644 index 00000000..b437c4b1 --- /dev/null +++ b/pkg/nginx/PKGBUILD @@ -0,0 +1,24 @@ +pkgname=nginx +pkgver=1.14 +pkgrel=2 +arch=('i386' 'x86_64' 'aarch64') +url="https://nginx.org" +license=('BSD') +depends=(pcre zlib openssl gcc) +source=("https://nginx.org/download/nginx-${pkgver}.#{pkgrel}.tar.gz") +sha256sums=('SKIP') + +build() +{ + benchmark_path="${pkgdir}/lkp/benchmarks/${pkgname}" + mkdir -p "${benchmark_path}" + cd "${srcdir}/nginx-${pkgver}.#{pkgrel}" + ./configure --with-http_ssl_module --prefix="${benchmark_path}" + make +} + +package() +{ + cd "$srcdir/$pkgname-$pkgver" + make install +} -- 2.23.0

3 2

[PATCH lkp-tests] fio-setup-basic: keep format when change create_task
by Xu Xijian 05 Nov '20

05 Nov '20

[why] When add time_based to create_task, add '\' first, keep this format with others in context. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- setup/fio-setup-basic | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..5b0d0fce 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -117,7 +117,7 @@ ioscheduler=$ioscheduler" fi if parse_bool -q "$time_based"; then - create_task=" + create_task="\ $create_task time_based " -- 2.23.0

1 0

[PATCH compass-ci 5/6] container/webhook: start-depends
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start-depends | 2 ++ 1 file changed, 2 insertions(+) create mode 100755 container/webhook/start-depends diff --git a/container/webhook/start-depends b/container/webhook/start-depends new file mode 100755 index 0000000..38fc696 --- /dev/null +++ b/container/webhook/start-depends @@ -0,0 +1,2 @@ +rabbitmq +es -- 2.23.0

1 0

[PATCH compass-ci] container/master-fluentd: fix buffer warn
by Wu Zhende 05 Nov '20

05 Nov '20

[warn] [warn]: suppressed same stacktrace failed to emit fluentd's log event tag="fluent.warn" event={"message"=>"failed to write data into buffer by buffer overflow"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" [warn]: failed to write data into buffer by buffer overflow [How] Set num_threads=10 Flush the cache with 10 thread pools， prevents logs from piling up Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index a57d4cd..62c0319 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -44,6 +44,7 @@ port "#{ENV['LOGGING_ES_PORT']}" suppress_type_name true flush_interval 1s + num_threads 10 index_name ${tag} ssl_verify false log_es_400_reason true -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] compare values by each metrics
by Lu Weitao 05 Nov '20

05 Nov '20

compare values by each metrics based on groups matrices, and format compare result as echart data_set background: To support compare with user-defined template feature, the work-flow of user-defined template feature: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: compare_values by each metrics ---> format/show results Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 107 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 078028a..ebe1cd2 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -399,8 +399,115 @@ def compare_group_matrices(group_matrices, options) result_str end +# input: groups_matrices +# { +# group_key_1 => { +# dimension_1 => matrix_1, (openeuler 20.03) +# dimension_2 => matrix_2, (openeuler 20.09) +# dimension_3 => matrix_3, (centos 7.6) +# }, +# group_key_2 => {...} +# } +# +# output: compare_metrics_values +# { +# group_key_1 => { +# metric_1 => { +# 'average' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'standard_deviation' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'change' => { +# 'dimension_2 vs dimension_1' => xxx +# 'dimension_3 vs dimension_1' => xxx +# 'dimension_3 vs dimension_2' => xxx +# } +# }, +# metric_2 => {...} +# } +# } +def compare_metrics_values(groups_matrices) + metrics_compare_values = {} + groups_matrices.each do |group_key, dimensions| + metrics_compare_values[group_key] = get_metric_values(dimensions) + end + metrics_compare_values +end + +def get_metric_values(dimensions) + metrics_values = {} + dimensions.each do |dim, matrix| + matrix.each do |metric, values| + assign_metric_values(metrics_values, dim, metric, values) + end + end + assign_metric_change(metrics_values) + metrics_values +end + +def assign_metric_values(metrics_values, dim, metric, values) + metrics_values[metric] ||= {} + metrics_values[metric]['average'] ||= {} + metrics_values[metric]['standard_deviation'] ||= {} + metric_value = get_values(values, true) + metrics_values[metric]['average'][dim] = metric_value[:average] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] +end + +def assign_metric_change(metrics_values) + metrics_values.each do |metric, values| + metrics_values[metric]['change'] = {} + next if values['average'].size < 2 + + dimension_list = values['average'].keys + change_titles = get_change_titles(dimension_list) + change_titles.each do |base_dimension, challenge_dimension| + change = get_compare_value(values['average'][base_dimension], values['average'][challenge_dimension], true) + values['change'] = { "#{challenge_dimension} vs #{base_dimension}" => change } + end + end +end + +# input: dimension_list +# eg: ['openeuler 20.03', 'debian sid', 'centos 7.6'] +# output: Array(base_dimension: String, challenge_dimension: String) +# [ +# ['openeuler 20.03', 'debian sid'], +# ['openeuler 20.03', 'centos 7.6'], +# ['debian sid', 'centos 7.6'] +# ] +def get_change_titles(dimension_list) + dims = [] + dimension_list_size = dimension_list.size + (1..dimension_list_size - 1).each do |i| + (i..dimension_list_size - 1).each do |j| + dims << [dimension_list[i - 1], dimension_list[j]] + end + end + dims +end + +def show_chart_compare_result(chart_result) + print JSON.pretty_generate(chart_result) +end + # Format Fields +def format_for_chart(metrics_compare_values, template_params) + chart_result = {} + chart_result['title'] = template_params['title'] || nil + chart_result['unit'] = template_params['unit'] || nil + chart_result['x_name'] = template_params['x_params'].join('|') || nil + chart_result['tables'] = metrics_compare_values + chart_result +end + def format_fails_runs(fails, runs) fails_width = (SUB_LONG_COLUMN_WIDTH * FAILS_PROPORTION).to_i runs_width = SUB_LONG_COLUMN_WIDTH - fails_width - 1 -- 2.23.0

2 1

[PATCH compass-ci] sbin/es-repo-mapping.sh: create script that generates "repo" mapping
by Zhang Yuhang 05 Nov '20

05 Nov '20

[why] We need a script to create "repo" mapping in es. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-repo-mapping.sh | 78 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 sbin/es-repo-mapping.sh diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh new file mode 100755 index 0000000..ce0ad74 --- /dev/null +++ b/sbin/es-repo-mapping.sh @@ -0,0 +1,78 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Determine whether curl is installed. If not, install curl. +if ! [ -x "$(command -v curl)" ] +then + echo "curl not exists, try to install." + if [ -x "$(command -v apk)" ] + then + apk add install -y curl + elif [ -x "$(command -v yum)" ] + then + yum install -y curl + elif [ -x "$(command -v apt-get)" ] + then + apt-get install -y curl + fi +else + echo "curl has install." +fi + +# Determine whether curl is installed successfully +if [ $? -ne 0 ] +then + echo "curl install failed, exit." + exit +fi + +# Determine whether repo index has created +status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) + +if [ $status_code -eq 200 ] +then + echo "repo index has create, exit." +else + echo "repo index not exists, begin create index." + curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' + { + "mappings": { + "_doc": { + "properties": { + "git_repo": { + "type": "keyword" + }, + "pkgbuild_repo": { + "type": "keyword" + }, + "url": { + "type": "keyword" + }, + "fetch_time": { + "type": "keyword" + }, + "new_refs_time": { + "type": "keyword" + }, + "offset_fetch": { + "type": "long" + }, + "offset_new_refs": { + "type": "long" + }, + "priority": { + "type": "long" + }, + "queued": { + "type": "boolean" + } + } + } + } + }' + if [ $? -ne 0 ] + then + echo "create repo index failed." + fi +fi -- 2.23.0

5 8

[PATCH v5 lkp-tests 1/2] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 05 Nov '20

05 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..026b97e2 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write,read,rand_rw block_size: - 64k -- 2.23.0

2 1

[PATCH compass-ci] container: delete invalid start depends fluentd
by Liu Yinsi 05 Nov '20

05 Nov '20

fluentd renamed master-fluentd, but scheduler and extract-stats have no dependency with master-fluentd now, so remove it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/extract-stats/start-depends | 1 - container/scheduler/start-depends | 1 - 2 files changed, 2 deletions(-) diff --git a/container/extract-stats/start-depends b/container/extract-stats/start-depends index 4a016e6..efbe5a2 100755 --- a/container/extract-stats/start-depends +++ b/container/extract-stats/start-depends @@ -1,2 +1 @@ taskqueue -fluentd diff --git a/container/scheduler/start-depends b/container/scheduler/start-depends index 3e28dc1..4a17b46 100755 --- a/container/scheduler/start-depends +++ b/container/scheduler/start-depends @@ -1,4 +1,3 @@ redis -fluentd es taskqueue -- 2.23.0

2 2

[PATCH lkp-tests 1/2] fio: fix error when test with write mode
by Wei Jihui 05 Nov '20

05 Nov '20

[Why] when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. [How] add allow_mounted_write when write mode Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..f90963aa 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -116,6 +116,12 @@ $create_task ioscheduler=$ioscheduler" fi +if [[ "$rw" =~ "write" ]]; then + create_task="\ +$create_task +allow_mounted_write=1" +fi + if parse_bool -q "$time_based"; then create_task=" $create_task -- 2.23.0

2 1

[PATCH compass-ci] jobfile_operate.cr: set lkp path as a constant var
by Xu Xijian 05 Nov '20

05 Nov '20

[why] "/c/lkp-tests" is twice used here, so set it as a constant var LKP_PATH. [how] ... Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/jobfile_operate.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index d3c26c5..68818ed 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -8,8 +8,9 @@ require "yaml" # require from '/c/lkp-tests/lib/' require "shellwords" -if ENV["LKP_SRC"] != "/c/lkp-tests" - raise "ENV LKP_SRC mismatch: #{ENV["LKP_SRC"]} '/c/lkp-tests'" +LKP_PATH = "/c/lkp-tests" +if ENV["LKP_SRC"] != LKP_PATH + raise "ENV LKP_SRC mismatch: #{ENV["LKP_SRC"]} #{LKP_PATH}" end module Jobfile::Operate -- 2.23.0

1 0

[PATCH v2 compass-ci] improve multi-qemu code to support queue parameter
by Xiao Shenwei 05 Nov '20

05 Nov '20

Usage: multi-qemu [-n] [-c] [-q] -n, --name HOSTNAME_PREFIX specify used hostname_prefix -c, --count count how much VM do you need -q, --queues queues requested queues, use "," to separate more than 2 values -h, --help show this message example: ./multi-qemu -n vm-2p8g.xsw -c 2 -q vm-2p8g~xsw,vm-2p8g.aarch64 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-qemu | 99 +++++++++++++++++++++++++++++++++----------- 1 file changed, 74 insertions(+), 25 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 23e0451..bddad08 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -4,46 +4,95 @@ # frozen_string_literal: true require 'fileutils' +require 'optparse' +require 'time' -PWD = Dir.pwd +opt = {} +options = OptionParser.new do |opts| + opts.banner = 'Usage: multi-qemu [-n] [-c] [-q]' + + opts.separator '' + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'specify used hostname_prefix') do |name| + opt['hostname_prefix'] = name + end + + opts.on('-c count', '--count count', 'how much VM do you need') do |num| + opt['nr_vm'] = num + end + + opts.on('-q queues', '--queues queues', 'requested queues, use "," to separate more than 2 values') do |queues| + opt['queues'] = queues + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.size.zero? + puts options + exit +end + +options.parse!(ARGV) # Run multiple QEMU in parallel -HOSTNAME = ARGV[0] || "vm-2p8g--#{ENV['USER']}" -NR_VM = ARGV[1] || 1 +PWD = Dir.pwd +HOSTNAME = opt['hostname_prefix'] || "vm-2p8g.#{ENV['USER']}" +NR_VM = opt['nr_vm'] || 1 +QUEUES = opt['queues'] || "vm-2p8g.#{RUBY_PLATFORM.split('-')[0]}" +LOG_DIR = '/srv/cci/serial/logs' -def run(seqno) - loop do - start_time = Time.new - hostname = "#{HOSTNAME}-#{seqno}" - log_file = "/srv/cci/serial/logs/#{hostname}" +def main(hostname) + start_time = Time.new.strftime('%Y-%m-%d %H:%M:%S') + log_file = "#{LOG_DIR}/#{hostname}" + record_runtime(log_file, start_time) + run_vm(hostname) + duration = ((Time.new - Time.parse(start_time)) / 60).round(2) + record_runtime(log_file, duration, is_start: false) +end +def record_runtime(log_file, message, is_start: true) + if is_start File.open(log_file, 'w') do |f| # fluentd refresh time is 1s # let fluentd to monitor this file first sleep(2) - f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting QEMU" + f.puts "\n#{message} starting QEMU" end + return + end + File.open(log_file, 'a') do |f| + f.puts "\nTotal QEMU duration: #{message} minutes" + end +end - pwd_hostname = File.join(PWD, hostname) - FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) - FileUtils.cd(pwd_hostname) - system( - { 'hostname' => hostname }, - ENV['CCI_SRC'] + '/providers/qemu.sh' - ) - - duration = ((Time.new - start_time) / 60).round(2) - File.open(log_file, 'a') do |f| - f.puts "\nTotal QEMU duration: #{duration} minutes" - end +def run_vm(hostname) + pwd_hostname = File.join(PWD, hostname) + FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) + FileUtils.cd(pwd_hostname) + system( + { 'hostname' => hostname, 'queues' => QUEUES }, + ENV['CCI_SRC'] + '/providers/qemu.sh' + ) +end - # sleep 5s is for fluentd to collect it's log - sleep(5) +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.backtrace + # if an exception happend, request the next time after 30 seconds + sleep 25 + ensure + sleep 5 + end end end def save_pid(arr) - FileUtils.rm('pid') if File.exist?('pid') f = File.new('pid', 'a') arr.each do |i| f.puts(i) @@ -55,7 +104,7 @@ def multiqemu pids = [] NR_VM.to_i.times do |i| pid = Process.fork do - run i + loop_main("#{HOSTNAME}-#{i}") end pids << pid end -- 2.23.0

3 8

[PATCH v2 lkp-tests] add PKGBUILD for sysbench-threads-git
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] Adapting sysbench-threads needs to old version to user Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..897799ff --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=0.4.8 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="http://github.com/dallasmarlow/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() +{ + cd "$srcdir/$pkgname" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() +{ + cd "$srcdir/$pkgname" + make DESTDIR=$pkgdir install +} -- 2.23.0

2 2

[PATCH v2 lkp-tests] add testcase for sysbench-threads-git
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] old version package needs to debug by new testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 tests/sysbench-threads-git diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git new file mode 100755 index 00000000..8b9e1c99 --- /dev/null +++ b/tests/sysbench-threads-git @@ -0,0 +1,17 @@ +#!/bin/sh +# - nr_threads +# - thread_yields +# - thread_locks + +. "$LKP_SRC/lib/reproduce-log.sh" + +: "${nr_threads:=2}" +: "${thread_yields:=100}" +: "${thread_locks:=2}" + +args="\ + --threads=$nr_threads\ + --thread-yields=$thread_yields\ + --thread-locks=$thread_locks +" +log_cmd sysbench --test=threads $args run -- 2.23.0

2 1

[PATCH v6 lkp-tests] tests/build-pkg: change location for *.cgz file
by Liu Shaofei 05 Nov '20

05 Nov '20

[why] the path of *.cgz file generated by build-pkg script cannot be obtained during test. So the path of cgz file need to be changed. [how] below is example for openeuler: - path: /srv/initrd/build-pkg/initramfs/openeuler/aarch64/20.03/nasm-git - filename: latest.cgz -> f71820d6055373c9b9381615e28a28f307bb2df4.cgz f71820d6055373c9b9381615e28a28f307bb2df4.cgz v1.8.1.cgz -> 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz v1.5.1.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz df0584e46db9e7b412626deab10b923504745b2c.cgz Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..7c494a15 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,15 +23,19 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount $LKP_SERVER:$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } @@ -75,12 +81,22 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 +} + +create_softlink() +{ + [ -n "$upstream_tag" ] || return 0 + [ -e "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" ] && return 0 + + ln -sf "$(basename $(realpath ${cgz_name}))" "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" || return 0 + echo "create soft link: $PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz -> $(basename $(realpath ${cgz_name}))" } check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

3 3

[PATCH lkp-tests 3/3] lib/job.rb: change the way get hostname
by Li Ping 05 Nov '20

05 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts before: get hostname from lab-z9, but the hostname not exist in lkp-tests/hosts the job will puts the error message: hosts_file not exist ... but there is no need to puts such error message, if we have got the hostname from lab-z9 --- lib/job.rb | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 2c06a475..12391b39 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,20 +334,14 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + return lab_hosts_file if lab_hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.file?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + puts("hosts_file not exist: #{hosts_file_name}, maybe need check testbox field") end def include_files -- 2.23.0

4 5

[PATCH v3 lkp-tests 3/3] lib/job2sh.rb: fix bug: can not find the file:pkg/$program/PKGBUILD
by Li Ping 05 Nov '20

05 Nov '20

[why] when the $dir not exist, cat > $dir/$file << 'EOF' xxx EOF will echo error message: can not find the file $dir/$file Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job2sh.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..7e6a5d7c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -350,6 +350,7 @@ class Job2sh < Job out_line out_line "\texport define_files='#{define_files.keys.join ' '}'" define_files.each do |file, val| + out_line "\tmkdir -p $LKP_SRC/$(dirname #{file})" out_line "\tcat > $LKP_SRC/#{file} <<'EOF'" out_line val out_line "EOF" -- 2.23.0

2 3

[PATCH v5 compass-ci 1/3] container/assign-account: assign-account.rb
by Luan Shengde 05 Nov '20

05 Nov '20

disable assigning account for user if there is no: - my_email - my_name - my_uuid [why]: my_email, my_name, my_uuid is required when initialize the default config file [how]: check if the parsed data has keys: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/assign-account.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..834f666 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,17 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + check_to_assign_account(data) +end + +def check_to_assign_account(data) + lacked_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lacked_info.join(', ')}." + raise error_message unless lacked_info.empty? + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

2 2

[PATCH v4 lkp-tests 2/2] tests/iozone: modify the way of parsing parameter
by Lu Kaiyi 05 Nov '20

05 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] modify the way of parsing parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 66 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..e126948d 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,52 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + # reset IFS to split $test by "," and then restore default + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write_rewrite") args+=" -i 0" + ;; + "read_reread") args+=" -i 1" + ;; + "random_read_write") args+=" -i 2" + ;; + "read_backwards") args+=" -i 3" + ;; + "rewrite_record") args+=" -i 4" + ;; + "stride_read") args+=" -i 5" + ;; + "fwrite_refwrite") args+=" -i 6" + ;; + "fread_refread") args+=" -i 7" + ;; + "random_mix") args+=" -i 8" + ;; + "pwrite_repwrite") args+=" -i 9" + ;; + "pread_repread") args+=" -i 10" + ;; + "pwritev_repwritev") args+=" -i 11" + ;; + "preadv_repreadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

2 1

[PATCH v2 compass-ci 06/10] container/mail-robot: run.rb
by Luan Shengde 05 Nov '20

05 Nov '20

invoke: send-apply-account-email assign-jumper-account parse-apply-account-email apply-account mail-robot Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/run.rb | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100755 container/mail-robot/run.rb diff --git a/container/mail-robot/run.rb b/container/mail-robot/run.rb new file mode 100755 index 0000000..8ac1d6e --- /dev/null +++ b/container/mail-robot/run.rb @@ -0,0 +1,10 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'send-apply-account-email' +require_relative 'assign-jumper-account' +require_relative 'parse-apply-account-email' +require_relative 'apply-account' +require_relative '../../lib/mail-robot' -- 2.23.0

3 2

[PATCH lkp-tests 1/3] lib/job2sh.rb: fix bug: can not find the file:pkg/$program/PKGBUILD
by Li Ping 05 Nov '20

05 Nov '20

[why] when the $dir not exist, cat > $dir/$file << 'EOF' xxx EOF will echo error message: can not find the file $dir/$file Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job2sh.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..7e6a5d7c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -350,6 +350,7 @@ class Job2sh < Job out_line out_line "\texport define_files='#{define_files.keys.join ' '}'" define_files.each do |file, val| + out_line "\tmkdir -p $LKP_SRC/$(dirname #{file})" out_line "\tcat > $LKP_SRC/#{file} <<'EOF'" out_line val out_line "EOF" -- 2.23.0

2 1

[PATCH v4 lkp-tests 1/2] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 05 Nov '20

05 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..f9ab305f 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write_rewrite,read_reread,random_read_write block_size: - 64k -- 2.23.0

1 0

[PATCH v1 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 05 Nov '20

05 Nov '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..bb66bd1d --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,33 @@ +#!/bin/bash +# - runtime +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is an old version, It is both a benchmark and a stress test for the Linux kernel scheduler. +## It's main job is to create a specified number of pairs of schedulable entities +## (either threads or traditional processes) which communicate via either sockets or +## pipes and time how long it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' + +[ -n "$runtime" ] || runtime=600 +[ -n "$iterations" ] || iterations=10 + + +run_hackbench() +{ + # args: groups number, mode [process/thread], times + local hackbench_path + [ -f /lkp/benchmarks/hackbench-git/hackbench/hackbench ] && hackbench_path=/lkp/benchmarks/hackbench-git/hackbench/hackbench + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench + + -- 2.23.0

2 2

[PATCH lkp-tests 2/3] lib/job.rb: fix bug: add_define_files not work
by Li Ping 05 Nov '20

05 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths actually don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..2c06a475 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -626,6 +626,7 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + program = $1 if program =~ /^(.*)-\d+?$/ # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

1 1

[PATCH lkp-tests] lib/job.rb: fix bug: add_define_files not work
by Li Ping 05 Nov '20

05 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths actually don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index fb826b8c..254562f7 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -620,6 +620,7 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + program = $1 if program =~ /^(.*)-\d+$/ # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

1 0

[PATCH v2 compass-ci 01/10] lib/mail-robot.rb
by Luan Shengde 05 Nov '20

05 Nov '20

mail robot for listening the mail dir monitor mailbox check new added email file in new email directory handle new email file check to apply account only if email's subject match 'apply account' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..ff1dbd8 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +MAILDIR = '/srv/cci/Maildir/.compass-ci' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + begin + handle_new_email(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + puts e.backtrace + end + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + check_to_apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def check_to_apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.check_to_send_account +end + +monitor_dir("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

2 1

[PATCH lkp-tests] lib/job.rb: fix bug: add_define_files not work
by Li Ping 05 Nov '20

05 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths actually don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index fb826b8c..12391b39 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -620,6 +620,7 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + program = $1 if program =~ /^(.*)-\d+?$/ # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: change the way get hostname
by Li Ping 05 Nov '20

05 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts --- lib/job.rb | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 3383897c..75882075 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,20 +334,14 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + return lab_hosts_file if lab_hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.file?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + puts("hosts_file not exist: #{hosts_file_name}, maybe need check testbox field") end def include_files -- 2.23.0

2 1

[PATCH v3 lkp-tests 1/3] lib/job.rb: fix bug: add_define_files not work
by Li Ping 05 Nov '20

05 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths actually don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..3383897c 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -626,6 +626,7 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + program = $1 if program =~ /^(.*)-\d+?/ # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

2 2

[PATCH v4 lkp-tests] pkg: add PKGBUILD for libmicro
by Bai Jing 05 Nov '20

05 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/libmicro/PKGBUILD | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 pkg/libmicro/PKGBUILD diff --git a/pkg/libmicro/PKGBUILD b/pkg/libmicro/PKGBUILD new file mode 100644 index 00000000..c384bb7e --- /dev/null +++ b/pkg/libmicro/PKGBUILD @@ -0,0 +1,19 @@ +pkgname=libMicro +pkgver=0.4 +pkgrel=0 +pkgdesc="A portable set of microbenchmarks focused on the syscall and system library interfaces." +arch=('i386' 'x86_64') +url="https://github.com/rzezeski/libMicro" +license=('CDDL') +source=("https://codeload.github.com/redhat-performance/libMicro/zip/$pkgver.$pkgrel…") +md5sums=('2653728dff1db357096bb9a30589a5b3') + +build() { + cd "$srcdir/$pkgname-$pkgver.$pkgrel-rh" + make +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/$pkgname" + cp -r "$srcdir/$pkgname-$pkgver.$pkgrel-rh/"* "$pkgdir/lkp/benchmarks/$pkgname" +} -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/3] lib/job.rb: change the way get hostname
by Li Ping 05 Nov '20

05 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts --- lib/job.rb | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 3383897c..89780920 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,20 +334,14 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + return lab_hosts_file if lab_hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if hosts_file - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") end def include_files -- 2.23.0

2 2

[PATCH lkp-tests] lib/job.rb: change the way get hostname
by Li Ping 05 Nov '20

05 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts --- lib/job.rb | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 3383897c..6a79799e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,20 +334,14 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + return lab_hosts_file if lab_hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.file?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") end def include_files -- 2.23.0

1 0

[PATCH v2 compass-ci] container/master-fluentd: get logging-es info from yaml
by Wu Zhende 05 Nov '20

05 Nov '20

[Why] "logging-es" and "master-fluentd" may be on different services and may be performed over the network. In this scenario, the logging-es information needs to be obtained from the configuretion file. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 4 ++-- container/master-fluentd/start | 14 +++++++++++++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index 4660988..a57d4cd 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -40,8 +40,8 @@ <store> @type elasticsearch - host 172.17.0.1 - port 9202 + host "#{ENV['LOGGING_ES_HOST']}" + port "#{ENV['LOGGING_ES_PORT']}" suppress_type_name true flush_interval 1s index_name ${tag} diff --git a/container/master-fluentd/start b/container/master-fluentd/start index b3e3dbb..a05881c 100755 --- a/container/master-fluentd/start +++ b/container/master-fluentd/start @@ -3,9 +3,20 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'set' require_relative '../defconfig.rb' -docker_rm "master-fluentd" +names = Set.new %w[ + LOGGING_ES_HOST + LOGGING_ES_PORT +] + +defaults = relevant_defaults(names) +defaults['LOGGING_ES_HOST'] ||= '172.17.0.1' +defaults['LOGGING_ES_PORT'] ||= '9202' +env = docker_env(defaults) + +docker_rm 'master-fluentd' cmd = %w[ docker run @@ -13,6 +24,7 @@ cmd = %w[ --name master-fluentd -v /etc/localtime:/etc/localtime:ro -d +] + env + %w[ -u 1090:1090 -p 24224:24224/tcp -p 24224:24224/udp -- 2.23.0

1 0

[PATCH lkp-tests] fio: fix error when test with write mode
by Wei Jihui 05 Nov '20

05 Nov '20

[Why] when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. [How] add allow_mounted_write when write mode Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 1 + 1 file changed, 1 insertion(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..3ea79fa2 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -95,6 +95,7 @@ fallocate=$fallocate io_size=$io_size file_service_type=$file_service_type random_distribution=$random_distribution +allow_mounted_write=1 group_reporting create_only=1 pre_read=$pre_read -- 2.23.0

2 1

[PATCH v3 compass-ci] container/sub-fluentd: fix heartbeat error
by Wu Zhende 05 Nov '20

05 Nov '20

[Error] fluent.warn: Unknown heartbeat response received from 172.17.0.1:24224. It may service out heartbeat_type default is tcp, delete the "heartbeat_type udp" to use tcp Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/sub-fluentd/docker-fluentd.conf | 1 - 1 file changed, 1 deletion(-) diff --git a/container/sub-fluentd/docker-fluentd.conf b/container/sub-fluentd/docker-fluentd.conf index c674fb1..09fa461 100644 --- a/container/sub-fluentd/docker-fluentd.conf +++ b/container/sub-fluentd/docker-fluentd.conf @@ -56,7 +56,6 @@ @type forward flush_interval 0 send_timeout 60 - heartbeat_type udp heartbeat_interval 1 recover_wait 10 hard_timeout 60 -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/3] lib/job.rb: no need puts the a warning
by Li Ping 04 Nov '20

04 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts --- lib/job.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..86cf3fa1 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -340,10 +340,8 @@ class Job lab_hosts_file = get_lab_hosts_file if lab_hosts_file hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file + elsif File.file?(hosts_file) + hosts_file else puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") nil -- 2.23.0

1 1

[PATCH v2 lkp-tests 3/3] lib/job2sh.rb: fix bug: can not find the file: pkg/$program/PKGBUILD
by Li Ping 04 Nov '20

04 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 15 ++++++--------- lib/job2sh.rb | 1 + 2 files changed, 7 insertions(+), 9 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5632ad88..e45ae682 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,18 +334,15 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + hosts_file = lab_hosts_file + return hosts_file if hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.file?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - elsif File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") end def include_files diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..7e6a5d7c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -350,6 +350,7 @@ class Job2sh < Job out_line out_line "\texport define_files='#{define_files.keys.join ' '}'" define_files.each do |file, val| + out_line "\tmkdir -p $LKP_SRC/$(dirname #{file})" out_line "\tcat > $LKP_SRC/#{file} <<'EOF'" out_line val out_line "EOF" -- 2.23.0

1 1

[PATCH v2 lkp-tests 2/3] lib/job.rb: fix bug: add_define_files not work
by Li Ping 04 Nov '20

04 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths actually don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 86cf3fa1..5632ad88 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -624,6 +624,7 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + program = $1 if program =~ /^(.*)-\d+?/ # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

1 0

[PATCH v4 lkp-tests] jobs/hackbench-git.yaml: fix datasize and threads filed
by Zhang Yale 04 Nov '20

04 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So fix datasize to loops, fix threads to thread. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/hackbench-git.yaml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/jobs/hackbench-git.yaml b/jobs/hackbench-git.yaml index 12943c04..e2320621 100644 --- a/jobs/hackbench-git.yaml +++ b/jobs/hackbench-git.yaml @@ -1,4 +1,4 @@ -suite: hackbench +suite: hackbench-git category: benchmark disable_latency_stats: 1 @@ -11,13 +11,13 @@ nr_threads: - 10 - 20 iterations: 10 -datasize: 1000 +loops: 1000 runtime: 300 -hackbench: +hackbench-git: mode: - process - - threads + - thread ipc: - pipe - socket -- 2.23.0

3 4

[PATCH lkp-tests] depends: add sysbench-mysql-dev
by Zhang Yu 04 Nov '20

04 Nov '20

[why] Dependency for code compilation. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/sysbench-mysql-dev | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 distro/depends/sysbench-mysql-dev diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev new file mode 100644 index 00000000..ca5b03d8 --- /dev/null +++ b/distro/depends/sysbench-mysql-dev @@ -0,0 +1,3 @@ +pkg-config +libmysqlclient-dev +mysql-server -- 2.23.0

1 0

[PATCH compass-ci] lkp-initrd: send the two vars necessary for rsync to rsync script
by Yu Chuan 04 Nov '20

04 Nov '20

[Why] When we rsync result lkp.cgz, we need the follow two vars: - lkp_initrd_user - ARCH Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/lkp-initrd/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/lkp-initrd/run b/container/lkp-initrd/run index a0c246f6ea31..34fe330727b0 100755 --- a/container/lkp-initrd/run +++ b/container/lkp-initrd/run @@ -27,5 +27,5 @@ echo "result: /srv/initrd/lkp/${lkp_initrd_user:-latest}/lkp-${ARCH}.cgz" # rsync lkp-${ARCH}.cgz if needed if [ -x /usr/local/bin/rsync-lkp-cgz ]; then - /usr/local/bin/rsync-lkp-cgz + /usr/local/bin/rsync-lkp-cgz "${lkp_initrd_user:-latest}" "${ARCH}" fi -- 2.23.0

1 0

[PATCH lkp-tests] pkg: add PKGBUILD of sysbench-mysql
by Zhang Yu 04 Nov '20

04 Nov '20

[why] The sysbench 1.0.19 version is required in test. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- pkg/sysbench-mysql | 1 - pkg/sysbench-mysql/PKGBUILD | 23 +++++++++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) delete mode 120000 pkg/sysbench-mysql create mode 100644 pkg/sysbench-mysql/PKGBUILD diff --git a/pkg/sysbench-mysql b/pkg/sysbench-mysql deleted file mode 120000 index a6f8719d..00000000 --- a/pkg/sysbench-mysql +++ /dev/null @@ -1 +0,0 @@ -sysbench \ No newline at end of file diff --git a/pkg/sysbench-mysql/PKGBUILD b/pkg/sysbench-mysql/PKGBUILD new file mode 100644 index 00000000..d9f342a5 --- /dev/null +++ b/pkg/sysbench-mysql/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench +pkgver=1.0 +pkgrel=19 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="https://github.com/akopytov/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("https://github.com/akopytov/sysbench/archive/$pkgver.$pkgrel.tar.gz") +md5sums=('2912bfe7238cac7351459019a84e2557') + +build() { + cd "$pkgname-$pkgver.$pkgrel" + ./autogen.sh + ./configure --with-mysql-includes=/usr/local/mysql/include/ --with-mysql-libs=/usr/local/mysql/lib/ --disable-dependency-tracking + make -j4 +} + +package() { + cd "$pkgname-$pkgver.$pkgrel" + make DESTDIR=$pkgdir install + [[ $os == openeuler ]] && mkdir -p $pkgdir/usr/lib64 + cp -rpf /usr/local/mysql/lib/* $pkgdir/usr/lib64/ +} -- 2.23.0

1 0

[PATCH compass-ci] sbin: fix git-bisect script get incomplete error_id info
by Cao Xueliang 04 Nov '20

04 Nov '20

[why] Before: use error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" get error_id="build-pkg.report-uri" After: use error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" get error_id="build-pkg.report-uri=~https://api.github.com/_private/browser/errors~" Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/git-bisect | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/sbin/git-bisect b/sbin/git-bisect index 4744a02..86f79d4 100755 --- a/sbin/git-bisect +++ b/sbin/git-bisect @@ -12,10 +12,9 @@ require_relative '../lib/mail_bisect_result' def parse_argv items = {} ARGV.each do |item| - key, value = item.split('=') + key, value = item.split('=', 2) if key && value - value_list = value.split(',') - items[key] = value_list.length > 1 ? value_list : value + items[key] = value end end items -- 2.23.0

2 1

[PATCH lkp-tests] fio: fix error when test with write mode
by Wei Jihui 04 Nov '20

04 Nov '20

[Why] when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. [How] add allow_mounted_write when write mode Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..f90963aa 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -116,6 +116,12 @@ $create_task ioscheduler=$ioscheduler" fi +if [[ "$rw" =~ "write" ]]; then + create_task="\ +$create_task +allow_mounted_write=1" +fi + if parse_bool -q "$time_based"; then create_task=" $create_task -- 2.23.0

2 1

[PATCH v3 lkp-tests] (job2sh.rb) add on_fail to job.sh
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh export_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 6e00b088..7d2c2d5c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -367,9 +367,7 @@ class Job2sh < Job @programs = available_programs(:workload_elements) - if !job - job = (@jobx || @job).clone - end + job ||= (@jobx || @job).clone @cur_func = :on_fail -- 2.23.0

2 1

[PATCH v3 lkp-tests] pkg: add PKGBUILD for libmicro
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/libmicro/PKGBUILD | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 pkg/libmicro/PKGBUILD diff --git a/pkg/libmicro/PKGBUILD b/pkg/libmicro/PKGBUILD new file mode 100644 index 00000000..8efc8b19 --- /dev/null +++ b/pkg/libmicro/PKGBUILD @@ -0,0 +1,19 @@ +pkgname=libMicro +pkgver=0.4 +pkgrel=0 +pkgdesc="A portable set of microbenchmarks focused on the syscall and system library interfaces." +arch=('i386' 'x86_64') +url="https://codeload.github.com/redhat-performance/libMicro" +license=('CDDL') +source=("https://codeload.github.com/redhat-performance/libMicro/zip/0.4.0-rh") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname-$pkgver.$pkgrel-rh" + make +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/libMicro" + cp -r "$srcdir/$pkgname-$pkgver.$pkgrel-rh/"* "$pkgdir/lkp/benchmarks/libMicro" +} -- 2.23.0

2 1

[PATCH v2 compass-ci 1/2] auto group jobs with user-defined template
by Lu Weitao 04 Nov '20

04 Nov '20

auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: auto group jobs_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list) + get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + new_job = get_new_job(job, metrics) + next if new_job.empty? + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + value = find_param_in_job(job, param) + group_key_list << value if value + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +def find_param_in_job(job, param) + return job[param] if job.key?(param) + + job.each_value do |v| + return v[param] if v.is_a?(Hash) && v.key?(param) + end +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job(job, metrics) + return {} unless job['stats'] + + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0

1 1

[PATCH v2 compass-ci 09/10] container/mail-robot: start
by Luan Shengde 04 Nov '20

04 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..b65b002 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v /srv/cci/Maildir/.compass-ci:/srv/cci/Maildir/.compass-ci:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + --log-driver json-file + mail-robot +] + +cmd += ['ruby', 'run.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v2 compass-ci 08/10] container/mail-robot: build
by Luan Shengde 04 Nov '20

04 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci 04/10] container/mail-robot: assign-jumper-account.rb
by Luan Shengde 04 Nov '20

04 Nov '20

AssignJumperAccount apply account with apply_account_info apply_account_info - my_email - my_name - my_uuid - my_ssh_pubkey check account exists Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/assign-jumper-account.rb | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 container/mail-robot/assign-jumper-account.rb diff --git a/container/mail-robot/assign-jumper-account.rb b/container/mail-robot/assign-jumper-account.rb new file mode 100755 index 0000000..0728b89 --- /dev/null +++ b/container/mail-robot/assign-jumper-account.rb @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +# apply jumper account for user +# the enter point is apply_jumper_account +# the apply_account_info hash need include +# my_email, my_name, my_uuid, my_ssh_pubkey +class AssignJumperAccount + def initialize(apply_account_info) + @apply_account_info = apply_account_info + end + + def send_error_email(error_message) + @apply_account_info['error_message'] = error_message + send_err_mail = SendApplyAccountEmail.new(@apply_account_info) + send_err_mail.send_error_email + end + + def apply_jumper_account + account_info_str = %x(curl -XGET "#{ENV['JUMPER_HOST']}:#{ENV['JUMPER_PORT']}/assign_account" \ + -d '#{(a)apply_account_info.to_json}') + account_info = JSON.parse account_info_str + + check_account_info(account_info) + + return account_info + end + + def check_account_info(account_info) + return unless account_info['my_login_name'].nil? + + error_message = 'No more available jumper account.' + send_error_email(error_message) + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] create matrices-groups according to group jobs
by Lu Weitao 04 Nov '20

04 Nov '20

create matrices-groups according to auto group job_list background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: create groups_matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 05f96dc..b38b2fb 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -120,3 +120,27 @@ def combine_group_query_data(query_data, dims) end groups end + +# input: +# 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() +# 2. params from user's template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# return: group_matrix of Hash(String, Hash(String, matrix)) +def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) + job_list = query_data['hits']['hits'] + groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) + groups.each do |group_key, dims| + dims.each do |dim_key, jobs| + groups[group_key][dim_key] = create_matrix(jobs) + end + end + groups +end -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] create matrices-groups according to group jobs
by Lu Weitao 04 Nov '20

04 Nov '20

create matrices-groups according to auto group job_list background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: create groups_matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 05f96dc..b38b2fb 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -120,3 +120,27 @@ def combine_group_query_data(query_data, dims) end groups end + +# input: +# 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() +# 2. params from user's template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# return: group_matrix of Hash(String, Hash(String, matrix)) +def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) + job_list = query_data['hits']['hits'] + groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) + groups.each do |group_key, dims| + dims.each do |dim_key, jobs| + groups[group_key][dim_key] = create_matrix(jobs) + end + end + groups +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] execute on_fail when exit-code isn't 0
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] when a job-task fails(exit_code != 0), the on_fail function needs to be called to keep the program in the waiting state to user adjustment Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- bin/run-lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/run-lkp b/bin/run-lkp index 26669331..c4d43db4 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -97,6 +97,7 @@ elif [ "$run_job_exit_code" != 0 ]; then else set_job_state 'failed' fi + $LKP_DEBUG_PREFIX $job_script on_fail elif [ -f '/var/log/wtmp' ] && has_cmd last && last -n1 | grep -q -E '(pts/|tty)[0-9].*(still logged in)'; then set_job_state 'disturbed' disturbed=1 -- 2.23.0

1 0

[PATCH v3 lkp-tests] (job2sh.rb) add on_fail to job.sh
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh export_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..d128e8ef 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -357,6 +359,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs: add libmicro.yaml
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/libmicro.yaml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 jobs/libmicro.yaml diff --git a/jobs/libmicro.yaml b/jobs/libmicro.yaml new file mode 100644 index 00000000..446ab23b --- /dev/null +++ b/jobs/libmicro.yaml @@ -0,0 +1,4 @@ +suite: libmicro +category: benchmark + +libmicro: -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix bug when program-0 add_define_files not work
by Li Ping 04 Nov '20

04 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 86cf3fa1..67bd6d5e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -624,6 +624,9 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + if program =~ /^(.*)-\d+?/ + program = $1 + end # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

4 6

[PATCH v5 compass-ci 2/3] container/assign-account: get_account_info
by Luan Shengde 04 Nov '20

04 Nov '20

add function: config default yaml file [why]: easier for user to config the default yaml file [how]: write received user infos to the defaults yaml file: ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 51 +++++++++++++------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..b2d2474 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,35 +95,48 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) + setup_authorized_key(login_name, pub_key) unless pub_key.nil? + setup_default_yaml(login_name) + return jumper_account_info end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def setup_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + end + + def setup_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

2 3

[PATCH lkp-tests 2/2] lib/job2sh.rb: fix bug: can not find the file: pkg/$program/PKGBUILD
by Li Ping 04 Nov '20

04 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job2sh.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..7e6a5d7c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -350,6 +350,7 @@ class Job2sh < Job out_line out_line "\texport define_files='#{define_files.keys.join ' '}'" define_files.each do |file, val| + out_line "\tmkdir -p $LKP_SRC/$(dirname #{file})" out_line "\tcat > $LKP_SRC/#{file} <<'EOF'" out_line val out_line "EOF" -- 2.23.0

3 3

[PATCH lkp-tests] jobs: add yaml for libmicro
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- jobs/libmicro.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/libmicro.yaml diff --git a/jobs/libmicro.yaml b/jobs/libmicro.yaml new file mode 100644 index 00000000..9a577eb8 --- /dev/null +++ b/jobs/libmicro.yaml @@ -0,0 +1,5 @@ +suite: libmicro +testcase: libmicro +category: benchmark + +libmicro: -- 2.23.0

4 3

[PATCH v2 lkp-tests] lib/job.rb: no need puts the a warning
by Li Ping 04 Nov '20

04 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts --- lib/job.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..86cf3fa1 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -340,10 +340,8 @@ class Job lab_hosts_file = get_lab_hosts_file if lab_hosts_file hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file + elsif File.file?(hosts_file) + hosts_file else puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") nil -- 2.23.0

2 2

[PATCH v5 compass-ci 3/3] container/assign-account: answerback-email.rb
by Luan Shengde 04 Nov '20

04 Nov '20

1. disable use golbal variable apply_info [why]: rubocop: Style/GlobalVars: Do not introduce global variables. [how]: use local variables instead of global variables 2. fix OptionParser argument: [why]: when write this: opts.on('-e|--email email_address', 'appoint email address') will lead to could not identify the options ./answerback-email.rb --email xxx(a)163.com ./answerback-email.rb:72:in `<main>': invalid option: --email (OptionParser::InvalidOption) [how]: write seprately like: opts.on('-e email_address', '--email email_address', 'appoint email address') 3. add transfer user info when execute apply account command [why]: when applying account, the assign-account service need to write the my info to user's default config file ~/.config/compass-ci/defaults/account.yaml my info: - my_email - my_name - my_uuid [how]: transfer the user info along with the pub_key when sending apply account request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 92 ++++++++++++-------- 1 file changed, 55 insertions(+), 37 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..bda627d 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,58 +12,65 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +apply_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, apply_info) mail_content = Mail.read(email_file) + apply_info['my_email'] = mail_content.from[0] + apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + apply_info end options = OptionParser.new do |opts| - opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key_file] [--raw-email email_file]\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" opts.banner += " -e or -f is required\n" opts.banner += ' -s is optional when use -e' opts.separator '' opts.separator 'options:' - opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + apply_info['my_email'] = email_address + # when apply account with email address, will get no user name + apply_info['my_name'] = '' end - opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ + 'ssh pub_key file, enable password-less login') do |pub_key_file| + apply_info['my_ssh_pubkey'] = File.read(pub_key_file) end - opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + init_info(email_file, apply_info) end - opts.on_tail('-h|--help', 'show this message') do + opts.on_tail('-h', '--help', 'show this message') do puts opts exit end @@ -71,10 +78,10 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: @@ -82,10 +89,10 @@ def build_message(email, acct_infos) You can use the following command to login the jumper server: login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} account password: - #{acct_infos['passwd']} + #{account_info['my_password']} regards compass-ci @@ -94,26 +101,37 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(apply_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(apply_info) message = "No email address specified\n" message += "use -e email_address add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? - - acct_info = account_info($apply_info['my_ssh_pubkey']) - - message = build_message($apply_info['my_email'], acct_info) + raise message if apply_info['my_email'].nil? + + account_info = apply_account(apply_info) + my_info = { + 'my_email' => apply_info['my_email'], + 'my_name' => apply_info['my_name'], + # there is no need to add a commit url for this tool + # the es has the key: my_commit_url + 'my_commit_url' => '', + 'my_login_name' => account_info['my_login_name'], + 'my_uuid' => apply_info['my_uuid'] + } + + store_account_info(my_info) + message = build_message(apply_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(apply_info) -- 2.23.0

2 2

[PATCH compass-ci] kernel_params.cr: add sched_steal_node_limit=8 when test mysql optimization kernel
by Xu Xijian 04 Nov '20

04 Nov '20

[why] When test the kernel with mysql optimization, the parameter "sched_steal_node_limit=8" is required to be added into kernel params. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 54438b5..546f012 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -31,6 +31,9 @@ class Job private def set_kernel_params self["kernel_params"] = " #{kernel_common_params()} #{kernel_append_root} #{kernel_console()}" + if "#{linux_vmlinuz_path}".includes?("mysql") + self["kernel_params"] += " sched_steal_node_limit=8" + end end end -- 2.23.0

3 3

[PATCH compass-ci 3/8] container/mail-robot: apply-account-component.rb
by Luan Shengde 04 Nov '20

04 Nov '20

ParseApplyAccountEmail parse commit url and pub_key check out commit url check commit url exists check base url in upstream-repos check commit available gitee.com clone the repo and check the commit exists non-gitee.com check the feedback of curl command to check the commit url available check out pub_key extract the attachment of the pub_key ApplyAccount apply account with apply_account_info check account SendMail send uuid email build email message send email send error email build error email message send error email Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- .../mail-robot/apply-account-component.rb | 232 ++++++++++++++++++ 1 file changed, 232 insertions(+) create mode 100755 container/mail-robot/apply-account-component.rb diff --git a/container/mail-robot/apply-account-component.rb b/container/mail-robot/apply-account-component.rb new file mode 100755 index 0000000..98272e7 --- /dev/null +++ b/container/mail-robot/apply-account-component.rb @@ -0,0 +1,232 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check whether there is a commit url and ssh pub_key in the email +# apply account on junper server +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + @my_email = mail_content.from[0] + @my_name = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + end + + def parse_commit_url_pub_key + commit_url = check_out_commit_url + pub_key = check_out_pub_key + + return commit_url, pub_key + end + + def send_error_email(error_message) + send_error_email_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_info' => '', + 'error_message' => error_message + } + send_err_mail = SendMail.new(send_error_email_info) + send_err_mail.send_error_email + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = 'No matched commit url found' + send_error_email(error_message) + + raise error_message + end + + def check_out_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + check_base_url_in_upstream_repos('/c/upstream-repos', base_url) + check_commit_available(url, base_url) + + return url + end + + def check_base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = 'The url is not in upstream-repo list' + send_error_email(error_message) + + raise error_message + end + + def check_commit_available(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + check_gitee_commit(url, base_url) + else + check_non_gitee_commit(url) + end + end + + def check_non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_email + + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_gitee_commit(url, base_url) + repo_dir = url.split('/')[-3] + repo_url = [base_url, 'git'].join('.') + commit_id = url.split('/')[-1] + + Dir.chdir ENV['ROBOT_TMP_REPOS_DIR'] + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_email + + FileUtils.rm_rf repo_dir + + check_gitee_commit_exist(email_index) + end + + def check_gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_out_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + check_pub_key(pub_key) + + return pub_key + end + + def check_pub_key(pub_key) + return unless pub_key.nil? + + error_message = 'No pub_key found' + send_error_email(error_message) + + raise error_message + end +end + +# apply jumper account for user +class ApplyAccount + def initialize(apply_account_info) + @apply_account_info = apply_account_info + end + + def apply_jumper_account + account_info_str = %x(curl -XGET "#{ENV['JUMPER_HOST']}:#{ENV['JUMPER_PORT']}/assign_account" \ + -d '#{(a)apply_account_info.to_json}') + account_info = JSON.parse account_info_str + + check_account_info(account_info) + + return account_info + end + + def check_account_info(account_info) + return unless account_info['account'].nil? + + error_message = 'No more available jumper account.' + send_error_email(error_message) + + raise error_message + end +end + +# build mail data and send mail +class SendMail + def initialize(send_mail_info) + @send_mail_host = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + @my_email = send_mail_info['my_email'] + @my_name = send_mail_info['my_name'] + @error_message = send_mail_info['error_message'] + @my_info = send_mail_info['my_info'] + end + + def send_uuid_email + email_msg = <<~EMAIL_MESSAGE + To: #{@my_email} + Subject: [compass-ci] Account Ready + + Dear #{@my_name}, + + Thank you for joining us. + You can use the following info to submit jobs: + + 1) setup default config + run the following command to add the below setup to default config file + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/\\${USER}.yaml <<-EOF + my_uuid: #{@my_info['my_uuid']} + my_email: #{@my_info['my_email']} + my_name: #{@my_info['my_name']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + find how to write job yaml + submit jobs/netperf.yaml + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_error_email + email_msg = <<~EMAIL_MESSAGE + To: #{@my_email} + Subject: apply account failed + + Dear #{@my_name} + + Your application for account failed with following errors + + #{@error_message} + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_mail(email_msg) + %x(curl -XPOST "#{@send_mail_host}:#{ENV['SEND_MAIL_PORT']}/send_mail_text" -d "#{email_msg}") + end +end -- 2.23.0

2 8

[PATCH v2 compass-ci] container/sub-fluentd: fix heartbeat error
by Wu Zhende 04 Nov '20

04 Nov '20

[Error] fluent.warn: Unknown heartbeat response received from 172.17.0.1:24224. It may service out heartbeat_type default is tcp, delete the "heartbeat_type udp" to use tcp Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/sub-fluentd/docker-fluentd.conf | 1 - 1 file changed, 1 deletion(-) diff --git a/container/sub-fluentd/docker-fluentd.conf b/container/sub-fluentd/docker-fluentd.conf index c674fb1..09fa461 100644 --- a/container/sub-fluentd/docker-fluentd.conf +++ b/container/sub-fluentd/docker-fluentd.conf @@ -56,7 +56,6 @@ @type forward flush_interval 0 send_timeout 60 - heartbeat_type udp heartbeat_interval 1 recover_wait 10 hard_timeout 60 -- 2.23.0

2 1

[PATCH v9 compass-ci 2/2] kernel_version.md: explain key "kernel_version"
by Xu Xijian 04 Nov '20

04 Nov '20

[why] Explain the meaning of new key "kernel_version" for scheduler, including what's the mapping between its typical values and actual files in the disk and how to set it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 doc/job/kernel_version.md diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md new file mode 100644 index 0000000..f6a5f4d --- /dev/null +++ b/doc/job/kernel_version.md @@ -0,0 +1,39 @@ +# kernel_version + +Meaning: +- Every os has its kernel, however an os can start with different kernels according to different need. +- kernel_version is a key for users to specify a kernel version. +- If kernel_version is not given by users, it will use the default one. +- Here are the kernel versions we support for different os: + openeuler/aarch64/1.0 + - 4.19.90-2001(default) + + openeuler/aarch64/20.03 + - 4.19.90-2003(default) + - 4.19.90-mysql + - 4.19.90-nginx + + debian/aarch64/sid + - 5.4.0-4-arm64(default) + - 5.8.0-1-arm64 + + centos/aarch64/7.6.1810 + - 4.14.0-115.el7a.0.1(default) + + centos/aarch64/7.8.2003 + - 4.18.0-147.8.1.el7(default) + + centos/aarch64/8.1.1911 + - 4.18.0-147.el8(default) + +Related files: +- In initramfs boot process, every kernel version is related with a vmlinuz, module and headers. +- Files like below under $boot_dir, an example $boot_dir can be "/srv/os/openeuler/aarch64/20.03/boot". +├── headers-4.19.90-2003.cgz +├── headers.cgz -> headers-4.19.90-2003.cgz +├── modules-4.19.90-2003.cgz +├── modules.cgz -> modules-4.19.90-2003.cgz +├── vmlinuz-4.19.90-2003 + +Usage example: +- submit iperf.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 -- 2.23.0

4 4

[PATCH compass-ci] container/master-fluentd: get logging-es info from yaml
by Wu Zhende 04 Nov '20

04 Nov '20

[Why] "logging-es" and "master-fluentd" may be on different services and may be performed over the network. In this scenario, the logging-es information needs to be obtained from the configuretion file. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 4 ++-- container/master-fluentd/start | 14 +++++++++++++- 2 files changed, 15 insertions(+), 3 deletions(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index 4660988..a57d4cd 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -40,8 +40,8 @@ <store> @type elasticsearch - host 172.17.0.1 - port 9202 + host "#{ENV['LOGGING_ES_HOST']}" + port "#{ENV['LOGGING_ES_PORT']}" suppress_type_name true flush_interval 1s index_name ${tag} diff --git a/container/master-fluentd/start b/container/master-fluentd/start index b3e3dbb..a05881c 100755 --- a/container/master-fluentd/start +++ b/container/master-fluentd/start @@ -3,9 +3,20 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'set' require_relative '../defconfig.rb' -docker_rm "master-fluentd" +names = Set.new %w[ + LOGGING_ES_HOST + LOGGING_ES_PORT +] + +defaults = relevant_defaults(names) +defaults['LOGGING_ES_HOST'] ||= '172.17.0.1' +defaults['LOGGING_ES_PORT'] ||= '9202' +env = docker_env(defaults) + +docker_rm 'master-fluentd' cmd = %w[ docker run @@ -13,6 +24,7 @@ cmd = %w[ --name master-fluentd -v /etc/localtime:/etc/localtime:ro -d +] + env + %w[ -u 1090:1090 -p 24224:24224/tcp -p 24224:24224/udp -- 2.23.0

2 1

[PATCH v2 lab-z9] hosts: fix taishan200-2280-2s48p-256g--a102 to taishan200-2280-2s48p-512g--a102
by Zhang Yu 04 Nov '20

04 Nov '20

[why] The memory of taishan200-2280-2s48p-256g--a102 is 512g, so need to fix 256g to 512g. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename hosts/{taishan200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} (100%) diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-512g--a102 similarity index 100% rename from hosts/taishan200-2280-2s48p-256g--a102 rename to hosts/taishan200-2280-2s48p-512g--a102 -- 2.23.0

2 2

[PATCH lkp-tests 1/2] lib/job.rb: no need puts the a warning
by Li Ping 04 Nov '20

04 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts --- lib/job.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..86cf3fa1 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -340,10 +340,8 @@ class Job lab_hosts_file = get_lab_hosts_file if lab_hosts_file hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file + elsif File.file?(hosts_file) + hosts_file else puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") nil -- 2.23.0

1 0

[PATCH lkp-tests] libmicro: add PKGBUILD for libmicro
by Bai Jing 04 Nov '20

04 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/libmicro/PKGBUILD | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 pkg/libmicro/PKGBUILD diff --git a/pkg/libmicro/PKGBUILD b/pkg/libmicro/PKGBUILD new file mode 100644 index 00000000..e421c402 --- /dev/null +++ b/pkg/libmicro/PKGBUILD @@ -0,0 +1,20 @@ +pkgname=libMicro +pkgver=0.4 +pkgrel=0 +pkgdesc="A portable set of microbenchmarks focused on the syscall and system library interfaces." +arch=('i386' 'x86_64') +url="https://github.com/rzezeski/libMicro" +license=('CDDL') +source=("https://codeload.github.com/redhat-performance/libMicro/zip/0.4.0-rh") +md5sums=('SKIP') + + +build() { + cd "$srcdir/$pkgname-$pkgver.$pkgrel-rh" + make +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/libMicro" + cp -r "$srcdir/$pkgname-$pkgver.$pkgrel-rh/"* "$pkgdir/lkp/benchmarks/libMicro" +} -- 2.23.0

2 2

[PATCH compass-ci] container/sub-fluentd: fix heartbeat error
by Wu Zhende 04 Nov '20

04 Nov '20

[Error] fluent.warn: Unknown heartbeat response received from 172.17.0.1:24224. It may service out Can't use udp to send heartbeat package. Change to tcp. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/sub-fluentd/docker-fluentd.conf | 1 - 1 file changed, 1 deletion(-) diff --git a/container/sub-fluentd/docker-fluentd.conf b/container/sub-fluentd/docker-fluentd.conf index c674fb1..09fa461 100644 --- a/container/sub-fluentd/docker-fluentd.conf +++ b/container/sub-fluentd/docker-fluentd.conf @@ -56,7 +56,6 @@ @type forward flush_interval 0 send_timeout 60 - heartbeat_type udp heartbeat_interval 1 recover_wait 10 hard_timeout 60 -- 2.23.0

3 2

[PATCH v5 lkp-tests] tests/build-pkg: modify save location for *.cgz file
by Liu Shaofei 04 Nov '20

04 Nov '20

[why] the path of *.cgz file generated by build-pkg script cannot be obtained during test. So the path of cgz file need to be changed. [how] below is example for openeuler: latest.cgz -> f71820d6055373c9b9381615e28a28f307bb2df4.cgz f71820d6055373c9b9381615e28a28f307bb2df4.cgz v1.8.1.cgz -> 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz v1.5.1.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz df0584e46db9e7b412626deab10b923504745b2c.cgz Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..7c494a15 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,15 +23,19 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount $LKP_SERVER:$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } @@ -75,12 +81,22 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 +} + +create_softlink() +{ + [ -n "$upstream_tag" ] || return 0 + [ -e "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" ] && return 0 + + ln -sf "$(basename $(realpath ${cgz_name}))" "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" || return 0 + echo "create soft link: $PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz -> $(basename $(realpath ${cgz_name}))" } check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

4 5

[PATCH v3 compass-ci] sparrow/3-code: add ENV LOGGING_ES_HOST and LOGGING_ES_PORT
by Liu Yinsi 04 Nov '20

04 Nov '20

need LOGGING_ES_HOST and LOGGING_ES_PORT to start logging-es service. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 64f34f8..f4e4fa9 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -13,6 +13,8 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${MASTER_FLUENTD_PORT:=24224} : ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} +: ${LOGGING_ES_HOST:=$server_ip} +: ${LOGGING_ES_PORT:=9202} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -28,9 +30,11 @@ MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT DOCKER_REGISTRY_HOST: $DOCKER_REGISTRY_HOST DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT +LOGGING_ES_HOST: $LOGGING_ES_HOST +LOGGING_ES_PORT: $LOGGING_ES_PORT EOF -cat > /etc/profile.d/compass.sh <<'EOF' +cat > /etc/profile.d/compass.sh <<EOF export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci export REPO_SRC=/c/upstream-repos -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/3-code: add ENV LOGGING_ES_HOST and LOGGING_ES_PORT
by Liu Yinsi 04 Nov '20

04 Nov '20

need LOGGING_ES_HOST and LOGGING_ES_PORT to start logging-es service. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 64f34f8..dc329fe 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -13,6 +13,8 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${MASTER_FLUENTD_PORT:=24224} : ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} +: ${LOGGING_ES_HOST:=$server_ip} +: ${LOGGING_ES_PORT:=9202} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -28,6 +30,8 @@ MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT DOCKER_REGISTRY_HOST: $DOCKER_REGISTRY_HOST DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT +LOGGING_ES_HOST: $LOGGING_ES_HOST +LOGGING_ES_PORT: $LOGGING_ES_PORT EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

2 2

[PATCH v3 compass-ci] monitoring/filter.cr: query value support regularity
by Wu Zhende 04 Nov '20

04 Nov '20

[Why] Enhanced monitoring function. Regular expressions can be used in query's value. query can be {"job_id": "z9.*"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index bc6f901..88702b6 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,13 +56,20 @@ class Filter def match_query(query : Hash(String, JSON::Any), msg : Hash(String, JSON::Any)) query.each do |key, value| - value = value.as_a - if value.includes?(nil) - return false unless msg.has_key?(key) - else - return false unless value.includes?(msg[key]?) - end + return false unless msg.has_key?(key) + + values = value.as_a + next if values.includes?(nil) || values.includes?(msg[key]?) + + return false unless regular_match(values, msg[key]?.to_s) end return true end + + private def regular_match(rules, string) + rules.each do |rule| + return true if string =~ /#{rule}/ + end + return false + end end -- 2.23.0

1 0

[PATCH lkp-tests] job.rb: fix add second program to pp
by Wei Jihui 04 Nov '20

04 Nov '20

[Why] Now we add new field in job yaml like this: on_fail: sleep: 300 and we want add sleep to pp, but at pp() function, it only traverse job's keys--->on_fail, this can not find sleep. it should find sleep at on_fail's keys. [How] when traverse job to find program, in this case, it is sleep, then find it at on_fail's keys, not job's keys. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..3fdf8c79 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -576,10 +576,10 @@ class Job @job[comment_to_symbol('auto generated by add_pp()')] = nil @job["pp"] = Hash.new() init_program_options() - for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, p_args| - k = hash_key_re_string(@job, p_n, @job['pp'].keys) - if k && !(a)job[k].is_a?(Hash) - @job['pp'][k] = @job[k] + for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, _p_args| + k = hash_key_re_string(h, p_n, @job['pp'].keys) + if k && h[k] && !h[k].is_a?(Hash) + @job['pp'][k] = h[k] next end options_hash = @referenced_programs[p_n] -- 2.23.0

1 0

[PATCH lab-z9] hosts: fix taishan200-2280-2s48p-256g--a102 to taishan200-2280-2s48p-512g--a102
by Zhang Yu 04 Nov '20

04 Nov '20

[Why] The memory of taishan200-2280-2s48p-256g--a95 is 512g, so need to fix 256g to 512g. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename hosts/{taishan200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} (100%) diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-512g--a102 similarity index 100% rename from hosts/taishan200-2280-2s48p-256g--a102 rename to hosts/taishan200-2280-2s48p-512g--a102 -- 2.23.0

1 1

[PATCH v4 lkp-tests] tests/build-pkg: modify save location for *.cgz file
by Liu Shaofei 04 Nov '20

04 Nov '20

[why] the path of *.cgz file generated by build-pkg script cannot be obtained during test. So the path of cgz file need to be changed. [how] below is example for openeuler: latest.cgz -> f71820d6055373c9b9381615e28a28f307bb2df4.cgz f71820d6055373c9b9381615e28a28f307bb2df4.cgz v1.8.1.cgz -> 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz v1.5.1.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz df0584e46db9e7b412626deab10b923504745b2c.cgz Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..f9bfa377 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,15 +23,19 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount $LKP_SERVER:$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } @@ -75,12 +81,22 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 } +create_softlink() +{ + [ -n "$upstream_tag" ] || return 0 + [ -e "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" ] && return 0 + + ln -sf "$(basename $(realpath ${cgz_name}))" "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" || return 0 + echo "create soft link: $PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz -> $(basename $(realpath ${cgz_name}))" +} + check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

3 4

[PATCH v2 compass-ci] monitoring/filter.cr: query value support regularity
by Wu Zhende 04 Nov '20

04 Nov '20

[Why] Enhanced monitoring function. Regular expressions can be used in query's value. query can be {"job_id": "z9.*"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index bc6f901..d520e6e 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,13 +56,21 @@ class Filter def match_query(query : Hash(String, JSON::Any), msg : Hash(String, JSON::Any)) query.each do |key, value| + return false unless msg.has_key?(key) + value = value.as_a - if value.includes?(nil) - return false unless msg.has_key?(key) - else - return false unless value.includes?(msg[key]?) - end + next if value.includes?(nil) + next if value.includes?(msg[key]?) + + return false unless regular_match(value, msg[key]?.to_s) end return true end + + private def regular_match(rules, string) + rules.each do |rule| + return true if string =~ /#{rule}/ + end + return false + end end -- 2.23.0

2 1

[PATCH v1 compass-ci] improve multi-qemu code to support queue parameter
by Xiao Shenwei 04 Nov '20

04 Nov '20

Usage: multi-qemu [-n] [-c] [-q] -n, --name HOSTNAME_PREFIX specify used hostname_prefix -c, --count count how much VM do you need -q, --queues queues queues from which tasks are obtained -h, --help show this message example: ./multi-qemu -n vm-2p8g.xsw -c 2 -q vm-2p8g~xsw,vm-2p8g.aarch64 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-qemu | 98 +++++++++++++++++++++++++++++++++----------- 1 file changed, 74 insertions(+), 24 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 23e0451..03564f7 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -4,41 +4,91 @@ # frozen_string_literal: true require 'fileutils' +require 'optparse' +require 'time' -PWD = Dir.pwd +opt = {} +options = OptionParser.new do |opts| + opts.banner = 'Usage: multi-qemu [-n] [-c] [-q]' + + opts.separator '' + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'specify used hostname_prefix') do |name| + opt['hostname_prefix'] = name + end + + opts.on('-c count', '--count count', 'how much VM do you need') do |num| + opt['nr_vm'] = num + end + + opts.on('-q queues', '--queues queues', 'requested queues, use "," to separate more than 2 values') do |queues| + opt['queues'] = queues + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.size.zero? + puts options + exit +end + +options.parse!(ARGV) # Run multiple QEMU in parallel -HOSTNAME = ARGV[0] || "vm-2p8g--#{ENV['USER']}" -NR_VM = ARGV[1] || 1 +PWD = Dir.pwd +HOSTNAME = opt['hostname_prefix'] || "vm-2p8g.#{ENV['USER']}" +NR_VM = opt['nr_vm'] || 1 +QUEUES = opt['queues'] || "vm-2p8g.#{RUBY_PLATFORM.split('-')[0]}" +LOG_DIR = '/srv/cci/serial/logs' -def run(seqno) - loop do - start_time = Time.new - hostname = "#{HOSTNAME}-#{seqno}" - log_file = "/srv/cci/serial/logs/#{hostname}" +def main(hostname) + start_time = Time.new.strftime('%Y-%m-%d %H:%M:%S') + log_file = "#{LOG_DIR}/#{hostname}" + record_runtime(log_file, start_time) + run_vm(hostname) + duration = ((Time.new - Time.parse(start_time)) / 60).round(2) + record_runtime(log_file, duration, is_start: false) +end +def record_runtime(log_file, message, is_start: true) + if is_start File.open(log_file, 'w') do |f| # fluentd refresh time is 1s # let fluentd to monitor this file first sleep(2) - f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting QEMU" + f.puts "\n#{message} starting QEMU" end + return + end + File.open(log_file, 'a') do |f| + f.puts "\nTotal QEMU duration: #{message} minutes" + end +end - pwd_hostname = File.join(PWD, hostname) - FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) - FileUtils.cd(pwd_hostname) - system( - { 'hostname' => hostname }, - ENV['CCI_SRC'] + '/providers/qemu.sh' - ) - - duration = ((Time.new - start_time) / 60).round(2) - File.open(log_file, 'a') do |f| - f.puts "\nTotal QEMU duration: #{duration} minutes" - end +def run_vm(hostname) + pwd_hostname = File.join(PWD, hostname) + FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) + FileUtils.cd(pwd_hostname) + system( + { 'hostname' => hostname, 'queues' => QUEUES }, + ENV['CCI_SRC'] + '/providers/qemu.sh' + ) +end - # sleep 5s is for fluentd to collect it's log - sleep(5) +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.backtrace + # if an exception happend, request the next time after 30 seconds + sleep 25 + ensure + sleep 5 + end end end @@ -55,7 +105,7 @@ def multiqemu pids = [] NR_VM.to_i.times do |i| pid = Process.fork do - run i + loop_main("#{HOSTNAME}-#{i}") end pids << pid end -- 2.23.0

2 3

[PATCH v2 lkp-tests 1/3] (job2sh.rb) add on_fail to job.sh
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh export_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..6e00b088 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors','on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -357,6 +359,30 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + if !job + job = (@jobx || @job).clone + end + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

3 2

[PATCH lkp-tests] add testcase for sysbench-threads-git
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] old version package needs to debug by new testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 tests/sysbench-threads-git diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git new file mode 100755 index 00000000..712b3359 --- /dev/null +++ b/tests/sysbench-threads-git @@ -0,0 +1,17 @@ +#!/bin/sh +# - nr_threads +# - thread_yields +# - thread_locks + +. "$LKP_SRC/lib/reproduce-log.sh" + +: "${nr_threads:=2}" +: "${thread_yields:=100}" +: "${thread_locks:=2}" + +args="\ + --threads=$nr_threads\ + --thread-yields=$thread_yields\ + --thread-locks=$thread_locks\ +" +log_cmd sysbench --test=threads $args run -- 2.23.0

2 1

[PATCH lkp-tests] add PKGBUILD for sysbench-threads-git
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] Adapting sysbench-threads needs to old version to user. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..2779c294 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,21 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=0.4.8 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="http://github.com/dallasmarlow/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() { + cd "$srcdir/$pkgname" + make DESTDIR=$pkgdir install +} -- 2.23.0

1 0

[PATCH lkp-tests] add PKGBUILD for sysbech-threads-git
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] Adapting sysbench-threads needs to old version to user. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..bf47d5a1 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,21 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=0.4.8 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="https://github.com/akopytov/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() { + cd "$srcdir/$pkgname" + make DESTDIR=$pkgdir install +} -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/3] submit on_fail to the scheduler
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] When job2sh.rb defined sh_on_fail, it needs to be invoked by the scheduler. Finally, the task is in the waiting state when the task is failed. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..0bb41f2d 100755 --- a/sbin/submit +++ b/sbin/submit @@ -115,12 +115,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_fail sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] job-task is waiting when exit_code != 0
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] when a job-task fails(exit_code != 0), the on_fail function needs to be called to keep the program in the waiting state to user adjustment Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- bin/run-lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/run-lkp b/bin/run-lkp index 26669331..c4d43db4 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -97,6 +97,7 @@ elif [ "$run_job_exit_code" != 0 ]; then else set_job_state 'failed' fi + $LKP_DEBUG_PREFIX $job_script on_fail elif [ -f '/var/log/wtmp' ] && has_cmd last && last -n1 | grep -q -E '(pts/|tty)[0-9].*(still logged in)'; then set_job_state 'disturbed' disturbed=1 -- 2.23.0

1 0

[PATCH v1 lkp-tests 2/2] accept queues parameter
by Xiao Shenwei 04 Nov '20

04 Nov '20

add queues field on multi-qemu.yaml: multi-qemu-0: nr_vm: 2 tbox_group: vm-2p8g queues: vm-2p8g.$HOSTNAME,xxx Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/multi-qemu | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index 5f4d976b..3918a6f6 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -1,6 +1,7 @@ #!/bin/sh # - tbox_group # - nr_vm +# - queues # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. @@ -9,10 +10,10 @@ multi_qemu() { [ -f "/usr/share/qemu-efi-aarch64/QEMU_EFI.fd" ] || return export CCI_SRC=/c/compass-ci - local hostname=$tbox_group--$HOSTNAME + local hostname=$tbox_group.$HOSTNAME cd "$CACHE_DIR" - $CCI_SRC/providers/multi-qemu "$hostname" "$nr_vm" + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" } multi_qemu -- 2.23.0

1 1

[PATCH v1 lkp-tests 1/2] check QEMU_EFI.fd file for multi-qemu
by Xiao Shenwei 04 Nov '20

04 Nov '20

[why] for multi-qemu job, the qemu-efi container will build error, then the QEMU_EFI.fd will not found, it caused one job will be remove from redis, but kvm can't execute it. so, this job looks like it's been executed, but not result file. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/multi-qemu | 1 + 1 file changed, 1 insertion(+) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index b407251f..5f4d976b 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -7,6 +7,7 @@ multi_qemu() { + [ -f "/usr/share/qemu-efi-aarch64/QEMU_EFI.fd" ] || return export CCI_SRC=/c/compass-ci local hostname=$tbox_group--$HOSTNAME -- 2.23.0

1 1

[PATCH compass-ci] sparrow/3-code: add ENV LOGGING_ES_HOST and LOGGING_ES_PORT
by Liu Yinsi 04 Nov '20

04 Nov '20

need LOGGING_ES_HOST and LOGGING_ES_PORT to start logging-es service. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 64f34f8..01393f8 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -9,6 +9,8 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${initrd_http_port:=8800} : ${gitcache_host:=$server_ip} : ${gitcache_port:=5000} +: $logging_es_host:=$server_ip +: $logging_es_port:=9202 : ${MASTER_FLUENTD_HOST:=$server_ip} : ${MASTER_FLUENTD_PORT:=24224} : ${DOCKER_REGISTRY_HOST:=$server_ip} @@ -28,6 +30,8 @@ MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT DOCKER_REGISTRY_HOST: $DOCKER_REGISTRY_HOST DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT +LOGGING_ES_HOST: $logging_es_host +LOGGING_ES_PORT: $logging_es_port EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

2 2

[PATCH v4 compass-ci 3/3] container/assign-account: answerback-email.rb
by Luan Shengde 04 Nov '20

04 Nov '20

1. disable use golbal variable apply_info [why]: rubocop: Style/GlobalVars: Do not introduce global variables. [how]: use local variables instead of global variables 2. add transfer user info when execute apply account command [why]: when applying account, the assign-account service need to write the my info to user's default config file ~/.config/compass-ci/defaults/account.yaml my info: - my_email - my_name - my_uuid [how]: transfer the user info along with the pub_key when sending apply account request Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 80 ++++++++++++-------- 1 file changed, 48 insertions(+), 32 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..5e6cc58 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,35 +12,38 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +apply_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, apply_info) mail_content = Mail.read(email_file) + apply_info['my_email'] = mail_content.from[0] + apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + apply_info end options = OptionParser.new do |opts| @@ -52,15 +55,17 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + apply_info['my_email'] = email_address + # when apply account with email address, will get no user name + apply_info['my_name'] = '' end opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + apply_info['my_ssh_pubkey'] = File.read(pub_key_file) end opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + init_info(email_file, apply_info) end opts.on_tail('-h|--help', 'show this message') do @@ -71,10 +76,10 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: @@ -82,10 +87,10 @@ def build_message(email, acct_infos) You can use the following command to login the jumper server: login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} account password: - #{acct_infos['passwd']} + #{account_info['my_password']} regards compass-ci @@ -94,26 +99,37 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(apply_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(apply_info) message = "No email address specified\n" message += "use -e email_address add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? - - acct_info = account_info($apply_info['my_ssh_pubkey']) - - message = build_message($apply_info['my_email'], acct_info) + raise message if apply_info['my_email'].nil? + + account_info = apply_account(apply_info) + my_info = { + 'my_email' => apply_info['my_email'], + 'my_name' => apply_info['my_name'], + # there is no need to add a commit url for this tool + # the es has the key: my_commit_url + 'my_commit_url' => '', + 'my_login_name' => account_info['my_login_name'], + 'my_uuid' => apply_info['my_uuid'] + } + + store_account_info(my_info) + message = build_message(apply_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(apply_info) -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container/assign-account: get_account_info
by Luan Shengde 04 Nov '20

04 Nov '20

add function: config default yaml file [why]: easier for user to config the default yaml file [how]: write received user infos to the defaults yaml file: ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/get_account_info.rb | 51 +++++++++++++------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..b2d2474 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,35 +95,48 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) + setup_authorized_key(login_name, pub_key) unless pub_key.nil? + setup_default_yaml(login_name) + return jumper_account_info end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def setup_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + end + + def setup_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] container/assign-account: assign-account.rb
by Luan Shengde 04 Nov '20

04 Nov '20

disable assigning account for user if there is no: - my_email - my_name - my_uuid [why]: my_email, my_name, my_uuid is required when initialize the default config file [how]: check if the parsed data has keys: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/assign-account.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..af10524 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,17 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + check_to_assign_account(data) +end + +def check_to_assign_account(data) + lack_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lack_info.join(', ')}." + raise error_message unless lack_info.empty? + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

1 0

[PATCH v3 lkp-tests 3/3] tests/iozone: modify the way of parsing parameter
by Lu Kaiyi 04 Nov '20

04 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] modify the way of parsing parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 65 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 37 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..b85bc4a9 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,51 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write_rewrite") args+=" -i 0" + ;; + "read_reread") args+=" -i 1" + ;; + "random_read_write") args+=" -i 2" + ;; + "read_backwards") args+=" -i 3" + ;; + "rewrite_record") args+=" -i 4" + ;; + "stride_read") args+=" -i 5" + ;; + "fwrite_refwrite") args+=" -i 6" + ;; + "fread_refread") args+=" -i 7" + ;; + "random_mix") args+=" -i 8" + ;; + "pwrite_repwrite") args+=" -i 9" + ;; + "pread_repread") args+=" -i 10" + ;; + "pwritev_repwritev") args+=" -i 11" + ;; + "preadv_repreadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

2 1

[PATCH v2 lab-z9] lab-z9/host: fix taishan200-2280-2s48p-256g--a95 to taishan200-2280-2s48p-512g--a95
by Zhang Yale 04 Nov '20

04 Nov '20

[Why] The memory of taishan200-2280-2s48p-256g--a95 is 512g, so need to fix 256g to 512g. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- ...an200-2280-2s48p-256g--a95 => taishan200-2280-2s48p-512g--a95} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename hosts/{taishan200-2280-2s48p-256g--a95 => taishan200-2280-2s48p-512g--a95} (100%) diff --git a/hosts/taishan200-2280-2s48p-256g--a95 b/hosts/taishan200-2280-2s48p-512g--a95 similarity index 100% rename from hosts/taishan200-2280-2s48p-256g--a95 rename to hosts/taishan200-2280-2s48p-512g--a95 -- 2.23.0

1 0

Re: [PATCH v3 compass-ci 3/3] container/assign-account: answerback-email.rb
by Zhang Yuhang 04 Nov '20

04 Nov '20

Comment message not clear to specify patch info. Reply-To: In-Reply-To: <20201103075716.1022070-3-shdluan(a)163.com> On Tue, Nov 03, 2020 at 03:57:16PM +0800, Luan Shengde wrote: >1. disable use golbal variable apply_info spell mistake: global > [why] > Style/GlobalVars: Do not introduce global variables. > [how] > use local variables instead of global variables > >2. enable transfer user info when execute apply account command > [why] > when applying account, the assign-account service will write the > my info to user's default config file > ~/.config/compass-ci/defaults/account.yaml > my info: > - my_email > - my_name > - my_uuid > [how] > transfer the user info along with the pub_key when sending apply > account request. Suggest to split it into two patch. >Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> >--- > container/assign-account/answerback-email.rb | 80 ++++++++++++-------- > 1 file changed, 48 insertions(+), 32 deletions(-) > >diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb >index bb8e809..5e6cc58 100755 >--- a/container/assign-account/answerback-email.rb >+++ b/container/assign-account/answerback-email.rb >@@ -12,35 +12,38 @@ require 'mail' > require 'set' > require 'optparse' > require_relative '../defconfig' >+require_relative '../../lib/es_client' > > names = Set.new %w[ > JUMPER_HOST > JUMPER_PORT >- SEND_MAIL_HOST_INTERNET >- SEND_MAIL_PORT_INTERNET >+ SEND_MAIL_HOST >+ SEND_MAIL_PORT > ] > > defaults = relevant_defaults(names) > > JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' > JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 >-SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' >-SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 >+SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' >+SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 > >-$apply_info = { >+apply_info = { > 'my_email' => nil, >+ 'my_name' => nil, >+ 'my_uuid' => %x(uuidgen).chomp, > 'my_ssh_pubkey' => nil > } > >-def init_info(email_file) >+def init_info(email_file, apply_info) > mail_content = Mail.read(email_file) >+ apply_info['my_email'] = mail_content.from[0] >+ apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') >+ apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' >+ mail_content.part[1].body.decoded >+ end > >- $apply_info['my_email'] = mail_content.from[0] >- $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' >- mail_content.part[1].body.decoded.gsub(/\r|\n/, '') >- end >- >- $apply_info >+ apply_info > end > > options = OptionParser.new do |opts| >@@ -52,15 +55,17 @@ options = OptionParser.new do |opts| > opts.separator 'options:' > > opts.on('-e|--email email_address', 'appoint email address') do |email_address| >- $apply_info['my_email'] = email_address >+ apply_info['my_email'] = email_address >+ # when apply account with email address, will get no user name >+ apply_info['my_name'] = '' > end > > opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| >- $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) >+ apply_info['my_ssh_pubkey'] = File.read(pub_key_file) > end > > opts.on('-f|--raw-email email_file', 'email file') do |email_file| >- init_info(email_file) >+ init_info(email_file, apply_info) > end > > opts.on_tail('-h|--help', 'show this message') do >@@ -71,10 +76,10 @@ end > > options.parse!(ARGV) > >-def build_message(email, acct_infos) >+def build_message(email, account_info) > message = <<~EMAIL_MESSAGE > To: #{email} >- Subject: jumper account is ready >+ Subject: [compass-ci] jumper account is ready > > Dear user: > >@@ -82,10 +87,10 @@ def build_message(email, acct_infos) > You can use the following command to login the jumper server: > > login command: >- ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} >+ ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} > > account password: >- #{acct_infos['passwd']} >+ #{account_info['my_password']} > > regards > compass-ci >@@ -94,26 +99,37 @@ def build_message(email, acct_infos) > return message > end > >-def account_info(pub_key) >- account_info_str = if pub_key.nil? >- %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') >- else >- %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") >- end >+def apply_account(apply_info) >+ account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') > JSON.parse account_info_str > end > >-def send_account >+def send_account(apply_info) The function name so wired. > message = "No email address specified\n" > message += "use -e email_address add a email address\n" > message += 'or use -f to add a email file' >- raise message if $apply_info['my_email'].nil? >- >- acct_info = account_info($apply_info['my_ssh_pubkey']) >- >- message = build_message($apply_info['my_email'], acct_info) >+ raise message if apply_info['my_email'].nil? Could we do like this? raise message unless apply_info['my_email'] Thanks, Zhang Yuhang >+ >+ account_info = apply_account(apply_info) >+ my_info = { >+ 'my_email' => apply_info['my_email'], >+ 'my_name' => apply_info['my_name'], >+ # there is no need to add a commit url for this tool >+ # the es has the key: my_commit_url >+ 'my_commit_url' => '', >+ 'my_login_name' => account_info['my_login_name'], >+ 'my_uuid' => apply_info['my_uuid'] >+ } >+ >+ store_account_info(my_info) >+ message = build_message(apply_info['my_email'], account_info) > > %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") > end > >-send_account >+def store_account_info(my_info) >+ es = ESClient.new(index: 'accounts') >+ es.put_source_by_id(my_info['my_email'], my_info) >+end >+ >+send_account(apply_info) >-- >2.23.0 >

1 0

[PATCH lab-z9] lab-z9/hosts: rename taishan200-2280-2s48p-256g--a95 to taishan200-2280-2s48p-512g--a95
by Zhang Yale 04 Nov '20

04 Nov '20

[Why] The memory of taishan200-2280-2s48p-256g--a95 is 512g, so need to fix the host name to 512g. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- hosts/taishan200-2280-2s48p-512g--a95 | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-512g--a95 diff --git a/hosts/taishan200-2280-2s48p-512g--a95 b/hosts/taishan200-2280-2s48p-512g--a95 new file mode 100644 index 0000000..213d10f --- /dev/null +++ b/hosts/taishan200-2280-2s48p-512g--a95 @@ -0,0 +1,20 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 512G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039988792e31 + - /dev/disk/by-id/scsi-35000039988793705 + - /dev/disk/by-id/scsi-35000039988792aa5 + - /dev/disk/by-id/scsi-350000399887932fd +mac_addr: + - cc:05:77:fd:fb:61 + - cc:05:77:fd:fb:62 + - cc:05:77:fd:fb:63 + - cc:05:77:fd:fb:64 + - cc:05:77:fd:fb:65 + - cc:05:77:fd:fb:66 + - cc:05:77:fd:fb:67 + - cc:05:77:fd:fb:68 +model_name: Kunpeng-920 -- 2.23.0

1 1

[PATCH compass-ci 1/2] compare values by each metrics
by Lu Weitao 04 Nov '20

04 Nov '20

compare values by each metrics based on groups matrices, and format compare result as echart data_set background: For support compare with user-defined template feature, the work-flow of user-defined template feature: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: compare_values by each metrics ---> format/show results Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 107 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 078028a..7e6dfb1 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -399,8 +399,115 @@ def compare_group_matrices(group_matrices, options) result_str end +# compare group_matrices with each dimension(matrix) of each group, +# and reorganize the result as group-metric-compare-values +# +# Input: groups_matrices +# { +# group_key_1 => { +# dimension_1 => matrix_1, (openeuler 20.03) +# dimension_2 => matrix_2, (openeuler 20.09) +# dimension_3 => matrix_3, (centos 7.6) +# }, +# group_key_2 => {...} +# } +# +# output: compare_metrics_values +# { +# group_key_1 => { +# metric_1 => { +# 'average' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'standard_deviation' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'change' => { +# 'dimension_2 vs dimension_1' => xxx +# 'dimension_3 vs dimension_1' => xxx +# 'dimension_3 vs dimension_2' => xxx +# } +# }, +# metric_2 => {...} +# } +# } +def compare_metrics_values(groups_matrices) + metrics_compare_values = {} + groups_matrices.each do |group_key, dimensions| + metrics_compare_values[group_key] = get_metric_values(dimensions) + end + metrics_compare_values +end + +def get_metric_values(dimensions) + metrics_values = {} + dimensions.each do |dim, matrix| + matrix.each do |metric, values| + metrics_values[metric] ||= {} + metrics_values[metric]['average'] ||= {} + metrics_values[metric]['standard_deviation'] ||= {} + metric_value = get_values(values, true) + metrics_values[metric]['average'][dim] = metric_value[:average] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] + end + end + get_metric_change(metrics_values) + metrics_values +end + +def get_metric_change(metrics_values) + metrics_values.each do |metric, values| + metrics_values[metric]['change'] = {} + next if values['average'].size < 2 + + dimension_list = values['average'].keys + change_titles = get_change_titles(dimension_list) + change_titles.each do |pk| + change = get_compare_value(values['average'][pk[0]], values['average'][pk[1]], true) + values['change'] = { "#{pk[1]} vs #{pk[0]}" => change } + end + end +end + +# create all dimension combinations as change_title +# input: dimension_list +# eg: ['openeuler 20.03', 'debian sid', 'centos 7.6'] +# return: Array(base_dimension: String, pk_dimension: String) +# [ +# ['openeuler 20.03', 'debian sid'], +# ['openeuler 20.03', 'centos 7.6'], +# ['debian sid', 'centos 7.6'] +# ] +def get_change_titles(dimension_list) + dims = [] + dimension_list_size = dimension_list.size + (1..dimension_list_size - 1).each do |i| + (i..dimension_list_size - 1).each do |j| + dims << [dimension_list[i - 1], dimension_list[j]] + end + end + dims +end + +def show_chart_compare_result(chart_result) + print JSON.pretty_generate(chart_result) +end + # Format Fields +def format_for_chart(metrics_compare_values, template_params) + chart_result = {} + chart_result['title'] = template_params['title'] || nil + chart_result['unit'] = template_params['unit'] || nil + chart_result['x_name'] = template_params['x_params'].join('|') || nil + chart_result['tables'] = metrics_compare_values + chart_result +end + def format_fails_runs(fails, runs) fails_width = (SUB_LONG_COLUMN_WIDTH * FAILS_PROPORTION).to_i runs_width = SUB_LONG_COLUMN_WIDTH - fails_width - 1 -- 2.23.0

2 7

[PATCH v3 lkp-tests 2/3] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 04 Nov '20

04 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..f9ab305f 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write_rewrite,read_reread,random_read_write block_size: - 64k -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/3] jobs/iozone-bs.yaml: fix mapper_parsing_exception
by Lu Kaiyi 04 Nov '20

04 Nov '20

[why] when key and value of fs: xfs in same line for iozone-bs.yaml, submit job will cause some error occurred like below: submit /home/lukaiyi/lkp-tests/jobs/iozone-bs.yaml failed, got job_id=z9.144590, error: [{"type" => "mapper_parsing_exception", "reason" => "failed to parse [pp.fs.fs]"}] [how] change key and value of fs: xfs to different line, same to iosched: kyber Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index 868979fc..e2cd9f48 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -14,6 +14,11 @@ block_size: - 16m disk: 1HDD -fs: xfs -iosched: kyber + +fs: +- xfs + +iosched: +- kyber + iozone: -- 2.23.0

1 0

[PATCH compass-ci] dnsmasq: fix script will continue to execute when cp failed
by Yu Chuan 04 Nov '20

04 Nov '20

[Why] 'load custom dnsmasq.conf' step will cp custom config to WORKSPACE, but when a user without permission execute this step, cp will failed. So add 'set -e' at the beginning of the script to solve this problem. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/dnsmasq/start | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/dnsmasq/start b/container/dnsmasq/start index ff4247046eb1..b2eefbd89c16 100755 --- a/container/dnsmasq/start +++ b/container/dnsmasq/start @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +set -e + . $CCI_SRC/container/defconfig.sh docker_rm dnsmasq -- 2.23.0

3 5

[PATCH compass-ci 2/2] add template compare mode
by Lu Weitao 03 Nov '20

03 Nov '20

add template compare mode in sbin/compare(command-line tool) by: - load and parse compare.yaml - calls lib/es_query.rb, query jobs data set from ES - calls lib/matrix.rb, convert jobs data set to group matrices - calls lib/compare_matrixes.rb, use groups matrices to compare background: For support compare with user-defined template feature, the work-flow of user-defined template feature: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results example: compare -t/--template compare_template.yaml, the yaml like: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s output:{ "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "bs|test_size", "tables": { "4k|1G": { "fio.write_iops": { "average": { "openeuler 20.03": 308298.5524761111, "debian sid": 220289.07563 }, "standard_deviation": { "openeuler 20.03": null, "debian sid": null }, "change": { "debian sid vs openeuler 20.03": -28.5 } } }, "16k|1G": {...}, ... } } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++ sbin/compare | 16 +++++++++++++--- 2 files changed, 65 insertions(+), 3 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 2837edc..26e6b49 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -6,6 +6,7 @@ require_relative './es_query.rb' require_relative './matrix2.rb' require_relative './compare_matrixes.rb' require_relative './constants.rb' +require 'yaml' # ------------------------------------------------------------------------------------------- # compare_matrices_list @@ -66,3 +67,54 @@ def create_groups_matrices_list(conditions, dims) query_results = es.multi_field_query(conditions) combine_group_query_data(query_results, dims) end + +# ------------------------------------------------------------------------------------------- +# compare with user defined compare_template.yaml +# the data change flow: +# load_compare_template.yaml --> query_results(ES) ---> auto_group_jobs_list +# ---> groups_matrices ---> compare_values ---> format/show_results +# + +def compare_by_template(template) + template_params = load_template(template) + groups_matrices = create_groups_matrices(template_params) + compare_values = compare_metrics_values(groups_matrices) + chart_results = format_for_chart(compare_values, template_params) + show_chart_compare_result(chart_results) +end + +def load_template(template) + unless File.file?(template) + warn 'template does not exist' + exit + end + YAML.safe_load(File.open(template)) +end + +# according to user's template to create groups_matrices +# input: template_params: Hash +# eg: +# { +# "compare_metrics"=>["fio.write_iops", "fio.read_iops"], +# "filter"=>[ +# {"suite"=>["fio-bisic"]}, +# {"os_arch"=>["aarch_64"]} +# ], +# "compare_dimensions"=>[ +# {"os"=>"openeuler", "os_version"=>20.03}, +# {"os"=>"centos", "os_version"=>7.6} +# ], +# "x_params"=>["block_size", "package_size"], +# "title"=>"Hackbench Performance Testing", +# "unit"=>"KB/s" +# } +def create_groups_matrices(template_params) + es = ESQuery.new + query_results = es.multi_field_query(template_params['filter']) + combine_group_query_data( + query_results, + template_params['x_params'], + template_params['compare_dimensions'], + template_params['compare_metrics'] + ) +end diff --git a/sbin/compare b/sbin/compare index cf3980c..b1bc49a 100755 --- a/sbin/compare +++ b/sbin/compare @@ -6,6 +6,7 @@ # Usage: # compare "conditions_1" "conditions_2" ... -c "common_conditions" # compare "conditions" -d "dimensions" +# compare -t template.yaml # Eg: # compare "id=6000,6001" "id=7000,7001" # compare "commit=a12d232e" "commit=b3bacc31" @@ -21,6 +22,7 @@ is_group = false dimensions = nil colorful = nil options = {} +template = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: compare "conditions" ... [option]' @@ -45,6 +47,10 @@ opt_parser = OptionParser.new do |opts| colorful = color end + opts.on('-t', '--template template', 'compare with user-defined template') do |t| + template = t + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -60,8 +66,12 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful -if is_group - compare_group(argv, dimensions, options) +if template + compare_by_template(template) else - compare_matrices_list(argv, common_conditions, options) + if is_group + compare_group(argv, dimensions, options) + else + compare_matrices_list(argv, common_conditions, options) + end end -- 2.23.0

2 2

[PATCH lkp-tests] job.rb: fix add second program to pp
by Wei Jihui 03 Nov '20

03 Nov '20

[Why] if we add new field in job yaml like this: on_fail: sleep: 300 if will do not add sleep to pp [How] when add sleep to pp, we check sleep's parent hash, not all job. it will find sleep in on_fail'keys. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..50323d31 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -576,10 +576,10 @@ class Job @job[comment_to_symbol('auto generated by add_pp()')] = nil @job["pp"] = Hash.new() init_program_options() - for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, p_args| - k = hash_key_re_string(@job, p_n, @job['pp'].keys) - if k && !(a)job[k].is_a?(Hash) - @job['pp'][k] = @job[k] + for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, _p_args| + k = hash_key_re_string(h, p_n, @job['pp'].keys) + if k && !h[k].is_a?(Hash) && !h[k].nil? + @job['pp'][k] = h[k] next end options_hash = @referenced_programs[p_n] -- 2.23.0

3 2

[PATCH v2 compass-ci] specifies the queue from which the VM obtains tasks
by Xiao Shenwei 03 Nov '20

03 Nov '20

[why] specify the VM that need to consume the queues one job will added one queue, but one VM can consume job from multi queues. [how] to solve this problem, the scheduler should obtains which queues should be request. solution-1: register mac hostname and queues relation before: mac2host change to: mac2host and host2queues then scheduler can get queues based on mac solution-2: /boot.ipxe/mac/${mac}?queues=xxx,yyy our HW and PXE-VM use /tftpboot/boot.ipxe to send request, unable to specify parameter, so solution-1 may be better. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/my-qemu.sh | 4 +++- providers/qemu.sh | 17 +++++++++++++---- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 954d3a0..8845a34 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -4,6 +4,8 @@ [[ $tbox_group ]] || tbox_group=vm-2p8g -export hostname=$tbox_group--$USER-$$ +export hostname=$tbox_group.$USER-$$ +# specify which queues will be request, use " " to separate more than 2 values +export queues="$tbox_group~$USER" $CCI_SRC/providers/qemu.sh diff --git a/providers/qemu.sh b/providers/qemu.sh index d97fd85..6c61f86 100755 --- a/providers/qemu.sh +++ b/providers/qemu.sh @@ -9,6 +9,7 @@ load_cci_defaults : ${hostname:="vm-1p1g-1"} +: ${queues:="vm-1p1g.$(arch)"} # unicast prefix: x2, x6, xA, xE export mac=$(echo $hostname | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/0a-\1-\2-\3-\4-\5/') echo hostname: $hostname @@ -17,14 +18,22 @@ echo $mac > mac echo "arp -n | grep ${mac//-/:}" > ip.sh chmod +x ip.sh -curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}" +set_host_info() +{ + # use "," replace " " + local api_queues=$(echo $queues | sed -r 's/ +/,/g') + curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}" + curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host2queues?host=${hostname}&queues=${api_queues}" +} +set_host_info -del_host_mac() +del_host_info() { curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/del_host_mac?mac=${mac}" > /dev/null 2>&1 + curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/del_host2queues?host=${hostname}" > /dev/null 2>&1 } -trap del_host_mac EXIT +trap del_host_info EXIT ( if [[ $hostname =~ ^(.*)-[0-9]+$ ]]; then @@ -33,7 +42,7 @@ trap del_host_mac EXIT tbox_group=$hostname fi - host=${tbox_group%%--*} + host=${tbox_group%.*} create_yaml_variables "$LKP_SRC/hosts/${host}" -- 2.23.0

3 4

help star
by Wu Fengguang 03 Nov '20

03 Nov '20

Xueliang is our code star, thanks Xueliang! wfg@crystal /c/todo% bin/help-star people/*/credit/2020-10-30.md 24.2 Cao Xueliang 18.2 Wu Fengguang 18.0 Yu Chuan 16.9 Wei Jihui 15.0 Wu Zhende 13.6 Liu Shaofei 9.8 Liu Yinsi 9.7 Xu Xijian 8.9 Wang Yong 8.8 Lu Weitao 8.7 Zhou Shengrui 8.1 Tong Qunfeng 7.7 Zhang Yuhang 7.4 Bai Jing 6.2 Cui Yili 6.0 Xiao Shenwei 5.6 Luan Shengde 5.6 Shi Zhichao 5.2 Wang Chenglong 5.1 Zhang Dewan 4.8 Zhang Yale 4.5 Zhang Yu 4.0 Lin Jiaxin 3.9 Sun Yukui 3.5 Li Ping 3.4 Ren Wen 2.3 Du Kaitian 2.3 Hu Xuejiao 1.5 Li Yuanchao 1.1 Lu Kaiyi 0.0 Wang Shuai 0.0 Cui Xucui

2 2

[PATCH compass-ci] os-cifs: keep the container name consistent with the name in code
by Yu Chuan 03 Nov '20

03 Nov '20

[Why] Current container name is 'samba', it is neither easy to remember nor easy to search. So change the container name from 'samba' to 'os-cifs'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/os-cifs/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/os-cifs/start b/container/os-cifs/start index cd33e207c709..6cc08c629fbd 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -17,7 +17,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/initrd:/srv/initrd -v /srv/result:/srv/result - --name samba + --name os-cifs --restart=always alpine/samba ) -- 2.23.0

2 1

[PATCH compass-ci] taskqueue: close connection after API call
by Cao Xueliang 03 Nov '20

03 Nov '20

[why] most client call to taskqueue do not need keep-alive. when API call is finished, close the connection. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/taskqueue/taskqueue.cr | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/taskqueue/taskqueue.cr b/src/taskqueue/taskqueue.cr index 13f48db..f41cfa8 100644 --- a/src/taskqueue/taskqueue.cr +++ b/src/taskqueue/taskqueue.cr @@ -42,6 +42,7 @@ class TaskQueue # # response: TaskQueue(a)v0.0.1 is alive. get "/" do |env| + env.response.headers["Connection"] = "close" response = "TaskQueue@v#{VERSION} is alive." debug_message(env, response, Time.utc) @@ -60,6 +61,7 @@ class TaskQueue # 400 "Missing parameter <queue>" # 400 "Missing http body" post "/add" do |env| + env.response.headers["Connection"] = "close" response = queue_respond_add(env) debug_message(env, response, Time.utc) response if env.response.status_code == 200 @@ -73,6 +75,7 @@ class TaskQueue # 201 ## when there has no task in queue (scheduler/$tbox_group) # 400 "Missing parameter <queue>" put "/consume" do |env| + env.response.headers["Connection"] = "close" response = queue_respond_consume(env) debug_message(env, response, Time.utc) response if env.response.status_code == 200 @@ -86,6 +89,7 @@ class TaskQueue # 400 "Missing parameter <from|to|id>" # 409 "Can not find id <$id> in queue <scheduler/$tbox_group>" put "/hand_over" do |env| + env.response.headers["Connection"] = "close" response = queue_respond_hand_over(env) debug_message(env, response, Time.utc) nil @@ -99,6 +103,7 @@ class TaskQueue # 400 "Missing parameter <queue|id>" # 409 "Can not find id <$id> in queue <scheduler/$tbox_group>" put "/delete" do |env| + env.response.headers["Connection"] = "close" response = queue_respond_delete(env) debug_message(env, response, Time.utc) nil @@ -114,6 +119,7 @@ class TaskQueue # 400 "Missing parameter <queue>" # 413 "Query results too large keys" get "/keys" do |env| + env.response.headers["Connection"] = "close" response = queue_respond_keys(env) # debug_message(env, response, Time.utc) # maybe too large response.to_json unless env.response.status_code == 201 -- 2.23.0

1 0

[PATCH compass-ci] lkp-initrd: add step: rsync the result lkp.cgz if needed
by Yu Chuan 03 Nov '20

03 Nov '20

[Why] About this container: - Everyone will run this container. - Docker permission is required to run this script. Generally, the production env will control docker permission, and will not give docker permission to everyone. So add the rsync step to allow the execution of local rsync script to perform result lkp.cgz synchronization. > local rsync script path: /usr/local/bin/rsync-lkp-cgz Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/lkp-initrd/run | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/container/lkp-initrd/run b/container/lkp-initrd/run index 522f5098ddc2..e2bff72ab83b 100755 --- a/container/lkp-initrd/run +++ b/container/lkp-initrd/run @@ -24,3 +24,8 @@ cmd=( "${cmd[@]}" echo "result: /srv/initrd/lkp/${lkp_initrd_user:-latest}/lkp-${ARCH}.cgz" + +# rsync lkp-${ARCH}.cgz if needed +if [ -f "/usr/local/bin/rsync-lkp-cgz" ]; then + . "/usr/local/bin/rsync-lkp-cgz" +fi -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] create matrices-groups according to group jobs
by Lu Weitao 03 Nov '20

03 Nov '20

create matrices-groups according to auto group job_list background: For support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: create groups_matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 05f96dc..a284c98 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -120,3 +120,28 @@ def combine_group_query_data(query_data, dims) end groups end + +# combine_group_query_data: combine query data from ES to group matrix +# input: +# 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() +# 2. params from user's template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# return: group_matrix of Hash(String, Hash(String, matrix)) +def combine_group_query_data(query_data, groups_params, dimensions, metrics) + job_list = query_data['hits']['hits'] + groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) + groups.each do |group_key, dimensions| + dimensions.each do |dimension_key, jobs| + groups[group_key][dimension_key] = create_matrix(jobs) + end + end + groups +end -- 2.23.0

3 4

[PATCH lkp-tests] lib/job.rb: no need puts the a warning
by Li Ping 03 Nov '20

03 Nov '20

[why] we can get hostname from the repo lab-z9 there is no need to check hostname file whether exist under the path lkp-tests/hosts --- lib/job.rb | 7 ------- 1 file changed, 7 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..1948a479 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -341,13 +341,6 @@ class Job if lab_hosts_file hosts_file = lab_hosts_file end - - if File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end end def include_files -- 2.23.0

2 2

[PATCH v3 compass-ci 1/2] auto group jobs with user-defined template
by Lu Weitao 03 Nov '20

03 Nov '20

auto group jobs(query form ES) with user-defined template background: For support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: auto group jobs_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 121 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..15bc2dd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,122 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +# auto_group_by_template: auto group job_list by user's template +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list) + get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + next unless job['stats'] + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + new_job = get_new_job_by_metrics(job, metrics) + next if new_job.empty? + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + if job.key?(param) + group_key_list << job[param] + next + end + job.each_value do |v| + if v.is_a?(Hash) && v.key?(param) + group_key_list << v[param] + break + end + end + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job_by_metrics(job, metrics) + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0

3 4

[PATCH compass-ci 2/8] container/mail-robot: apply-account.rb
by Luan Shengde 03 Nov '20

03 Nov '20

check to assign account invoke ParseApplyAccountEmail to parse commit url and pubkey generate uuid and invoke ApplyAccount to apply account store account info to es send uuid Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/apply-account.rb | 71 +++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 container/mail-robot/apply-account.rb diff --git a/container/mail-robot/apply-account.rb b/container/mail-robot/apply-account.rb new file mode 100755 index 0000000..55245ac --- /dev/null +++ b/container/mail-robot/apply-account.rb @@ -0,0 +1,71 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative "../../lib/es_client.rb" + +# assign uuid/account +class AssignAccount + def initialize(mail_content) + @mail_content = mail_content + @my_email = mail_content.from[0] + @my_name = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + @send_mail_host = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + end + + def check_to_send_account + my_commit_url, pub_key = check_out_commit_url_pub_key + my_uuid, acct_infos = check_to_apply_account(pub_key) + + my_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_commit_url' => my_commit_url, + 'my_login_name' => acct_infos['account'], + 'my_uuid' => my_uuid + } + store_account_info(my_info) + my_info['acct_infos'] = acct_infos + send_my_uuid(my_info) + end + + def check_out_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + my_commit_url, pub_key = parse_apply_account_email.parse_commit_url_pub_key + + return my_commit_url, pub_key + end + + def check_to_apply_account(pub_key) + my_uuid = %x(uuidgen).chomp + apply_account_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_uuid' => my_uuid, + 'my_ssh_pubkey' => pub_key + } + apply_account = ApplyAccount.new(apply_account_info) + acct_infos = apply_account.apply_jumper_account + + return my_uuid, acct_infos + end + + def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) + end + + def send_my_uuid(my_info) + send_uuid_email_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_info' => my_info, + 'error_message' => '' + } + send_uuid = SendMail.new(send_uuid_email_info) + send_uuid.send_uuid_email + end +end -- 2.23.0

2 3

[PATCH compass-ci] monitoring/filter.cr: query value support regularity
by Wu Zhende 03 Nov '20

03 Nov '20

[Why] Enhanced monitoring function. Regular expressions can be used in query's value. query can be {"job_id": "z9.*"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index bc6f901..5ddd547 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,12 +56,20 @@ class Filter def match_query(query : Hash(String, JSON::Any), msg : Hash(String, JSON::Any)) query.each do |key, value| + return false unless msg.has_key?(key) + value = value.as_a - if value.includes?(nil) - return false unless msg.has_key?(key) - else - return false unless value.includes?(msg[key]?) + next if value.includes?(nil) + next if value.includes?(msg[key]?) + + flag = false + value.each do |val| + if msg[key]?.to_s =~ /#{val}/ + flag = true + break + end end + return false unless flag end return true end -- 2.23.0

1 0

[PATCH v9 compass-ci 1/2] job.cr: add a key "kernel_version" for initramfs
by Xu Xijian 03 Nov '20

03 Nov '20

When run job with initramfs, there is a default combination of kernel, modules and headers, and I add a key "kernel_version" to make it optional. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e49da6..6309809 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -87,6 +87,10 @@ class Job kernel_append_root kernel_params docker_image + kernel_version + linux_vmlinuz_path + linux_modules_initrd + linux_headers_initrd ) macro method_missing(call) @@ -135,6 +139,7 @@ class Job set_result_service() set_os_mount() set_depends_initrd() + set_kernel_version() set_initrds_uri() set_kernel_uri() set_kernel_append_root() @@ -293,9 +298,17 @@ class Job return true end + private def set_kernel_version + boot_dir = "#{SRV_OS}/#{os_dir}/boot" + suffix = "-#{kernel_version}" if self["kernel_version"]? + self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") + self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") + self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") + end + private def set_kernel_uri self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{linux_vmlinuz_path}")}" end private def common_initrds @@ -316,6 +329,10 @@ class Job "#{JobHelper.service_path("#{osimage_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{linux_modules_initrd}")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{linux_headers_initrd}")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] container/assign-account: answerback-email.rb
by Luan Shengde 03 Nov '20

03 Nov '20

1. disable use golbal variable apply_info [why] Style/GlobalVars: Do not introduce global variables. [how] use local variables instead of global variables 2. enable transfer user info when execute apply account command [why] when applying account, the assign-account service will write the my info to user's default config file ~/.config/compass-ci/defaults/account.yaml my info: - my_email - my_name - my_uuid [how] transfer the user info along with the pub_key when sending apply account request. Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 80 ++++++++++++-------- 1 file changed, 48 insertions(+), 32 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..5e6cc58 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,35 +12,38 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +apply_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, apply_info) mail_content = Mail.read(email_file) + apply_info['my_email'] = mail_content.from[0] + apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + apply_info end options = OptionParser.new do |opts| @@ -52,15 +55,17 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + apply_info['my_email'] = email_address + # when apply account with email address, will get no user name + apply_info['my_name'] = '' end opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + apply_info['my_ssh_pubkey'] = File.read(pub_key_file) end opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + init_info(email_file, apply_info) end opts.on_tail('-h|--help', 'show this message') do @@ -71,10 +76,10 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: @@ -82,10 +87,10 @@ def build_message(email, acct_infos) You can use the following command to login the jumper server: login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} account password: - #{acct_infos['passwd']} + #{account_info['my_password']} regards compass-ci @@ -94,26 +99,37 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(apply_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(apply_info) message = "No email address specified\n" message += "use -e email_address add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? - - acct_info = account_info($apply_info['my_ssh_pubkey']) - - message = build_message($apply_info['my_email'], acct_info) + raise message if apply_info['my_email'].nil? + + account_info = apply_account(apply_info) + my_info = { + 'my_email' => apply_info['my_email'], + 'my_name' => apply_info['my_name'], + # there is no need to add a commit url for this tool + # the es has the key: my_commit_url + 'my_commit_url' => '', + 'my_login_name' => account_info['my_login_name'], + 'my_uuid' => apply_info['my_uuid'] + } + + store_account_info(my_info) + message = build_message(apply_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(apply_info) -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] container/assign-account: get_account_info
by Luan Shengde 03 Nov '20

03 Nov '20

add new function: config default yaml file [why]: easier for user to config the default yaml file [how]: parse received data and extract the user infos store user infos to defaults yaml file: ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/get_account_info.rb | 50 +++++++++++++------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..51e3e03 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,35 +95,49 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + jumper_host = jumper_info[0].chomp + jumper_port = jumper_info[1].chomp + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_host, 'jumper_port' => jumper_port } - setup_authorized_key(account, pub_key) + setup_authorized_key(login_name, pub_key) unless pub_key.nil? + setup_default_yaml(login_name) + return jumper_account_info end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def setup_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + end + + def setup_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] container/assign-account: assign-account.rb
by Luan Shengde 03 Nov '20

03 Nov '20

disable assigning account for user if there is no: - my_email - my_name - my_uuid [why]: my_email, my_name, my_uuid is required when initialize the default config file [how] check if the parsed data has keys: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/assign-account.rb | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..9481068 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,22 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + check_to_assign_account(data) +end + +def check_to_assign_account(data) + error_message = 'lack of my infos: my_email' + raise error_message unless data.key? 'my_email' + + error_message = 'lack of my infos: my_name' + raise error_message unless data.key? 'my_name' + + error_message = 'lack of my infos: my_uuid' + raise error_message unless data.key? 'my_uuid' + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

1 0

[PATCH compass-ci] taskqueue_api: each interface uses its own client to request taskqueue
by Cao Xueliang 03 Nov '20

03 Nov '20

[why] Different interfaces use one client may get incorrect data. So, each interface uses its own client to request taskqueue and close it when finish the request. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/taskqueue_api.cr | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/src/lib/taskqueue_api.cr b/src/lib/taskqueue_api.cr index a8bbf16..f447266 100644 --- a/src/lib/taskqueue_api.cr +++ b/src/lib/taskqueue_api.cr @@ -17,20 +17,23 @@ end class TaskQueueAPI def initialize - port = ENV.has_key?("TASKQUEUE_PORT") ? ENV["TASKQUEUE_PORT"].to_i32 : 3060 - host = ENV.has_key?("TASKQUEUE_HOST") ? ENV["TASKQUEUE_HOST"] : "172.17.0.1" - @client = HTTP::Client.new(host, port: port) + @port = ENV.has_key?("TASKQUEUE_PORT") ? ENV["TASKQUEUE_PORT"].to_i32 : 3060 + @host = ENV.has_key?("TASKQUEUE_HOST") ? ENV["TASKQUEUE_HOST"] : "172.17.0.1" end def add_task(service_queue_path : String, task : JSON::Any) params = HTTP::Params.encode({"queue" => service_queue_path}) - response = loop_till_connectable { @client.post("/add?" + params, body: task.to_json) } + client = HTTP::Client.new(@host, port: @port) + response = loop_till_connectable { client.post("/add?" + params, body: task.to_json) } + client.close() arrange_response(response) end def query_keys(service_key_with_wild_char : String) params = HTTP::Params.encode({"queue" => service_key_with_wild_char}) - response = loop_till_connectable { @client.get("/keys?" + params) } + client = HTTP::Client.new(@host, port: @port) + response = loop_till_connectable { client.get("/keys?" + params) } + client.close() arrange_response(response) end @@ -50,7 +53,9 @@ class TaskQueueAPI end private def response_put_api(cmd : String, params : String) - response = loop_till_connectable { @client.put("/#{cmd}?" + params) } + client = HTTP::Client.new(@host, port: @port) + response = loop_till_connectable { client.put("/#{cmd}?" + params) } + client.close() arrange_response(response) end -- 2.23.0

1 0

[PATCH lkp-tests] sbin/monitor: fix JSON.parse error
by Wu Zhende 03 Nov '20

03 Nov '20

[Why] query's value has three types. Can be "xxx", nil, "[xxx]". Only "[xxx]" can do JSON.parse. [Error1] Traceback (most recent call last): 6: from /c/lkp-tests/sbin/monitor:73:in `<main>' 5: from /home/code/lkp-tests/lib/monitor.rb:103:in `run' 4: from /home/code/lkp-tests/lib/monitor.rb:103:in `each' 3: from /home/code/lkp-tests/lib/monitor.rb:104:in `block in run' 2: from /usr/share/ruby/json/common.rb:156:in `parse' 1: from /usr/share/ruby/json/common.rb:156:in `new' /usr/share/ruby/json/common.rb:156:in `initialize': no implicit conversion of nil into String (TypeError) [Error2] Traceback (most recent call last): 6: from /c/lkp-tests/sbin/monitor:73:in `<main>' 5: from /home/code/lkp-tests/lib/monitor.rb:103:in `run' 4: from /home/code/lkp-tests/lib/monitor.rb:103:in `each' 3: from /home/code/lkp-tests/lib/monitor.rb:104:in `block in run' 2: from /usr/share/ruby/json/common.rb:156:in `parse' 1: from /usr/share/ruby/json/common.rb:156:in `new' /usr/share/ruby/json/common.rb:156:in `initialize': no implicit conversion of nil into String (TypeError) Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/monitor.rb b/lib/monitor.rb index e0098390..56283e10 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -102,6 +102,7 @@ class Monitor @query.each do |k, v| @query[k] = JSON.parse(v) + rescue end query = @query.to_json puts "query=>#{query}" -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 03 Nov '20

03 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..f9ab305f 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write_rewrite,read_reread,random_read_write block_size: - 64k -- 2.23.0

2 3

[PATCH v7 compass-ci 2/2] kernel_version.md: explain key "kernel_version"
by Xu Xijian 03 Nov '20

03 Nov '20

[why] Explain the meaning of new key "kernel_version" for scheduler, including what's the mapping between its typical values and actual files in the disk and how to set it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 doc/job/kernel_version.md diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md new file mode 100644 index 0000000..d76a3bd --- /dev/null +++ b/doc/job/kernel_version.md @@ -0,0 +1,18 @@ +# kernel_version + +Meaning: +- Every os has its kernel, however an os can start with different kernels according to different need. +- kernel_version is a key for users to specify a kernel version. +- If kernel_version is not given by users, it will use the default one. + +Related files: +- In initramfs boot process, every kernel version is related with a vmlinuz, module and headers. +- Files like below under $boot_dir, an example $boot_dir can be "/srv/os/openeuler/aarch64/20.03/boot". +├── headers-4.19.90-2003.cgz +├── headers.cgz -> headers-4.19.90-2003.cgz +├── modules-4.19.90-2003.cgz +├── modules.cgz -> modules-4.19.90-2003.cgz +├── vmlinuz-4.19.90-2003 + +Usage example: +- submit iperf.yaml testbox=vm-hi1620-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 -- 2.23.0

2 3

[PATCH compass-ci] specifies the queue from which the VM obtains tasks
by Xiao Shenwei 03 Nov '20

03 Nov '20

[why] specify the VM that need to consume the queues one job will added one queue, but one VM can consume job from multi queues. [how] to solve this problem, the scheduler should obtains which queues should be request. solution-1: register mac hostname and queues relation before: mac2host change to: mac2host and host2queues then scheduler can get queues based on mac solution-2: /boot.ipxe/mac/${mac}?queues=xxx,yyy our HW and PXE-VM use /tftpboot/boot.ipxe to send request, unable to specify parameter, so solution-1 may be better. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/my-qemu.sh | 4 +++- providers/qemu.sh | 18 ++++++++++++++---- 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 954d3a0..f42a1fd 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -4,6 +4,8 @@ [[ $tbox_group ]] || tbox_group=vm-2p8g -export hostname=$tbox_group--$USER-$$ +export hostname=$tbox_group.$USER-$$ +# specify which queues will be request, use " " to separate more than 2 values +export queues="vm-2p8g~$USER vm-2p8g.aarch64" $CCI_SRC/providers/qemu.sh diff --git a/providers/qemu.sh b/providers/qemu.sh index d97fd85..7e4f074 100755 --- a/providers/qemu.sh +++ b/providers/qemu.sh @@ -9,6 +9,7 @@ load_cci_defaults : ${hostname:="vm-1p1g-1"} +: ${queues:="vm-1p1g.$(arch)"} # unicast prefix: x2, x6, xA, xE export mac=$(echo $hostname | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/0a-\1-\2-\3-\4-\5/') echo hostname: $hostname @@ -17,14 +18,23 @@ echo $mac > mac echo "arp -n | grep ${mac//-/:}" > ip.sh chmod +x ip.sh -curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}" +set_host_info() +{ + # use "," replace " " + queues=$(echo $queues | sed -r 's/ +/,/g') + curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}" + curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host2queues?host=${hostname}&queues=${queues}" + +} +set_host_info -del_host_mac() +del_host_info() { curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/del_host_mac?mac=${mac}" > /dev/null 2>&1 + curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/del_host2queues?host=${hostname}" > /dev/null 2>&1 } -trap del_host_mac EXIT +trap del_host_info EXIT ( if [[ $hostname =~ ^(.*)-[0-9]+$ ]]; then @@ -33,7 +43,7 @@ trap del_host_mac EXIT tbox_group=$hostname fi - host=${tbox_group%%--*} + host=${tbox_group%.*} create_yaml_variables "$LKP_SRC/hosts/${host}" -- 2.23.0

3 4

[PATCH v5 compass-ci] LICENSES: add THIRD PARTY OPEN SOURCE SOFTWARE NOTICE
by Lu Kaiyi 03 Nov '20

03 Nov '20

add THIRD PARTY OPEN SOURCE SOFTWARE NOTICE for references. the directory structure of LICENSES as below: tree . ├── CCBY-4.0 ├── coreutils │ ├── coreutils │ ├── files │ └── GPL-3.0 -> ../GPL-3.0 ├── docker-sshd │ ├── docker-sshd │ ├── files │ └── MIT -> ../MIT ├── dracut │ ├── dracut │ ├── files │ └── GPL-2.0 -> ../GPL-2.0 ├── GPL-2.0 ├── GPL-3.0 ├── kemal │ ├── files │ ├── kemal │ └── MIT -> ../MIT ├── lkp-tests │ ├── files │ ├── GPL-2.0 -> ../GPL-2.0 │ └── lkp-tests ├── MIT ├── mritd.github.io │ ├── files │ ├── MIT -> ../MIT │ └── mritd.github.io ├── MulanPSL-2.0 └── Scout ├── files ├── Scout └── MIT -> ../MIT Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- LICENSES/Scout/MIT | 1 + LICENSES/Scout/Scout | 17 +++++++++++++++++ LICENSES/Scout/files | 1 + LICENSES/coreutils/GPL-3.0 | 1 + LICENSES/coreutils/coreutils | 17 +++++++++++++++++ LICENSES/coreutils/files | 1 + LICENSES/docker-sshd/MIT | 1 + LICENSES/docker-sshd/docker-sshd | 17 +++++++++++++++++ LICENSES/docker-sshd/files | 1 + LICENSES/dracut/GPL-2.0 | 1 + LICENSES/dracut/dracut | 18 ++++++++++++++++++ LICENSES/dracut/files | 1 + LICENSES/kemal/MIT | 1 + LICENSES/kemal/files | 1 + LICENSES/kemal/kemal | 17 +++++++++++++++++ LICENSES/lkp-tests/GPL-2.0 | 1 + LICENSES/lkp-tests/files | 8 ++++++++ LICENSES/lkp-tests/lkp-tests | 18 ++++++++++++++++++ LICENSES/mritd.github.io/MIT | 1 + LICENSES/mritd.github.io/files | 1 + LICENSES/mritd.github.io/mritd.github.io | 17 +++++++++++++++++ 21 files changed, 142 insertions(+) create mode 120000 LICENSES/Scout/MIT create mode 100644 LICENSES/Scout/Scout create mode 100644 LICENSES/Scout/files create mode 120000 LICENSES/coreutils/GPL-3.0 create mode 100644 LICENSES/coreutils/coreutils create mode 100644 LICENSES/coreutils/files create mode 120000 LICENSES/docker-sshd/MIT create mode 100644 LICENSES/docker-sshd/docker-sshd create mode 100644 LICENSES/docker-sshd/files create mode 120000 LICENSES/dracut/GPL-2.0 create mode 100644 LICENSES/dracut/dracut create mode 100644 LICENSES/dracut/files create mode 120000 LICENSES/kemal/MIT create mode 100644 LICENSES/kemal/files create mode 100644 LICENSES/kemal/kemal create mode 120000 LICENSES/lkp-tests/GPL-2.0 create mode 100644 LICENSES/lkp-tests/files create mode 100644 LICENSES/lkp-tests/lkp-tests create mode 120000 LICENSES/mritd.github.io/MIT create mode 100644 LICENSES/mritd.github.io/files create mode 100644 LICENSES/mritd.github.io/mritd.github.io diff --git a/LICENSES/Scout/MIT b/LICENSES/Scout/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/Scout/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/Scout/Scout b/LICENSES/Scout/Scout new file mode 100644 index 0000000..6844406 --- /dev/null +++ b/LICENSES/Scout/Scout @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: Scout db170ce4ff160c4a2c30483e9410df1773cf572c + +Copyright notice: Copyright(c) 2010 Menno van Slooten, http://mennovanslooten.nl/ +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/Scout/files b/LICENSES/Scout/files new file mode 100644 index 0000000..0d1e2e9 --- /dev/null +++ b/LICENSES/Scout/files @@ -0,0 +1 @@ +lib/themes.rb diff --git a/LICENSES/coreutils/GPL-3.0 b/LICENSES/coreutils/GPL-3.0 new file mode 120000 index 0000000..481ee24 --- /dev/null +++ b/LICENSES/coreutils/GPL-3.0 @@ -0,0 +1 @@ +../GPL-3.0 \ No newline at end of file diff --git a/LICENSES/coreutils/coreutils b/LICENSES/coreutils/coreutils new file mode 100644 index 0000000..1250a6f --- /dev/null +++ b/LICENSES/coreutils/coreutils @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: coreutils 6.9.89 + +Copyright notice: Copyright(C) 2007 Free Software Foundation, Inc. <https://fsf.org/> +License: The GPL-3.0 License +License-Text: please refer to file GPL-3.0 diff --git a/LICENSES/coreutils/files b/LICENSES/coreutils/files new file mode 100644 index 0000000..2256719 --- /dev/null +++ b/LICENSES/coreutils/files @@ -0,0 +1 @@ +container/os-nfs/entrypoint.sh diff --git a/LICENSES/docker-sshd/MIT b/LICENSES/docker-sshd/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/docker-sshd/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/docker-sshd/docker-sshd b/LICENSES/docker-sshd/docker-sshd new file mode 100644 index 0000000..603f668 --- /dev/null +++ b/LICENSES/docker-sshd/docker-sshd @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: docker-sshd 729faee00bcf0eff5f80059cd92aa425505820f0 + +Copyright notice: Copyright(c) 2015-2020 Volt Grid Pty Ltd +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/docker-sshd/files b/LICENSES/docker-sshd/files new file mode 100644 index 0000000..3289565 --- /dev/null +++ b/LICENSES/docker-sshd/files @@ -0,0 +1 @@ +container/ssh-r/entry.sh diff --git a/LICENSES/dracut/GPL-2.0 b/LICENSES/dracut/GPL-2.0 new file mode 120000 index 0000000..9824569 --- /dev/null +++ b/LICENSES/dracut/GPL-2.0 @@ -0,0 +1 @@ +../GPL-2.0 \ No newline at end of file diff --git a/LICENSES/dracut/dracut b/LICENSES/dracut/dracut new file mode 100644 index 0000000..f09df60 --- /dev/null +++ b/LICENSES/dracut/dracut @@ -0,0 +1,18 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: dracut RHEL-7.1 + +Copyright notice: Copyright(C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, +Fifth Floor, Boston, MA 02110-1301 USA. +License: The GPL-2.0 License +License-Text: please refer to file GPL-2.0 diff --git a/LICENSES/dracut/files b/LICENSES/dracut/files new file mode 100644 index 0000000..d629bf6 --- /dev/null +++ b/LICENSES/dracut/files @@ -0,0 +1 @@ +container/dracut-initrd/bin/cifs-lib.sh diff --git a/LICENSES/kemal/MIT b/LICENSES/kemal/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/kemal/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/kemal/files b/LICENSES/kemal/files new file mode 100644 index 0000000..de6eda6 --- /dev/null +++ b/LICENSES/kemal/files @@ -0,0 +1 @@ +src/spec/scheduler/boot_spec.cr diff --git a/LICENSES/kemal/kemal b/LICENSES/kemal/kemal new file mode 100644 index 0000000..e08255f --- /dev/null +++ b/LICENSES/kemal/kemal @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: kemal 0.17.0 + +Copyright notice: Copyright(c) 2016 Serdar Doğruyol +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/lkp-tests/GPL-2.0 b/LICENSES/lkp-tests/GPL-2.0 new file mode 120000 index 0000000..9824569 --- /dev/null +++ b/LICENSES/lkp-tests/GPL-2.0 @@ -0,0 +1 @@ +../GPL-2.0 \ No newline at end of file diff --git a/LICENSES/lkp-tests/files b/LICENSES/lkp-tests/files new file mode 100644 index 0000000..df39f15 --- /dev/null +++ b/LICENSES/lkp-tests/files @@ -0,0 +1,8 @@ +sbin/compare +providers/multi-docker +container/open-scheduler/build-depends +user-client/jobs/iperf-pxe.yaml +user-client/jobs/iperf-vm.yaml +src/features/jobs/right_iperf.yaml +lib/matrix2.rb +sbin/es-find diff --git a/LICENSES/lkp-tests/lkp-tests b/LICENSES/lkp-tests/lkp-tests new file mode 100644 index 0000000..3f40d5f --- /dev/null +++ b/LICENSES/lkp-tests/lkp-tests @@ -0,0 +1,18 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: lkp-tests fcb47ccfc50c1f4da388aed9596c0acaac04a917 + +Copyright notice: Copyright(C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, +Fifth Floor, Boston, MA 02110-1301 USA +License: The GPL-2.0 License +License-Text: please refer to file GPL-2.0 diff --git a/LICENSES/mritd.github.io/MIT b/LICENSES/mritd.github.io/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/mritd.github.io/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/mritd.github.io/files b/LICENSES/mritd.github.io/files new file mode 100644 index 0000000..eeaa82a --- /dev/null +++ b/LICENSES/mritd.github.io/files @@ -0,0 +1 @@ +container/registry/config.yml diff --git a/LICENSES/mritd.github.io/mritd.github.io b/LICENSES/mritd.github.io/mritd.github.io new file mode 100644 index 0000000..ee22568 --- /dev/null +++ b/LICENSES/mritd.github.io/mritd.github.io @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: mritd.github.io ed4350e91e9b76078a850ede8302854d3393f493 + +Copyright notice: Copyright(c) 2013-2016 Blackrock Digital LLC. +License: The MIT License +License-Text: please refer to file MIT -- 2.23.0

1 0

[PATCH v8 compass-ci 1/2] kernel_version.md: explain key "kernel_version"
by Xu Xijian 03 Nov '20

03 Nov '20

[why] Explain the meaning of new key "kernel_version" for scheduler, including what's the mapping between its typical values and actual files in the disk and how to set it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 doc/job/kernel_version.md diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md new file mode 100644 index 0000000..d76a3bd --- /dev/null +++ b/doc/job/kernel_version.md @@ -0,0 +1,18 @@ +# kernel_version + +Meaning: +- Every os has its kernel, however an os can start with different kernels according to different need. +- kernel_version is a key for users to specify a kernel version. +- If kernel_version is not given by users, it will use the default one. + +Related files: +- In initramfs boot process, every kernel version is related with a vmlinuz, module and headers. +- Files like below under $boot_dir, an example $boot_dir can be "/srv/os/openeuler/aarch64/20.03/boot". +├── headers-4.19.90-2003.cgz +├── headers.cgz -> headers-4.19.90-2003.cgz +├── modules-4.19.90-2003.cgz +├── modules.cgz -> modules-4.19.90-2003.cgz +├── vmlinuz-4.19.90-2003 + +Usage example: +- submit iperf.yaml testbox=vm-hi1620-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 -- 2.23.0

2 2

[PATCH v4 compass-ci] LICENSES: add THIRD PARTY OPEN SOURCE SOFTWARE NOTICE
by Lu Kaiyi 03 Nov '20

03 Nov '20

add THIRD PARTY OPEN SOURCE SOFTWARE NOTICE for references. the directory structure of LICENSES as below: tree . ├── CCBY-4.0 ├── coreutils │ ├── coreutils │ ├── files │ └── GPL-3.0 -> ../GPL-3.0 ├── docker-sshd │ ├── docker-sshd │ ├── files │ └── MIT -> ../MIT ├── dracut │ ├── dracut │ ├── files │ └── GPL-2.0 -> ../GPL-2.0 ├── GPL-2.0 ├── GPL-3.0 ├── kemal │ ├── files │ ├── kemal │ └── MIT -> ../MIT ├── lkp-tests │ ├── files │ ├── GPL-2.0 -> ../GPL-2.0 │ └── lkp-tests ├── MIT ├── mritd.github.io │ ├── files │ ├── MIT -> ../MIT │ └── mritd.github.io ├── MulanPSL-2.0 └── Scout ├── files ├── Scout └── MIT -> ../MIT Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- LICENSES/Scout/MIT | 1 + LICENSES/Scout/Scout | 17 +++++++++++++++++ LICENSES/Scout/files | 1 + LICENSES/coreutils/GPL-3.0 | 1 + LICENSES/coreutils/coreutils | 17 +++++++++++++++++ LICENSES/coreutils/files | 1 + LICENSES/docker-sshd/MIT | 1 + LICENSES/docker-sshd/docker-sshd | 17 +++++++++++++++++ LICENSES/docker-sshd/files | 1 + LICENSES/dracut/GPL-2.0 | 1 + LICENSES/dracut/dracut | 18 ++++++++++++++++++ LICENSES/dracut/files | 1 + LICENSES/kemal/MIT | 1 + LICENSES/kemal/files | 1 + LICENSES/kemal/kemal | 17 +++++++++++++++++ LICENSES/lkp-tests/GPL-2.0 | 1 + LICENSES/lkp-tests/files | 8 ++++++++ LICENSES/lkp-tests/lkp-tests | 18 ++++++++++++++++++ LICENSES/mritd.github.io/MIT | 1 + LICENSES/mritd.github.io/files | 1 + LICENSES/mritd.github.io/mritd.github.io | 17 +++++++++++++++++ 21 files changed, 142 insertions(+) create mode 120000 LICENSES/Scout/MIT create mode 100644 LICENSES/Scout/Scout create mode 100644 LICENSES/Scout/files create mode 120000 LICENSES/coreutils/GPL-3.0 create mode 100644 LICENSES/coreutils/coreutils create mode 100644 LICENSES/coreutils/files create mode 120000 LICENSES/docker-sshd/MIT create mode 100644 LICENSES/docker-sshd/docker-sshd create mode 100644 LICENSES/docker-sshd/files create mode 120000 LICENSES/dracut/GPL-2.0 create mode 100644 LICENSES/dracut/dracut create mode 100644 LICENSES/dracut/files create mode 120000 LICENSES/kemal/MIT create mode 100644 LICENSES/kemal/files create mode 100644 LICENSES/kemal/kemal create mode 120000 LICENSES/lkp-tests/GPL-2.0 create mode 100644 LICENSES/lkp-tests/files create mode 100644 LICENSES/lkp-tests/lkp-tests create mode 120000 LICENSES/mritd.github.io/MIT create mode 100644 LICENSES/mritd.github.io/files create mode 100644 LICENSES/mritd.github.io/mritd.github.io diff --git a/LICENSES/Scout/MIT b/LICENSES/Scout/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/Scout/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/Scout/Scout b/LICENSES/Scout/Scout new file mode 100644 index 0000000..6844406 --- /dev/null +++ b/LICENSES/Scout/Scout @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: Scout db170ce4ff160c4a2c30483e9410df1773cf572c + +Copyright notice: Copyright(c) 2010 Menno van Slooten, http://mennovanslooten.nl/ +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/Scout/files b/LICENSES/Scout/files new file mode 100644 index 0000000..e120df8 --- /dev/null +++ b/LICENSES/Scout/files @@ -0,0 +1 @@ +/compass-ci/lib/themes.rb diff --git a/LICENSES/coreutils/GPL-3.0 b/LICENSES/coreutils/GPL-3.0 new file mode 120000 index 0000000..481ee24 --- /dev/null +++ b/LICENSES/coreutils/GPL-3.0 @@ -0,0 +1 @@ +../GPL-3.0 \ No newline at end of file diff --git a/LICENSES/coreutils/coreutils b/LICENSES/coreutils/coreutils new file mode 100644 index 0000000..1250a6f --- /dev/null +++ b/LICENSES/coreutils/coreutils @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: coreutils 6.9.89 + +Copyright notice: Copyright(C) 2007 Free Software Foundation, Inc. <https://fsf.org/> +License: The GPL-3.0 License +License-Text: please refer to file GPL-3.0 diff --git a/LICENSES/coreutils/files b/LICENSES/coreutils/files new file mode 100644 index 0000000..8a2b690 --- /dev/null +++ b/LICENSES/coreutils/files @@ -0,0 +1 @@ +/compass-ci/container/os-nfs/entrypoint.sh diff --git a/LICENSES/docker-sshd/MIT b/LICENSES/docker-sshd/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/docker-sshd/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/docker-sshd/docker-sshd b/LICENSES/docker-sshd/docker-sshd new file mode 100644 index 0000000..603f668 --- /dev/null +++ b/LICENSES/docker-sshd/docker-sshd @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: docker-sshd 729faee00bcf0eff5f80059cd92aa425505820f0 + +Copyright notice: Copyright(c) 2015-2020 Volt Grid Pty Ltd +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/docker-sshd/files b/LICENSES/docker-sshd/files new file mode 100644 index 0000000..018684c --- /dev/null +++ b/LICENSES/docker-sshd/files @@ -0,0 +1 @@ +/compass-ci/container/ssh-r/entry.sh diff --git a/LICENSES/dracut/GPL-2.0 b/LICENSES/dracut/GPL-2.0 new file mode 120000 index 0000000..9824569 --- /dev/null +++ b/LICENSES/dracut/GPL-2.0 @@ -0,0 +1 @@ +../GPL-2.0 \ No newline at end of file diff --git a/LICENSES/dracut/dracut b/LICENSES/dracut/dracut new file mode 100644 index 0000000..f09df60 --- /dev/null +++ b/LICENSES/dracut/dracut @@ -0,0 +1,18 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: dracut RHEL-7.1 + +Copyright notice: Copyright(C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, +Fifth Floor, Boston, MA 02110-1301 USA. +License: The GPL-2.0 License +License-Text: please refer to file GPL-2.0 diff --git a/LICENSES/dracut/files b/LICENSES/dracut/files new file mode 100644 index 0000000..78235de --- /dev/null +++ b/LICENSES/dracut/files @@ -0,0 +1 @@ +/compass-ci/container/dracut-initrd/bin/cifs-lib.sh diff --git a/LICENSES/kemal/MIT b/LICENSES/kemal/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/kemal/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/kemal/files b/LICENSES/kemal/files new file mode 100644 index 0000000..71a8cab --- /dev/null +++ b/LICENSES/kemal/files @@ -0,0 +1 @@ +/compass-ci/src/spec/scheduler/boot_spec.cr diff --git a/LICENSES/kemal/kemal b/LICENSES/kemal/kemal new file mode 100644 index 0000000..e08255f --- /dev/null +++ b/LICENSES/kemal/kemal @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: kemal 0.17.0 + +Copyright notice: Copyright(c) 2016 Serdar Doğruyol +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/lkp-tests/GPL-2.0 b/LICENSES/lkp-tests/GPL-2.0 new file mode 120000 index 0000000..9824569 --- /dev/null +++ b/LICENSES/lkp-tests/GPL-2.0 @@ -0,0 +1 @@ +../GPL-2.0 \ No newline at end of file diff --git a/LICENSES/lkp-tests/files b/LICENSES/lkp-tests/files new file mode 100644 index 0000000..fde5c05 --- /dev/null +++ b/LICENSES/lkp-tests/files @@ -0,0 +1,8 @@ +/compass-ci/sbin/compare +/compass-ci/providers/multi-docker +/compass-ci/container/open-scheduler/build-depends +/compass-ci/user-client/jobs/iperf-pxe.yaml +/compass-ci/user-client/jobs/iperf-vm.yaml +/compass-ci/src/features/jobs/right_iperf.yaml +/compass-ci/lib/matrix2.rb +/compass-ci/sbin/es-find diff --git a/LICENSES/lkp-tests/lkp-tests b/LICENSES/lkp-tests/lkp-tests new file mode 100644 index 0000000..3f40d5f --- /dev/null +++ b/LICENSES/lkp-tests/lkp-tests @@ -0,0 +1,18 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: lkp-tests fcb47ccfc50c1f4da388aed9596c0acaac04a917 + +Copyright notice: Copyright(C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, +Fifth Floor, Boston, MA 02110-1301 USA +License: The GPL-2.0 License +License-Text: please refer to file GPL-2.0 diff --git a/LICENSES/mritd.github.io/MIT b/LICENSES/mritd.github.io/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/mritd.github.io/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/mritd.github.io/files b/LICENSES/mritd.github.io/files new file mode 100644 index 0000000..38b67c1 --- /dev/null +++ b/LICENSES/mritd.github.io/files @@ -0,0 +1 @@ +/compass-ci/container/registry/config.yml diff --git a/LICENSES/mritd.github.io/mritd.github.io b/LICENSES/mritd.github.io/mritd.github.io new file mode 100644 index 0000000..ee22568 --- /dev/null +++ b/LICENSES/mritd.github.io/mritd.github.io @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: mritd.github.io ed4350e91e9b76078a850ede8302854d3393f493 + +Copyright notice: Copyright(c) 2013-2016 Blackrock Digital LLC. +License: The MIT License +License-Text: please refer to file MIT -- 2.23.0

2 1

[PATCH v2 lkp-tests 3/3] tests/iozone: modify the way of parsing parameter
by Lu Kaiyi 03 Nov '20

03 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool need change the way of parsing parameter. [how] modify the way of parsing parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 50 ++++++++++++++++++++++---------------------------- 1 file changed, 22 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..19c246f1 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,36 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + [ "$ele" == "write_rewrite" ] && args+=" -i 0" + [ "$ele" == "read_reread" ] && args+=" -i 1" + [ "$ele" == "random_read_write" ] && args+=" -i 2" + [ "$ele" == "read_backwards" ] && args+=" -i 3" + [ "$ele" == "rewrite_record" ] && args+=" -i 4" + [ "$ele" == "stride_read" ] && args+=" -i 5" + [ "$ele" == "fwrite_refwrite" ] && args+=" -i 6" + [ "$ele" == "fread_refread" ] && args+=" -i 7" + [ "$ele" == "random_mix" ] && args+=" -i 8" + [ "$ele" == "pwrite_repwrite" ] && args+=" -i 9" + [ "$ele" == "pread_repread" ] && args+=" -i 10" + [ "$ele" == "pwritev_repwritev" ] && args+=" -i 11" + [ "$ele" == "preadv_repreadv" ] && args+=" -i 12" + done else args+=" -a" fi -- 2.23.0

2 1

[PATCH v2 lkp-tests 1/3] jobs/iozone-bs.yaml: fix mapper_parsing_exception
by Lu Kaiyi 03 Nov '20

03 Nov '20

[why] when key and value of fs: xfs in same line for iozone-bs.yaml, submit job will cause some error occurred like below: submit /home/lukaiyi/lkp-tests/jobs/iozone-bs.yaml failed, got job_id=z9.144590, error: [{"type" => "mapper_parsing_exception", "reason" => "failed to parse [pp.fs.fs]"}] [how] change key and value of fs: xfs to different line, same to iosched: kyber Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index 868979fc..e2cd9f48 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -14,6 +14,11 @@ block_size: - 16m disk: 1HDD -fs: xfs -iosched: kyber + +fs: +- xfs + +iosched: +- kyber + iozone: -- 2.23.0

2 1

[PATCH v8 compass-ci 2/2] job.cr: add a key "kernel_version" for initramfs
by Xu Xijian 03 Nov '20

03 Nov '20

When run job with initramfs, there is a default combination of kernel, modules and headers, and I add a key "kernel_version" to make it optional. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e49da6..6309809 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -87,6 +87,10 @@ class Job kernel_append_root kernel_params docker_image + kernel_version + linux_vmlinuz_path + linux_modules_initrd + linux_headers_initrd ) macro method_missing(call) @@ -135,6 +139,7 @@ class Job set_result_service() set_os_mount() set_depends_initrd() + set_kernel_version() set_initrds_uri() set_kernel_uri() set_kernel_append_root() @@ -293,9 +298,17 @@ class Job return true end + private def set_kernel_version + boot_dir = "#{SRV_OS}/#{os_dir}/boot" + suffix = "-#{kernel_version}" if self["kernel_version"]? + self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") + self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") + self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") + end + private def set_kernel_uri self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{linux_vmlinuz_path}")}" end private def common_initrds @@ -316,6 +329,10 @@ class Job "#{JobHelper.service_path("#{osimage_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{linux_modules_initrd}")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{linux_headers_initrd}")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

1 0

[PATCH v7 compass-ci 1/2] job.cr: add a key "kernel_version" for initramfs
by Xu Xijian 03 Nov '20

03 Nov '20

When run job with initramfs, there is a default combination of kernel, modules and headers, and I add a key "kernel_version" to make it optional. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0fd57e0..2fa1125 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -87,6 +87,10 @@ class Job kernel_append_root kernel_params docker_image + kernel_version + vmlinuz_path + modules_path + headers_path ) macro method_missing(call) @@ -135,6 +139,7 @@ class Job set_result_service() set_os_mount() set_depends_initrd() + set_kernel_version() set_initrds_uri() set_kernel_uri() set_kernel_append_root() @@ -293,9 +298,17 @@ class Job return true end + private def set_kernel_version + boot_dir = "#{SRV_OS}/#{os_dir}/boot" + suffix = "-#{kernel_version}" if self["kernel_version"]? + self["vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") + self["modules_path"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") + self["headers_path"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") + end + private def set_kernel_uri self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{vmlinuz_path}")}" end private def common_initrds @@ -316,6 +329,10 @@ class Job "#{JobHelper.service_path("#{osimage_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{modules_path}")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{headers_path}")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

3 4

[PATCH v2 compass-ci 3/3] container/assign-account: answerback-email.rb
by Luan Shengde 03 Nov '20

03 Nov '20

1. disable use golbal variable apply_info [why] Style/GlobalVars: Do not introduce global variables. [how] use local variables instead of global variables 2. enable transfer user info when execute apply account command [why] when applying account, the assign-account service will write the my info to user's default config file ~/.config/compass-ci/defaults/account.yaml my info: - my_email - my_name - my_uuid [how] transfer the user info along with the pub_key to assign-account service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 62 ++++++++++++-------- 1 file changed, 38 insertions(+), 24 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..c174302 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,6 +12,7 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST @@ -24,23 +25,25 @@ defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +apply_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, apply_info) mail_content = Mail.read(email_file) + apply_info['my_email'] = mail_content.from[0] + apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + apply_info end options = OptionParser.new do |opts| @@ -52,15 +55,17 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + apply_info['my_email'] = email_address + # when apply account with email address, will get no user name + apply_info['my_name'] = '' end opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + apply_info['my_ssh_pubkey'] = File.read(pub_key_file) end opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + init_info(email_file, apply_info) end opts.on_tail('-h|--help', 'show this message') do @@ -94,26 +99,35 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def account_info(user_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{user_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(apply_info) message = "No email address specified\n" message += "use -e email_address add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? + raise message if apply_info['my_email'].nil? - acct_info = account_info($apply_info['my_ssh_pubkey']) + acct_info = account_info(apply_info) + my_info = { + 'my_email' => apply_info['my_email'], + 'my_name' => apply_info['my_name'], + 'my_commit_url' => '', + 'my_login_name' => acct_info['account'], + 'my_uuid' => apply_info['my_uuid'] + } - message = build_message($apply_info['my_email'], acct_info) + store_account_info(my_info) + message = build_message(apply_info['my_email'], acct_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(apply_info) -- 2.23.0

2 2

[PATCH v2 compass-ci 2/3] container/assign-account: get_account_info
by Luan Shengde 03 Nov '20

03 Nov '20

add new function to config default yaml file [why]: easier for user to config the default yaml file [how]: when get request data, parse the data and write the user infos to default yaml file: ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/get_account_info.rb | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..ae55dcb 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,7 +95,7 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? jumper_ip = jumper_info[0].chomp jumper_port = jumper_info[1].chomp @@ -111,9 +113,23 @@ class AccountStorage } setup_authorized_key(account, pub_key) + setup_default_yaml(account) + return jumper_account_info end + def setup_default_yaml(account) + default_yaml_dir = File.join('/home', account, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/#{account}.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{account}:#{account} "/home/#{account}/.config") + end + def setup_authorized_key(account, pub_key) ssh_dir = File.join('/home/', account, '.ssh') Dir.mkdir ssh_dir, 0o700 -- 2.23.0

2 2

[PATCH v3 compass-ci] lib/job.cr: fix pp dir layout
by Wang Yong 03 Nov '20

03 Nov '20

[Why] due to cci-makepkg and cci-depends dir layout change, pp need update together Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0fd57e0..1e49da6 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -405,8 +405,8 @@ class Job program = $1 end - deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/latest.cgz" if File.exists?(deps_dest_file) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] tests/*: fix cci job dir layout
by Wang Yong 03 Nov '20

03 Nov '20

[Why] for multi version support, cgz file for cci-makepkg need change dir layout, and change cci-depends dir layout together to easily update scheduler code path: /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/ => /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/${pkgname}/ filename: cci-makepkg ${pkgver}-${pkgrel}.cgz latest.cgz -> ${pkgver}-${pkgrel}.cgz filename: cci-depends stay original ${benchmark}.cgz ${benchmark}.cgz -> ${benchmark}_${date}.cgz E.g. output: cci-depends stay original $ tree iperf ├── iperf_20201102.cgz └── iperf.cgz -> iperf_20201102.cgz output: cci-makepkg $ tree sar ├── git-1.cgz └── latest.cgz -> git-1.cgz Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 2 +- tests/cci-makepkg | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/cci-depends b/tests/cci-depends index 36923ea4d..efa9a025b 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -18,7 +18,7 @@ DEPS_MNT=/initrd/deps && mkdir -p "$DEPS_MNT" DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} pack_arch=$os_arch [[ "$benchmark" = "${benchmark##*.}" ]] || diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9d5082629..9551ebfe2 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -26,7 +26,7 @@ mkdir -p "$PKG_MNT" export DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} . $LKP_SRC/distro/${DISTRO} . $LKP_SRC/lib/install.sh @@ -65,8 +65,8 @@ update_shared_pkg() [ "$bm_name" = "$benchmark" ] && return # benchmark is a symlink - ln -sf "$bm_link" "$sync_dest/${benchmark}.cgz" || return - echo "update shared pkg link ${benchmark}.cgz -> $bm_link" + ln -sf latest.cgz "$sync_dest/${benchmark}.cgz" || return + echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } distro_install_depends lkp-dev @@ -86,7 +86,7 @@ date=$(date +"%Y%m%d") pkgver=$(get_pkg_info pkgver) pkgrel=$(get_pkg_info pkgrel) bm_name=$(check_shared_pkg) -cgz_name="${bm_name}-${pkgver:-0}-${pkgrel:-0}_${date}.cgz" +cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy @@ -98,8 +98,8 @@ update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return - ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/${bm_name}.cgz" || return - echo "create package: $sync_dest/${bm_name}.cgz -> $(realpath $sync_dest/$cgz_name)" + ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/latest.cgz" || return + echo "create package: $sync_dest/latest.cgz -> $(realpath $sync_dest/$cgz_name)" update_shared_pkg "${bm_name}.cgz" -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] fix(distro/depends): add depends file
by Wang Yong 03 Nov '20

03 Nov '20

[Why] there are pack-deps/makepkg depends file before and use nfs-common, now we use cci-depends/cci-makepkg file and use cifs-utils. Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/cci-depends | 2 ++ distro/depends/cci-makepkg | 10 ++++++++++ 2 files changed, 12 insertions(+) create mode 100644 distro/depends/cci-depends create mode 100644 distro/depends/cci-makepkg diff --git a/distro/depends/cci-depends b/distro/depends/cci-depends new file mode 100644 index 000000000..1f7f73ac3 --- /dev/null +++ b/distro/depends/cci-depends @@ -0,0 +1,2 @@ +cifs-utils +cpio diff --git a/distro/depends/cci-makepkg b/distro/depends/cci-makepkg new file mode 100644 index 000000000..a7ee67537 --- /dev/null +++ b/distro/depends/cci-makepkg @@ -0,0 +1,10 @@ +curl +libarchive-tools +bzip2 +fakeroot +gnupg +gettext +openssl +ncurses-bin +binutils +cifs-utils -- 2.23.0

1 0

[PATCH compass-ci] sparrow/7-systemd: fix failed to resume dnsmasq and es service
by Liu Yinsi 03 Nov '20

03 Nov '20

[why] when execute sparrow/4-docker/buildall reboot, error: ==================== es ======================= [2020-11-02T10:24:23,046][INFO ][o.e.n.Node ] [] initializing ... [2020-11-02T10:24:23,084][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/srv/es/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])? at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.3.jar:6.4.3] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.3.jar:6.4.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.3.jar:6.4.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.3.jar:6.4.3] =================== dnsmasq =================== dnsmasq: failed to bind DHCP server socket: Address in use because sometimes even if 'docker stop&&rm -f $container', there will still be a legacy process running, causing container to fail to start. [how] kill process before start container Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/7-systemd/cci-network | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/7-systemd/cci-network b/sparrow/7-systemd/cci-network index 6e59dbb..dfe7cc9 100755 --- a/sparrow/7-systemd/cci-network +++ b/sparrow/7-systemd/cci-network @@ -12,6 +12,9 @@ $CCI_SRC/sparrow/2-network/iptables $CCI_SRC/sparrow/2-network/nfs $CCI_SRC/sparrow/2-network/cifs +# fix failed to start container dnsmasq and es. +kill $(ps -ef| grep -E "dnsmasq|elasticsearch"| awk '{print $2}' ) 2> /dev/null + # --restart=always option is not absolutely reliable # container need to start its dependences firstly. $CCI_SRC/sparrow/4-docker/buildall reboot -- 2.23.0

2 1

[PATCH v2 compass-ci 1/3] container/assign-account: assign-account.rb
by Luan Shengde 03 Nov '20

03 Nov '20

disable assigning account for user if there is no my_email, my_name, my_uuid [why]: my_email, my_name, my_uuid is required when initialize the default config file [how]: check if the data has keys: my_email, my_name, my_uuid when get apply account request. the apply account will return with 'no my info' if missing any of my_email, my_name or my_uuid. Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/assign-account.rb | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..3afcc9f 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,12 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + message = 'lack of my infos(my_email, my_name, my_uuid)' + raise message unless %w[my_email my_name my_uuid].to_set.subset? data.keys.to_set + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

3 4

[PATCH compass-ci] sched: hand over the job from job.queue to extract queue
by Cao Xueliang 03 Nov '20

03 Nov '20

[why] The job add to job.queue, when lkp finish the job need hand over the job from job.queue to extract queue. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 46b06da..a4b12b4 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -650,7 +650,7 @@ class Sched end response = @task_queue.hand_over_task( - "sched/#{job.tbox_group}", "extract_stats", job_id + "sched/#{job.queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: add ENV TASKQUEUE_PORT
by Liu Yinsi 03 Nov '20

03 Nov '20

need env TASKQUEUE_PORT to start taskqueue service. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f36aaca..64f34f8 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,6 +18,7 @@ mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port +TASKQUEUE_HOST: $sched_host OS_HTTP_HOST: $os_http_host INITRD_HTTP_HOST: $initrd_http_host INITRD_HTTP_PORT: $initrd_http_port -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: fix pp dir layout
by Wang Yong 03 Nov '20

03 Nov '20

[Why] due to cci-makepkg and cci-depends dir layout change, pp need update together Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 955bc4b..4076745 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -434,8 +434,8 @@ class Job program = $1 end - deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/latest.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/latest.cgz" + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" if File.exists?(deps_dest_file) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) -- 2.23.0

2 2

[PATCH compass-ci] container: fix failed to start git-mirror
by Liu Yinsi 03 Nov '20

03 Nov '20

[why] when start git-mirror container in machine, error: /usr/lib/ruby/2.5.0/net/http.rb:939:in `rescue in block in connect': Failed to open TCP connection to 172.17.0.1:9200 (Connection refused - connect(2) for "172.17.0.1" port 9200) (Faraday::ConnectionFailed) Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/git-mirror/start-depends | 1 + 1 file changed, 1 insertion(+) diff --git a/container/git-mirror/start-depends b/container/git-mirror/start-depends index 5232abf..38fc696 100755 --- a/container/git-mirror/start-depends +++ b/container/git-mirror/start-depends @@ -1 +1,2 @@ rabbitmq +es -- 2.23.0

1 0

[PATCH compass-ci 1/8] lib/mail-robot.rb
by Luan Shengde 03 Nov '20

03 Nov '20

mail robot for applying account monitor mailbox check new added email file in new email directory handle new email file check to apply account only if email's subject match 'apply account' Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- lib/mail-robot.rb | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..9437078 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + begin + handle_new_email(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + puts e.backtrace + end + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + check_to_apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def check_to_apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.check_to_send_account +end + +monitor_dir("#{ENV['MAILDIR']}/new", "#{ENV['MAILDIR']}/cur") -- 2.23.0

2 1

[PATCH lkp-tests] bin/lkp-setup-rootfs: change the script that called when packaging
by Li Ping 02 Nov '20

02 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- bin/lkp-setup-rootfs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 2890601e..680027e6 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -187,14 +187,14 @@ define_files_pkg_pack() ( local benchmark=${1#pkg/} export benchmark=${benchmark%/PKGBUILD} - $LKP_SRC/tests/makepkg + $LKP_SRC/tests/cci-makepkg ) } # There may be $program-dev or $program-dev-dev in the # "$LKP_SRC/distro/depends" directory, # when the value passed by $define_files is "distro/depends/$program-dev", -# you don't know which script in "makepkg" or "pack-deps" to execute, +# you don't know which script in "cci-makepkg" or "cci-depends" to execute, # so it needs to be handled sparately. # Save the result to # "/srv/initrd/deps/$os/$os_arch/$os_version/$deps_benchmark.cgz", @@ -208,7 +208,7 @@ define_files_deps_pack() [ $suffix == "dev" ] && [ -f "$LKP_SRC/pkg/$pkg_benchmark/PKGBUILD" ] && { ( export benchmark=$pkg_benchmark - $LKP_SRC/tests/makepkg + $LKP_SRC/tests/cci-makepkg ) return 0 } @@ -216,7 +216,7 @@ define_files_deps_pack() ( export benchmark=$deps_benchmark export pack_arch=$os_arch - $LKP_SRC/tests/pack-deps + $LKP_SRC/tests/cci-depends ) } -- 2.23.0

1 0

[PATCH compass-ci 8/8] doc/mail-robot.md
by Luan Shengde 02 Nov '20

02 Nov '20

documentation for mail-robot service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 76 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..d8ac188 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,76 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + my oss commit: commit_url +attahcment + ssh pub_key + - id_rsa.pub + +## steps overview + +1. monitor + - the monitor will listen to the mailbox for new email files + handle new email file: + check to send account + email subject: apply account + +2. assign account + - check email + check commit url + check whether there is a standard commit url + check whether the repo's url in upstream-repos + check whether the commit url valid + if the hub name is gitee.com + # authentication is needed qwhen use curl to check the url + clone the repo and check the commit + if the hub name is not gitee.com + check the comit with curl + + check pub_key + check whether there is a ssh pub_key in the email file + + it will return failure email if any check end with failure + + - generate uuid + - apply account + build user_info + - my_email + - my_name + - my_uuid + - lab + - pub_key + execute command to apply account on jumper and return the account info + curl -XGET "${JUMPER_HOST}:${JUMPER_PORT}/assign_account" -d '${user_info}' + write my_email, my_name, my_uuid, lab to the defaule config file: + + - store apply infos + apply_infos: + - my_email + - my_name + - my_commit_url + - my_login_name + - my_uuid + invoke es to store the apply infos + + - send uuid + build send uuid email message with the apply_infos + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

1 0

[PATCH compass-ci 7/8] container/mail-robot: start
by Luan Shengde 02 Nov '20

02 Nov '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/start | 41 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..671eaf2 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT + MAILDIR + ROBOT_TMP_REPOS_DIR +] + +defaults = relevant_defaults(names) + +defaults['CCI_SRC'] = '/c/compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:#{defaults['CCI_SRC']}:rw + -w #{defaults['CCI_SRC']}/container/mail-robot + --log-driver json-file + mail-robot +] + +cmd += ['ruby', 'run.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci 6/8] container/mail-robot: build
by Luan Shengde 02 Nov '20

02 Nov '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH compass-ci 5/8] container/mail-robot: Dockerfile
by Luan Shengde 02 Nov '20

02 Nov '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..5eea701 --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER shdluan(a)163.com + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport + +RUN mkdir -p /repos_dir -- 2.23.0

1 0

[PATCH compass-ci 4/8] container/mail-robot: run.rb
by Luan Shengde 02 Nov '20

02 Nov '20

invoke apply-account-component, apply-account.rb, mail-rebot.rb to run the service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/run.rb | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 container/mail-robot/run.rb diff --git a/container/mail-robot/run.rb b/container/mail-robot/run.rb new file mode 100755 index 0000000..b432d7d --- /dev/null +++ b/container/mail-robot/run.rb @@ -0,0 +1,8 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'apply-account-component' +require_relative 'apply-account' +require_relative '../../lib/mail-robot' -- 2.23.0

1 0

[PATCH lkp-tests 2/2] tests/iozone: modify parameter parse process
by Lu Kaiyi 02 Nov '20

02 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool need change parse process. [how] modify parse process Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 50 ++++++++++++++++++++++---------------------------- 1 file changed, 22 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..19c246f1 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,36 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + [ "$ele" == "write_rewrite" ] && args+=" -i 0" + [ "$ele" == "read_reread" ] && args+=" -i 1" + [ "$ele" == "random_read_write" ] && args+=" -i 2" + [ "$ele" == "read_backwards" ] && args+=" -i 3" + [ "$ele" == "rewrite_record" ] && args+=" -i 4" + [ "$ele" == "stride_read" ] && args+=" -i 5" + [ "$ele" == "fwrite_refwrite" ] && args+=" -i 6" + [ "$ele" == "fread_refread" ] && args+=" -i 7" + [ "$ele" == "random_mix" ] && args+=" -i 8" + [ "$ele" == "pwrite_repwrite" ] && args+=" -i 9" + [ "$ele" == "pread_repread" ] && args+=" -i 10" + [ "$ele" == "pwritev_repwritev" ] && args+=" -i 11" + [ "$ele" == "preadv_repreadv" ] && args+=" -i 12" + done else args+=" -a" fi -- 2.23.0

1 0

[PATCH lkp-tests 1/2] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 02 Nov '20

02 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..f9ab305f 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write_rewrite,read_reread,random_read_write block_size: - 64k -- 2.23.0

1 0

[PATCH lkp-tests 2/2] tests/iozone: modify parameter parse process
by Lu Kaiyi 02 Nov '20

02 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool need change parse process. [how] modify parse process Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 50 ++++++++++++++++++++++---------------------------- 1 file changed, 22 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..19c246f1 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,36 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + [ "$ele" == "write_rewrite" ] && args+=" -i 0" + [ "$ele" == "read_reread" ] && args+=" -i 1" + [ "$ele" == "random_read_write" ] && args+=" -i 2" + [ "$ele" == "read_backwards" ] && args+=" -i 3" + [ "$ele" == "rewrite_record" ] && args+=" -i 4" + [ "$ele" == "stride_read" ] && args+=" -i 5" + [ "$ele" == "fwrite_refwrite" ] && args+=" -i 6" + [ "$ele" == "fread_refread" ] && args+=" -i 7" + [ "$ele" == "random_mix" ] && args+=" -i 8" + [ "$ele" == "pwrite_repwrite" ] && args+=" -i 9" + [ "$ele" == "pread_repread" ] && args+=" -i 10" + [ "$ele" == "pwritev_repwritev" ] && args+=" -i 11" + [ "$ele" == "preadv_repreadv" ] && args+=" -i 12" + done else args+=" -a" fi -- 2.23.0

1 0

[PATCH lkp-tests 1/2] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 02 Nov '20

02 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..f9ab305f 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write_rewrite,read_reread,random_read_write block_size: - 64k -- 2.23.0

1 0

[PATCH v3 lkp-tests] jobs/hackbench-git.yaml: fix testsuite and datasize filed
by Zhang Yale 02 Nov '20

02 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/hackbench-git.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/jobs/hackbench-git.yaml b/jobs/hackbench-git.yaml index 12943c04..4a4b6a06 100644 --- a/jobs/hackbench-git.yaml +++ b/jobs/hackbench-git.yaml @@ -1,4 +1,4 @@ -suite: hackbench +suite: hackbench-git category: benchmark disable_latency_stats: 1 @@ -11,10 +11,10 @@ nr_threads: - 10 - 20 iterations: 10 -datasize: 1000 +loops: 1000 runtime: 300 -hackbench: +hackbench-git: mode: - process - threads -- 2.23.0

1 1

[PATCH v2 lkp-tests] jobs: fix testsuite and datasize filed
by Zhang Yale 02 Nov '20

02 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/hackbench-git.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/jobs/hackbench-git.yaml b/jobs/hackbench-git.yaml index 12943c04..4a4b6a06 100644 --- a/jobs/hackbench-git.yaml +++ b/jobs/hackbench-git.yaml @@ -1,4 +1,4 @@ -suite: hackbench +suite: hackbench-git category: benchmark disable_latency_stats: 1 @@ -11,10 +11,10 @@ nr_threads: - 10 - 20 iterations: 10 -datasize: 1000 +loops: 1000 runtime: 300 -hackbench: +hackbench-git: mode: - process - threads -- 2.23.0

1 1

[PATCH v2 compass-ci] container/rsync-server: modify data upload directory
by Cui Yili 02 Nov '20

02 Nov '20

[why] Modify data upload directory to upload qemu image. qemu image uploaded by the user, used for testing [how] Change the rsync.conf configuration and docker operating parameters. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/rsync-server/rsyncd.conf | 6 ++++-- container/rsync-server/start | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index 528e8bd..6494bc8 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -1,10 +1,12 @@ uid = lkp gid = lkp +port = 11387 use chroot = no +log file=/var/log/rsync.log -[result] -path = /result +[upload] +path = /srv/result/upload/ read only = no max connections = 200 refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 4124192..b24605b 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -9,9 +9,9 @@ cmd=( docker run -id --name=rsync_server - -v /srv/result:/result + -v /srv/result/upload:/srv/result/upload -v /etc/localtime:/etc/localtime:ro - -p 873:873 + -p 11387:11387 rsync_server:latest ) -- 2.23.0

3 2

[PATCH v3 compass-ci] container/rsync-server: modify data upload directory
by Cui Yili 02 Nov '20

02 Nov '20

[why] Modify data upload directory to upload qemu image. qemu image uploaded by the user, used for testing [how] Change the rsync.conf configuration and docker operating parameters. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/rsync-server/rsyncd.conf | 6 ++++-- container/rsync-server/start | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index 528e8bd..cdba89c 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -1,10 +1,12 @@ uid = lkp gid = lkp +port = 11387 use chroot = no +log file = /var/log/rsync.log -[result] -path = /result +[upload] +path = /srv/result/upload/ read only = no max connections = 200 refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 4124192..b24605b 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -9,9 +9,9 @@ cmd=( docker run -id --name=rsync_server - -v /srv/result:/result + -v /srv/result/upload:/srv/result/upload -v /etc/localtime:/etc/localtime:ro - -p 873:873 + -p 11387:11387 rsync_server:latest ) -- 2.23.0

1 0

[PATCH v2 compass-ci] container: fix failed to build archlinux images
by Liu Yinsi 02 Nov '20

02 Nov '20

[why] when build archlinux images in x86_64 machine error: error: failed retrieving file 'core.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update core (failed to retrieve some files) error: failed retrieving file 'extra.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Resolving timed out after 10000 milliseconds error: failed to update extra (download library error) error: failed retrieving file 'community.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update community (failed to retrieve some files) error: failed to synchronize all databases The command '/bin/sh -c pacman --needed --noprogressbar --noconfirm -Syu && pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk' returned a non-zero code: 1 becaue archlinux mirror not support x86_64 machine. [how] when deploy in x86_64 machine, change mirror http://mirrors.tuna.tsinghua.edu.cn/archlinuxarm/$arch/$repo => http://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/archlinux/build | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/archlinux/build b/container/archlinux/build index 81feda2..f96ad40 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[ "$(arch)" == "x86_64" ] && + sed -i 's#archlinuxarm/$arch/$repo#archlinux/$repo/os/$arch#g' root/etc/pacman.d/mirrorlist + docker build -t archlinux:testbed . -- 2.23.0

3 4

[PATCH v2 compass-ci] src/lib/web_backend.rb: "/compare_candidates" tbox_group regex error
by Zhang Yuhang 02 Nov '20

02 Nov '20

[error info] 1. /\d+$/ will be matched, such as "xxx123". 2. `index = "xxx123".index('--') || "xxx123".rindex('-')` - index equal nil. 3. `r = r[0, nil]` - raise a TypeError. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index c3f6bef..55c5e4a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -47,7 +47,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end -- 2.23.0

3 4

[PATCH v6 compass-ci 2/3] job.cr: add a key "kernel_version" for initramfs
by Xu Xijian 02 Nov '20

02 Nov '20

When run job with initramfs, there is a default combination of kernel, modules and headers, and I add a key "kernel_version" to make it optional. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0fd57e0..8bf7a65 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -87,6 +87,10 @@ class Job kernel_append_root kernel_params docker_image + kernel_version + vmlinuz_path + modules_path + headers_path ) macro method_missing(call) @@ -135,6 +139,7 @@ class Job set_result_service() set_os_mount() set_depends_initrd() + set_kernel_version() set_initrds_uri() set_kernel_uri() set_kernel_append_root() @@ -293,9 +298,22 @@ class Job return true end + private def set_kernel_version + boot_dir = "#{SRV_OS}/#{os_dir}/boot" + if self["kernel_version"]? + self["vmlinuz_path"] = "#{boot_dir}/vmlinuz-#{kernel_version}" + self["modules_path"] = "#{boot_dir}/modules-#{kernel_version}.cgz" + self["headers_path"] = "#{boot_dir}/headers-#{kernel_version}.cgz" + else + self["vmlinuz_path"] = "#{boot_dir}/vmlinuz" + self["modules_path"] = "#{boot_dir}/modules.cgz" + self["headers_path"] = "#{boot_dir}/headers.cgz" + end + end + private def set_kernel_uri self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{vmlinuz_path}")}" end private def common_initrds @@ -316,6 +334,10 @@ class Job "#{JobHelper.service_path("#{osimage_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{modules_path}")}" + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{headers_path}")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

2 2

[PATCH lkp-tests] jobs/iozone-bs.yaml: fix mapper_parsing_exception
by Lu Kaiyi 02 Nov '20

02 Nov '20

[why] when key:value of fs: xfs in same line for .yaml file, submit job will cause some error occurred like below: submit /home/lukaiyi/lkp-tests/jobs/iozone-bs.yaml failed, got job_id=z9.144590, error: [{"type" => "mapper_parsing_exception", "reason" => "failed to parse [pp.fs.fs]"}] [how] change key and value of fs: xfs to different line, same to iosched: kyber Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index 868979fc..e2cd9f48 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -14,6 +14,11 @@ block_size: - 16m disk: 1HDD -fs: xfs -iosched: kyber + +fs: +- xfs + +iosched: +- kyber + iozone: -- 2.23.0

1 0

[PATCH compass-ci] based on queue field add the job to specified queue
by Xiao Shenwei 02 Nov '20

02 Nov '20

add queue field into job.yaml, refer to commit-id 912214cf97c125ed5387dc20ce394ee6db2b11b6 then use queue field replace tbox_group to add the job Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- src/lib/sched.cr | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 80cf0b0..b4745aa 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -278,24 +278,24 @@ class Sched # success: [{"job_id" => job_id, "message" => "", job_state => "submit"}] # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) - tbox_group = job.tbox_group + queue = job.queue return [{ "job_id" => "0", - "message" => "get tbox group failed", + "message" => "get queue failed", "job_state" => "submit", - }] unless tbox_group + }] unless queue # only single job will has "idle job" and "execute rate limiter" if job["idle_job"].empty? - tbox_group += "#{job.get_uuid_tag}" + queue += "#{job.get_uuid_tag}" else - tbox_group = "#{tbox_group}/idle" + queue = "#{queue}/idle" end - job_id = add_task(tbox_group, job.lab) + job_id = add_task(queue, job.lab) return [{ "job_id" => "0", - "message" => "add task queue sched/#{tbox_group} failed", + "message" => "add task queue sched/#{queue} failed", "job_state" => "submit", }] unless job_id @@ -311,9 +311,9 @@ class Sched end # return job_id - def add_task(tbox_group, lab) + def add_task(queue, lab) task_desc = JSON.parse(%({"domain": "compass-ci", "lab": "#{lab}"})) - response = @task_queue.add_task("sched/#{tbox_group}", task_desc) + response = @task_queue.add_task("sched/#{queue}", task_desc) JSON.parse(response[1].to_json)["id"].to_s if response[0] == 200 end -- 2.23.0

1 0

[PATCH lab-z9] modify hosts file
by Zhang Dewan 02 Nov '20

02 Nov '20

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a101 | 5 +---- hosts/taishan200-2280-2s48p-256g--a11 | 5 +---- hosts/taishan200-2280-2s48p-256g--a13 | 5 +---- hosts/taishan200-2280-2s48p-256g--a14 | 5 +---- hosts/taishan200-2280-2s48p-256g--a15 | 5 +---- hosts/taishan200-2280-2s48p-256g--a17 | 5 +---- hosts/taishan200-2280-2s48p-256g--a18 | 5 +---- hosts/taishan200-2280-2s48p-256g--a19 | 5 +---- hosts/taishan200-2280-2s48p-256g--a21 | 5 +---- hosts/taishan200-2280-2s48p-256g--a22 | 5 +---- hosts/taishan200-2280-2s48p-256g--a24 | 5 +---- hosts/taishan200-2280-2s48p-256g--a25 | 5 +---- hosts/taishan200-2280-2s48p-256g--a26 | 5 +---- hosts/taishan200-2280-2s48p-256g--a27 | 5 +---- hosts/taishan200-2280-2s48p-256g--a28 | 5 +---- hosts/taishan200-2280-2s48p-256g--a29 | 5 +---- hosts/taishan200-2280-2s48p-256g--a30 | 5 +---- hosts/taishan200-2280-2s48p-256g--a31 | 5 +---- hosts/taishan200-2280-2s48p-256g--a32 | 5 +---- hosts/taishan200-2280-2s48p-256g--a33 | 5 +---- hosts/taishan200-2280-2s48p-256g--a34 | 5 +---- hosts/taishan200-2280-2s48p-256g--a52 | 5 +---- hosts/taishan200-2280-2s48p-256g--a53 | 5 +---- hosts/taishan200-2280-2s48p-256g--a60 | 5 +---- hosts/taishan200-2280-2s48p-256g--a61 | 5 +---- hosts/taishan200-2280-2s48p-256g--a62 | 5 +---- hosts/taishan200-2280-2s48p-256g--a66 | 5 +---- hosts/taishan200-2280-2s48p-256g--a67 | 5 +---- hosts/taishan200-2280-2s48p-256g--a68 | 5 +---- hosts/taishan200-2280-2s48p-256g--a69 | 5 +---- hosts/taishan200-2280-2s48p-256g--a70 | 5 +---- hosts/taishan200-2280-2s48p-256g--a71 | 5 +---- ...280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} | 5 +---- ...-2280-2s48p-256g--a95 => taishan200-2280-2s48p-512g--a95} | 5 +---- hosts/taishan200-2280-2s64p-256g--a10 | 5 +---- hosts/taishan200-2280-2s64p-256g--a3 | 5 +---- hosts/taishan200-2280-2s64p-256g--a36 | 5 +---- hosts/taishan200-2280-2s64p-256g--a4 | 5 +---- hosts/taishan200-2280-2s64p-256g--a40 | 5 +---- hosts/taishan200-2280-2s64p-256g--a41 | 5 +---- hosts/taishan200-2280-2s64p-256g--a42 | 5 +---- hosts/taishan200-2280-2s64p-256g--a8 | 5 +---- hosts/taishan200-2280-2s64p-256g--a9 | 5 +---- 43 files changed, 43 insertions(+), 172 deletions(-) rename hosts/{taishan200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} (93%) rename hosts/{taishan200-2280-2s48p-256g--a95 => taishan200-2280-2s48p-512g--a95} (89%) diff --git a/hosts/taishan200-2280-2s48p-256g--a101 b/hosts/taishan200-2280-2s48p-256g--a101 index f8a0429..200087c 100644 --- a/hosts/taishan200-2280-2s48p-256g--a101 +++ b/hosts/taishan200-2280-2s48p-256g--a101 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 nr_ssd_partitions: 1 hdd_partitions: diff --git a/hosts/taishan200-2280-2s48p-256g--a11 b/hosts/taishan200-2280-2s48p-256g--a11 index 619dc06..54db822 100644 --- a/hosts/taishan200-2280-2s48p-256g--a11 +++ b/hosts/taishan200-2280-2s48p-256g--a11 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8891431-part1 diff --git a/hosts/taishan200-2280-2s48p-256g--a13 b/hosts/taishan200-2280-2s48p-256g--a13 index 468c175..0c9fe89 100644 --- a/hosts/taishan200-2280-2s48p-256g--a13 +++ b/hosts/taishan200-2280-2s48p-256g--a13 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-35000c500c2799bb3 diff --git a/hosts/taishan200-2280-2s48p-256g--a14 b/hosts/taishan200-2280-2s48p-256g--a14 index 24ea9dc..f4c87bb 100644 --- a/hosts/taishan200-2280-2s48p-256g--a14 +++ b/hosts/taishan200-2280-2s48p-256g--a14 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88b4475 diff --git a/hosts/taishan200-2280-2s48p-256g--a15 b/hosts/taishan200-2280-2s48p-256g--a15 index 9c28258..4eb4d2e 100644 --- a/hosts/taishan200-2280-2s48p-256g--a15 +++ b/hosts/taishan200-2280-2s48p-256g--a15 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b841fff5 diff --git a/hosts/taishan200-2280-2s48p-256g--a17 b/hosts/taishan200-2280-2s48p-256g--a17 index 16b065d..1f9df96 100644 --- a/hosts/taishan200-2280-2s48p-256g--a17 +++ b/hosts/taishan200-2280-2s48p-256g--a17 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88b44cd diff --git a/hosts/taishan200-2280-2s48p-256g--a18 b/hosts/taishan200-2280-2s48p-256g--a18 index 9d1ec01..d570c02 100644 --- a/hosts/taishan200-2280-2s48p-256g--a18 +++ b/hosts/taishan200-2280-2s48p-256g--a18 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b891b4d1 diff --git a/hosts/taishan200-2280-2s48p-256g--a19 b/hosts/taishan200-2280-2s48p-256g--a19 index 217eaa9..bf856a8 100644 --- a/hosts/taishan200-2280-2s48p-256g--a19 +++ b/hosts/taishan200-2280-2s48p-256g--a19 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8923939 diff --git a/hosts/taishan200-2280-2s48p-256g--a21 b/hosts/taishan200-2280-2s48p-256g--a21 index 738738d..8f68f5d 100644 --- a/hosts/taishan200-2280-2s48p-256g--a21 +++ b/hosts/taishan200-2280-2s48p-256g--a21 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bf2d5 diff --git a/hosts/taishan200-2280-2s48p-256g--a22 b/hosts/taishan200-2280-2s48p-256g--a22 index bb29d94..8efa398 100644 --- a/hosts/taishan200-2280-2s48p-256g--a22 +++ b/hosts/taishan200-2280-2s48p-256g--a22 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8919069 diff --git a/hosts/taishan200-2280-2s48p-256g--a24 b/hosts/taishan200-2280-2s48p-256g--a24 index 60ea1d2..4f85b6c 100644 --- a/hosts/taishan200-2280-2s48p-256g--a24 +++ b/hosts/taishan200-2280-2s48p-256g--a24 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8917d59 diff --git a/hosts/taishan200-2280-2s48p-256g--a25 b/hosts/taishan200-2280-2s48p-256g--a25 index 916513e..eab76f0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a25 +++ b/hosts/taishan200-2280-2s48p-256g--a25 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88b4499 diff --git a/hosts/taishan200-2280-2s48p-256g--a26 b/hosts/taishan200-2280-2s48p-256g--a26 index 4a5a0f3..46d4b57 100644 --- a/hosts/taishan200-2280-2s48p-256g--a26 +++ b/hosts/taishan200-2280-2s48p-256g--a26 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bf131 diff --git a/hosts/taishan200-2280-2s48p-256g--a27 b/hosts/taishan200-2280-2s48p-256g--a27 index 129589c..7e7692a 100644 --- a/hosts/taishan200-2280-2s48p-256g--a27 +++ b/hosts/taishan200-2280-2s48p-256g--a27 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8918ea9 diff --git a/hosts/taishan200-2280-2s48p-256g--a28 b/hosts/taishan200-2280-2s48p-256g--a28 index f8a77f8..fa788e9 100644 --- a/hosts/taishan200-2280-2s48p-256g--a28 +++ b/hosts/taishan200-2280-2s48p-256g--a28 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-35000c500cdf1eb07 diff --git a/hosts/taishan200-2280-2s48p-256g--a29 b/hosts/taishan200-2280-2s48p-256g--a29 index 37814cc..87f1f18 100644 --- a/hosts/taishan200-2280-2s48p-256g--a29 +++ b/hosts/taishan200-2280-2s48p-256g--a29 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8918425 diff --git a/hosts/taishan200-2280-2s48p-256g--a30 b/hosts/taishan200-2280-2s48p-256g--a30 index f722edb..5e69e11 100644 --- a/hosts/taishan200-2280-2s48p-256g--a30 +++ b/hosts/taishan200-2280-2s48p-256g--a30 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b883b881 diff --git a/hosts/taishan200-2280-2s48p-256g--a31 b/hosts/taishan200-2280-2s48p-256g--a31 index 7d6c67c..87448ab 100644 --- a/hosts/taishan200-2280-2s48p-256g--a31 +++ b/hosts/taishan200-2280-2s48p-256g--a31 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b883b929 diff --git a/hosts/taishan200-2280-2s48p-256g--a32 b/hosts/taishan200-2280-2s48p-256g--a32 index fb17e38..28c9a85 100644 --- a/hosts/taishan200-2280-2s48p-256g--a32 +++ b/hosts/taishan200-2280-2s48p-256g--a32 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bf141 diff --git a/hosts/taishan200-2280-2s48p-256g--a33 b/hosts/taishan200-2280-2s48p-256g--a33 index a3a6b03..74be9f5 100644 --- a/hosts/taishan200-2280-2s48p-256g--a33 +++ b/hosts/taishan200-2280-2s48p-256g--a33 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8919555 diff --git a/hosts/taishan200-2280-2s48p-256g--a34 b/hosts/taishan200-2280-2s48p-256g--a34 index 5a15949..ca35af0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a34 +++ b/hosts/taishan200-2280-2s48p-256g--a34 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88be219 diff --git a/hosts/taishan200-2280-2s48p-256g--a52 b/hosts/taishan200-2280-2s48p-256g--a52 index 2bd7096..f3442dc 100644 --- a/hosts/taishan200-2280-2s48p-256g--a52 +++ b/hosts/taishan200-2280-2s48p-256g--a52 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-35000c500cdefd89b diff --git a/hosts/taishan200-2280-2s48p-256g--a53 b/hosts/taishan200-2280-2s48p-256g--a53 index bf0894b..bffcb38 100644 --- a/hosts/taishan200-2280-2s48p-256g--a53 +++ b/hosts/taishan200-2280-2s48p-256g--a53 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-35000c500bd6aa4e3 diff --git a/hosts/taishan200-2280-2s48p-256g--a60 b/hosts/taishan200-2280-2s48p-256g--a60 index 2255f2f..092e0bb 100644 --- a/hosts/taishan200-2280-2s48p-256g--a60 +++ b/hosts/taishan200-2280-2s48p-256g--a60 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-35000c500bd67640b diff --git a/hosts/taishan200-2280-2s48p-256g--a61 b/hosts/taishan200-2280-2s48p-256g--a61 index 73079cb..e496d34 100644 --- a/hosts/taishan200-2280-2s48p-256g--a61 +++ b/hosts/taishan200-2280-2s48p-256g--a61 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399886a7869 diff --git a/hosts/taishan200-2280-2s48p-256g--a62 b/hosts/taishan200-2280-2s48p-256g--a62 index abaac6e..ef5dc25 100644 --- a/hosts/taishan200-2280-2s48p-256g--a62 +++ b/hosts/taishan200-2280-2s48p-256g--a62 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b892d131 diff --git a/hosts/taishan200-2280-2s48p-256g--a66 b/hosts/taishan200-2280-2s48p-256g--a66 index fce93d1..a22322d 100644 --- a/hosts/taishan200-2280-2s48p-256g--a66 +++ b/hosts/taishan200-2280-2s48p-256g--a66 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-35000c5009a671773 diff --git a/hosts/taishan200-2280-2s48p-256g--a67 b/hosts/taishan200-2280-2s48p-256g--a67 index 6b32609..f3f92c5 100644 --- a/hosts/taishan200-2280-2s48p-256g--a67 +++ b/hosts/taishan200-2280-2s48p-256g--a67 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88aed21 diff --git a/hosts/taishan200-2280-2s48p-256g--a68 b/hosts/taishan200-2280-2s48p-256g--a68 index 4bdd8f4..f90f5c1 100644 --- a/hosts/taishan200-2280-2s48p-256g--a68 +++ b/hosts/taishan200-2280-2s48p-256g--a68 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8909f81 diff --git a/hosts/taishan200-2280-2s48p-256g--a69 b/hosts/taishan200-2280-2s48p-256g--a69 index 3d3be80..a1eb381 100644 --- a/hosts/taishan200-2280-2s48p-256g--a69 +++ b/hosts/taishan200-2280-2s48p-256g--a69 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88be92d diff --git a/hosts/taishan200-2280-2s48p-256g--a70 b/hosts/taishan200-2280-2s48p-256g--a70 index 89c3c77..e61655d 100644 --- a/hosts/taishan200-2280-2s48p-256g--a70 +++ b/hosts/taishan200-2280-2s48p-256g--a70 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b890931d diff --git a/hosts/taishan200-2280-2s48p-256g--a71 b/hosts/taishan200-2280-2s48p-256g--a71 index 3121212..49eff04 100644 --- a/hosts/taishan200-2280-2s48p-256g--a71 +++ b/hosts/taishan200-2280-2s48p-256g--a71 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 256G +<<: hosts/taishan200-2280-2s48p-256g mac_addr: - 44:67:47:d7:6d:e6 - 44:67:47:d7:6d:e7 diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-512g--a102 similarity index 93% rename from hosts/taishan200-2280-2s48p-256g--a102 rename to hosts/taishan200-2280-2s48p-512g--a102 index e1d3f10..ea249e6 100644 --- a/hosts/taishan200-2280-2s48p-256g--a102 +++ b/hosts/taishan200-2280-2s48p-512g--a102 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 512G +<<: hosts/taishan200-2280-2s48p-512g nr_hdd_partitions: 2 nr_ssd_partitions: 3 hdd_partitions: diff --git a/hosts/taishan200-2280-2s48p-256g--a95 b/hosts/taishan200-2280-2s48p-512g--a95 similarity index 89% rename from hosts/taishan200-2280-2s48p-256g--a95 rename to hosts/taishan200-2280-2s48p-512g--a95 index 213d10f..bcd135f 100644 --- a/hosts/taishan200-2280-2s48p-256g--a95 +++ b/hosts/taishan200-2280-2s48p-512g--a95 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 -memory: 512G +<<: hosts/taishan200-2280-2s48p-512g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-35000039988792e31 diff --git a/hosts/taishan200-2280-2s64p-256g--a10 b/hosts/taishan200-2280-2s64p-256g--a10 index cdb32f5..e630c59 100644 --- a/hosts/taishan200-2280-2s64p-256g--a10 +++ b/hosts/taishan200-2280-2s64p-256g--a10 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 128 -memory: 256G +<<: hosts/taishan200-2280-2s64p-256g nr_ssd_partitions: 1 rootfs_partition: - /dev/disk/by-id/scsi-350000399c8030fd5 diff --git a/hosts/taishan200-2280-2s64p-256g--a3 b/hosts/taishan200-2280-2s64p-256g--a3 index 8f34f3f..53f7a81 100644 --- a/hosts/taishan200-2280-2s64p-256g--a3 +++ b/hosts/taishan200-2280-2s64p-256g--a3 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 128 -memory: 256G +<<: hosts/taishan200-2280-2s64p-256g nr_ssd_partitions: 3 ssd_partitions: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part1 diff --git a/hosts/taishan200-2280-2s64p-256g--a36 b/hosts/taishan200-2280-2s64p-256g--a36 index 9757144..eb828b8 100644 --- a/hosts/taishan200-2280-2s64p-256g--a36 +++ b/hosts/taishan200-2280-2s64p-256g--a36 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 128 -memory: 256G +<<: hosts/taishan200-2280-2s64p-256g nr_ssd_partitions: 1 rootfs_partition: - /dev/disk/by-id/scsi-3500003999813487d diff --git a/hosts/taishan200-2280-2s64p-256g--a4 b/hosts/taishan200-2280-2s64p-256g--a4 index bcfaea7..50161f6 100644 --- a/hosts/taishan200-2280-2s64p-256g--a4 +++ b/hosts/taishan200-2280-2s64p-256g--a4 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 128 -memory: 256G +<<: hosts/taishan200-2280-2s64p-256g nr_ssd_partitions: 3 ssd_partitions: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43892-part1 diff --git a/hosts/taishan200-2280-2s64p-256g--a40 b/hosts/taishan200-2280-2s64p-256g--a40 index d9a9937..e8a7b2e 100644 --- a/hosts/taishan200-2280-2s64p-256g--a40 +++ b/hosts/taishan200-2280-2s64p-256g--a40 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 128 -memory: 256G +<<: hosts/taishan200-2280-2s64p-256g nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-3500003999812ed81 diff --git a/hosts/taishan200-2280-2s64p-256g--a41 b/hosts/taishan200-2280-2s64p-256g--a41 index 91531fd..3cbda60 100644 --- a/hosts/taishan200-2280-2s64p-256g--a41 +++ b/hosts/taishan200-2280-2s64p-256g--a41 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 128 -memory: 256G +<<: hosts/taishan200-2280-2s64p-256g nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8919495 diff --git a/hosts/taishan200-2280-2s64p-256g--a42 b/hosts/taishan200-2280-2s64p-256g--a42 index ed1f188..61dc01d 100644 --- a/hosts/taishan200-2280-2s64p-256g--a42 +++ b/hosts/taishan200-2280-2s64p-256g--a42 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 128 -memory: 256G +<<: hosts/taishan200-2280-2s64p-256g nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8918a11 diff --git a/hosts/taishan200-2280-2s64p-256g--a8 b/hosts/taishan200-2280-2s64p-256g--a8 index e515306..8dc0734 100644 --- a/hosts/taishan200-2280-2s64p-256g--a8 +++ b/hosts/taishan200-2280-2s64p-256g--a8 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 128 -memory: 256G +<<: hosts/taishan200-2280-2s64p-256g rootfs_partition: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43325 mac_addr: diff --git a/hosts/taishan200-2280-2s64p-256g--a9 b/hosts/taishan200-2280-2s64p-256g--a9 index ce3aa0a..bdd27f0 100644 --- a/hosts/taishan200-2280-2s64p-256g--a9 +++ b/hosts/taishan200-2280-2s64p-256g--a9 @@ -1,7 +1,4 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 128 -memory: 256G +<<: hosts/taishan200-2280-2s64p-256g nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-35000c500bd6a1347 -- 2.23.0

2 1

[PATCH v4 compass-ci] save hostname and queues relation in redis
by Xiao Shenwei 02 Nov '20

02 Nov '20

[why] the test machine has the ability to know which queues should be requested [how] for VM HW, we will register mac2host and host2queues. then they can find queues based on mac. API: boot.ipxe/mac/${mac} for docker, we will register host2queues, then container can find queues based on hostname. API: boot.container/hostname/${hostname Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- src/lib/sched.cr | 8 ++++++++ src/scheduler/scheduler.cr | 24 ++++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index b2ef6ce..80cf0b0 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -38,6 +38,14 @@ class Sched @redis.hash_del("sched/mac2host", normalize_mac(mac)) end + def set_host2queues(hostname : String, queues : String) + @redis.hash_set("sched/host2queues", hostname, queues) + end + + def del_host2queues(hostname : String) + @redis.hash_del("sched/host2queues", hostname) + end + # return: # Hash(String, Hash(String, String)) def get_cluster_state(cluster_id) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 371e4ed..a946ea0 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -113,6 +113,18 @@ module Scheduler end end + # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" + put "/set_host2queues" do |env| + env.response.headers["Connection"] = "close" + if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) + sched.set_host2queues(client_host, client_queues) + + "Done" + else + "No yet" + end + end + # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| env.response.headers["Connection"] = "close" @@ -125,6 +137,18 @@ module Scheduler end end + # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" + put "/del_host2queues" do |env| + env.response.headers["Connection"] = "close" + if client_host = env.params.query["host"]? + sched.del_host2queues(client_host) + + "Done" + else + "No yet!" + end + end + # client(runner) report job's status # /~lkp/cgi-bin/lkp-jobfile-append-var # ?job_file=/lkp/scheduled/job.yaml&job_state=running&job_id=10 -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] auto group jobs with user-defined template
by Lu Weitao 02 Nov '20

02 Nov '20

auto group jobs with user-defined template to support template compare mode background: before compare, create job-groups for all jobs that query from ES with user's filter then, create matrices-groups according job-groups finally, compare each group matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 121 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 120 insertions(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..3c6de7a 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,122 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } +# + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list) + get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + next unless job['stats'] + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + new_job = get_new_job_by_metrics(job, metrics) + next if new_job.empty? + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + if job.key?(param) + group_key_list << job[param] + next + end + job.each_value do |v| + if v.is_a?(Hash) && v.key?(param) + group_key_list << v[param] + break + end + end + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job_by_metrics(job, metrics) + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0

3 7

[PATCH v3 compass-ci] LICENSES: add THIRD PARTY OPEN SOURCE SOFTWARE NOTICE
by Lu Kaiyi 02 Nov '20

02 Nov '20

add THIRD PARTY OPEN SOURCE SOFTWARE NOTICE for references. the directory structure of LICENSES as below: . ├── CCBY-4.0 ├── coreutils │ ├── coreutils │ ├── files │ └── GPL-3.0 -> ../GPL-3.0 ├── docker-sshd │ ├── docker-sshd │ ├── files │ └── MIT -> ../MIT ├── dracut │ ├── dracut │ ├── files │ └── GPL-2.0 -> ../GPL-2.0 ├── GPL-2.0 ├── GPL-3.0 ├── kemal │ ├── files │ ├── kemal │ └── MIT -> ../MIT ├── lkp-tests │ ├── files │ ├── GPL-2.0 -> ../GPL-2.0 │ └── lkp-tests ├── MIT ├── mritd.github.io │ ├── files │ ├── MIT -> ../MIT │ └── mritd.github.io ├── MulanPSL-2.0 └── Scout ├── files ├── Scout └── MIT -> ../MIT Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- LICENSES/Scout/MIT | 1 + LICENSES/Scout/Scout | 17 +++++++++++++++++ LICENSES/Scout/files | 1 + LICENSES/coreutils/GPL-3.0 | 1 + LICENSES/coreutils/coreutils | 17 +++++++++++++++++ LICENSES/coreutils/files | 1 + LICENSES/docker-sshd/MIT | 1 + LICENSES/docker-sshd/docker-sshd | 17 +++++++++++++++++ LICENSES/docker-sshd/files | 1 + LICENSES/dracut/GPL-2.0 | 1 + LICENSES/dracut/dracut | 18 ++++++++++++++++++ LICENSES/dracut/files | 1 + LICENSES/kemal/MIT | 1 + LICENSES/kemal/files | 1 + LICENSES/kemal/kemal | 17 +++++++++++++++++ LICENSES/lkp-tests/GPL-2.0 | 1 + LICENSES/lkp-tests/files | 8 ++++++++ LICENSES/lkp-tests/lkp-tests | 18 ++++++++++++++++++ LICENSES/mritd.github.io/MIT | 1 + LICENSES/mritd.github.io/files | 1 + LICENSES/mritd.github.io/mritd.github.io | 17 +++++++++++++++++ 21 files changed, 142 insertions(+) create mode 120000 LICENSES/Scout/MIT create mode 100644 LICENSES/Scout/Scout create mode 100644 LICENSES/Scout/files create mode 120000 LICENSES/coreutils/GPL-3.0 create mode 100644 LICENSES/coreutils/coreutils create mode 100644 LICENSES/coreutils/files create mode 120000 LICENSES/docker-sshd/MIT create mode 100644 LICENSES/docker-sshd/docker-sshd create mode 100644 LICENSES/docker-sshd/files create mode 120000 LICENSES/dracut/GPL-2.0 create mode 100644 LICENSES/dracut/dracut create mode 100644 LICENSES/dracut/files create mode 120000 LICENSES/kemal/MIT create mode 100644 LICENSES/kemal/files create mode 100644 LICENSES/kemal/kemal create mode 120000 LICENSES/lkp-tests/GPL-2.0 create mode 100644 LICENSES/lkp-tests/files create mode 100644 LICENSES/lkp-tests/lkp-tests create mode 120000 LICENSES/mritd.github.io/MIT create mode 100644 LICENSES/mritd.github.io/files create mode 100644 LICENSES/mritd.github.io/mritd.github.io diff --git a/LICENSES/Scout/MIT b/LICENSES/Scout/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/Scout/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/Scout/Scout b/LICENSES/Scout/Scout new file mode 100644 index 0000000..6844406 --- /dev/null +++ b/LICENSES/Scout/Scout @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: Scout db170ce4ff160c4a2c30483e9410df1773cf572c + +Copyright notice: Copyright(c) 2010 Menno van Slooten, http://mennovanslooten.nl/ +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/Scout/files b/LICENSES/Scout/files new file mode 100644 index 0000000..e120df8 --- /dev/null +++ b/LICENSES/Scout/files @@ -0,0 +1 @@ +/compass-ci/lib/themes.rb diff --git a/LICENSES/coreutils/GPL-3.0 b/LICENSES/coreutils/GPL-3.0 new file mode 120000 index 0000000..481ee24 --- /dev/null +++ b/LICENSES/coreutils/GPL-3.0 @@ -0,0 +1 @@ +../GPL-3.0 \ No newline at end of file diff --git a/LICENSES/coreutils/coreutils b/LICENSES/coreutils/coreutils new file mode 100644 index 0000000..1250a6f --- /dev/null +++ b/LICENSES/coreutils/coreutils @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: coreutils 6.9.89 + +Copyright notice: Copyright(C) 2007 Free Software Foundation, Inc. <https://fsf.org/> +License: The GPL-3.0 License +License-Text: please refer to file GPL-3.0 diff --git a/LICENSES/coreutils/files b/LICENSES/coreutils/files new file mode 100644 index 0000000..8a2b690 --- /dev/null +++ b/LICENSES/coreutils/files @@ -0,0 +1 @@ +/compass-ci/container/os-nfs/entrypoint.sh diff --git a/LICENSES/docker-sshd/MIT b/LICENSES/docker-sshd/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/docker-sshd/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/docker-sshd/docker-sshd b/LICENSES/docker-sshd/docker-sshd new file mode 100644 index 0000000..603f668 --- /dev/null +++ b/LICENSES/docker-sshd/docker-sshd @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: docker-sshd 729faee00bcf0eff5f80059cd92aa425505820f0 + +Copyright notice: Copyright(c) 2015-2020 Volt Grid Pty Ltd +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/docker-sshd/files b/LICENSES/docker-sshd/files new file mode 100644 index 0000000..018684c --- /dev/null +++ b/LICENSES/docker-sshd/files @@ -0,0 +1 @@ +/compass-ci/container/ssh-r/entry.sh diff --git a/LICENSES/dracut/GPL-2.0 b/LICENSES/dracut/GPL-2.0 new file mode 120000 index 0000000..9824569 --- /dev/null +++ b/LICENSES/dracut/GPL-2.0 @@ -0,0 +1 @@ +../GPL-2.0 \ No newline at end of file diff --git a/LICENSES/dracut/dracut b/LICENSES/dracut/dracut new file mode 100644 index 0000000..f09df60 --- /dev/null +++ b/LICENSES/dracut/dracut @@ -0,0 +1,18 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: dracut RHEL-7.1 + +Copyright notice: Copyright(C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, +Fifth Floor, Boston, MA 02110-1301 USA. +License: The GPL-2.0 License +License-Text: please refer to file GPL-2.0 diff --git a/LICENSES/dracut/files b/LICENSES/dracut/files new file mode 100644 index 0000000..78235de --- /dev/null +++ b/LICENSES/dracut/files @@ -0,0 +1 @@ +/compass-ci/container/dracut-initrd/bin/cifs-lib.sh diff --git a/LICENSES/kemal/MIT b/LICENSES/kemal/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/kemal/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/kemal/files b/LICENSES/kemal/files new file mode 100644 index 0000000..71a8cab --- /dev/null +++ b/LICENSES/kemal/files @@ -0,0 +1 @@ +/compass-ci/src/spec/scheduler/boot_spec.cr diff --git a/LICENSES/kemal/kemal b/LICENSES/kemal/kemal new file mode 100644 index 0000000..e08255f --- /dev/null +++ b/LICENSES/kemal/kemal @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: kemal 0.17.0 + +Copyright notice: Copyright(c) 2016 Serdar Doğruyol +License: The MIT License +License-Text: please refer to file MIT diff --git a/LICENSES/lkp-tests/GPL-2.0 b/LICENSES/lkp-tests/GPL-2.0 new file mode 120000 index 0000000..9824569 --- /dev/null +++ b/LICENSES/lkp-tests/GPL-2.0 @@ -0,0 +1 @@ +../GPL-2.0 \ No newline at end of file diff --git a/LICENSES/lkp-tests/files b/LICENSES/lkp-tests/files new file mode 100644 index 0000000..fde5c05 --- /dev/null +++ b/LICENSES/lkp-tests/files @@ -0,0 +1,8 @@ +/compass-ci/sbin/compare +/compass-ci/providers/multi-docker +/compass-ci/container/open-scheduler/build-depends +/compass-ci/user-client/jobs/iperf-pxe.yaml +/compass-ci/user-client/jobs/iperf-vm.yaml +/compass-ci/src/features/jobs/right_iperf.yaml +/compass-ci/lib/matrix2.rb +/compass-ci/sbin/es-find diff --git a/LICENSES/lkp-tests/lkp-tests b/LICENSES/lkp-tests/lkp-tests new file mode 100644 index 0000000..3f40d5f --- /dev/null +++ b/LICENSES/lkp-tests/lkp-tests @@ -0,0 +1,18 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: lkp-tests fcb47ccfc50c1f4da388aed9596c0acaac04a917 + +Copyright notice: Copyright(C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, +Fifth Floor, Boston, MA 02110-1301 USA +License: The GPL-2.0 License +License-Text: please refer to file GPL-2.0 diff --git a/LICENSES/mritd.github.io/MIT b/LICENSES/mritd.github.io/MIT new file mode 120000 index 0000000..91e6d98 --- /dev/null +++ b/LICENSES/mritd.github.io/MIT @@ -0,0 +1 @@ +../MIT \ No newline at end of file diff --git a/LICENSES/mritd.github.io/files b/LICENSES/mritd.github.io/files new file mode 100644 index 0000000..38b67c1 --- /dev/null +++ b/LICENSES/mritd.github.io/files @@ -0,0 +1 @@ +/compass-ci/container/registry/config.yml diff --git a/LICENSES/mritd.github.io/mritd.github.io b/LICENSES/mritd.github.io/mritd.github.io new file mode 100644 index 0000000..ee22568 --- /dev/null +++ b/LICENSES/mritd.github.io/mritd.github.io @@ -0,0 +1,17 @@ +THIRD PARTY OPEN SOURCE SOFTWARE NOTICE + +Please note we provide an open source software notice for the third party open source software along with +this software and/or this software component contributed by Huawei (in the following just "this SOFTWARE"). +The open source software licenses are granted by respective right holders. + +Warranty Disclaimer +THE OPEN SOURCE SOFTWARE IN THIS SOFTWARE IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, +BUT WITHOUT ANY WARRANTY, WITHOUT EVEN THE IMPLIED WARRANTY OF MERCHANTABILITY OR FITNESS +FOR A PARTICULAR PURPOSE. SEE THE APPLICABLE LICENSES FOR MORE DETAILS. + +Copyright Notice and License Texts +Software: mritd.github.io ed4350e91e9b76078a850ede8302854d3393f493 + +Copyright notice: Copyright(c) 2013-2016 Blackrock Digital LLC. +License: The MIT License +License-Text: please refer to file MIT -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/*: fix cci job dir layout
by Wang Yong 02 Nov '20

02 Nov '20

[Why] for multi version support, cgz file for cci-makepkg need change dir layout, and change cci-depends dir layout together to easily update scheduler code path: /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/ => /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/${pkgname}/ filename: cci-makepkg ${pkgver}-${pkgrel}.cgz latest.cgz -> ${pkgver}-${pkgrel}.cgz filename: cci-depends stay original ${benchmark}.cgz ${benchmark}.cgz -> ${benchmark}_${date}.cgz E.g. output: cci-depends stay original $ tree iperf ├── iperf_20201102.cgz └── iperf.cgz -> iperf_20201102.cgz output: cci-makepkg $ tree sar ├── git-1.cgz └── latest.cgz -> git-1.cgz Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 2 +- tests/cci-makepkg | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/cci-depends b/tests/cci-depends index 36923ea4d..efa9a025b 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -18,7 +18,7 @@ DEPS_MNT=/initrd/deps && mkdir -p "$DEPS_MNT" DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} pack_arch=$os_arch [[ "$benchmark" = "${benchmark##*.}" ]] || diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9d5082629..9551ebfe2 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -26,7 +26,7 @@ mkdir -p "$PKG_MNT" export DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} . $LKP_SRC/distro/${DISTRO} . $LKP_SRC/lib/install.sh @@ -65,8 +65,8 @@ update_shared_pkg() [ "$bm_name" = "$benchmark" ] && return # benchmark is a symlink - ln -sf "$bm_link" "$sync_dest/${benchmark}.cgz" || return - echo "update shared pkg link ${benchmark}.cgz -> $bm_link" + ln -sf latest.cgz "$sync_dest/${benchmark}.cgz" || return + echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } distro_install_depends lkp-dev @@ -86,7 +86,7 @@ date=$(date +"%Y%m%d") pkgver=$(get_pkg_info pkgver) pkgrel=$(get_pkg_info pkgrel) bm_name=$(check_shared_pkg) -cgz_name="${bm_name}-${pkgver:-0}-${pkgrel:-0}_${date}.cgz" +cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy @@ -98,8 +98,8 @@ update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return - ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/${bm_name}.cgz" || return - echo "create package: $sync_dest/${bm_name}.cgz -> $(realpath $sync_dest/$cgz_name)" + ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/latest.cgz" || return + echo "create package: $sync_dest/latest.cgz -> $(realpath $sync_dest/$cgz_name)" update_shared_pkg "${bm_name}.cgz" -- 2.23.0

1 0

[PATCH compass-ci] container: fix failed to build archlinux images
by Liu Yinsi 02 Nov '20

02 Nov '20

[why] when build archlinux images in x86_64 machine error: error: failed retrieving file 'core.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update core (failed to retrieve some files) error: failed retrieving file 'extra.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Resolving timed out after 10000 milliseconds error: failed to update extra (download library error) error: failed retrieving file 'community.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update community (failed to retrieve some files) error: failed to synchronize all databases The command '/bin/sh -c pacman --needed --noprogressbar --noconfirm -Syu && pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk' returned a non-zero code: 1 becaue archlinux mirror not support x86_64 machine. [how] when deploy in x86_64 machine, change mirror http://mirrors.tuna.tsinghua.edu.cn/archlinuxarm/$arch/$repo => http://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/archlinux/build | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/archlinux/build b/container/archlinux/build index 81feda2..9194f4e 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +[ "$(arch)" == "x86_64" ] && + sed -i 's/archlinuxarm\/$arch\/$repo/archlinux\/$repo\/os\/$arch/g' root/etc/pacman.d/mirrorlist + docker build -t archlinux:testbed . -- 2.23.0

2 2

[PATCH compass-ci 2/2] container/assign-account: answerback-email.rb
by Luan Shengde 02 Nov '20

02 Nov '20

1. disable use golbal variable apply_info [why] Style/GlobalVars: Do not introduce global variables. [how] use local variables instead of global variables 2. enable transfer user info when execute apply account command [why] when applying account, the assign-account service will write the user info to user's default config file ~/.config/compass-ci/default/${USER}.yaml user info: - my_email - my_name - my_uuid - lab [how] transfer the user info along with the pub_key to assign-account service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 60 ++++++++++++-------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..4227081 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -18,29 +18,31 @@ names = Set.new %w[ JUMPER_PORT SEND_MAIL_HOST_INTERNET SEND_MAIL_PORT_INTERNET + LAB ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = defaults['LAB'] || z9 -$apply_info = { +apply_info = { 'my_email' => nil, + 'my_name' => nil, 'my_ssh_pubkey' => nil } -def init_info(email_file) - mail_content = Mail.read(email_file) +def init_info(mail_content, apply_info) + apply_info['my_email'] = mail_content.from[0] + apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded.gsub(/\r|\n/, '') + end - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + apply_info end options = OptionParser.new do |opts| @@ -52,15 +54,18 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + apply_info['my_email'] = email_address + # when apply account with email address, will not get the user name + apply_info['my_name'] = '' end opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + apply_info['my_ssh_pubkey'] = File.read(pub_key_file) end opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + mail_content = Mail.read(email_file) + init_info(mail_content, apply_info) end opts.on_tail('-h|--help', 'show this message') do @@ -94,26 +99,31 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def account_info(user_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{user_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(apply_info) message = "No email address specified\n" message += "use -e email_address add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? + raise message if apply_info['my_email'].nil? + + user_info = { + 'my_email': apply_info['my_email'], + 'my_name': apply_info['my_name'], + # for manually assign account, there will be no my_uuid specified + 'my_uuid': '', + 'lab': LAB, + 'pub_key': apply_info['my_ssh_pubkey'] + } - acct_info = account_info($apply_info['my_ssh_pubkey']) + acct_info = account_info(user_info) - message = build_message($apply_info['my_email'], acct_info) + message = build_message(apply_info['my_email'], acct_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +send_account(apply_info) -- 2.23.0

4 6

[PATCH v6 compass-ci 3/3] kernel_version.md: explain key "kernel_version"
by Xu Xijian 02 Nov '20

02 Nov '20

[why] Explain the meaning of new key "kernel_version" for scheduler, including what's the mapping between its typical values and actual files in the disk and how to set it. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 doc/job/kernel_version.md diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md new file mode 100644 index 0000000..56a6b1a --- /dev/null +++ b/doc/job/kernel_version.md @@ -0,0 +1,32 @@ +# kernel_version + +Meaning: +- Every os has its kernel, however an os can start with different kernels according to different need. +- kernel_version is a key for users to specify a kernel version. +- If kernel_version is not given by users, it will use the default one. + +Related files: +- In initramfs boot process, every kernel version is related with a vmlinuz, module and headers. +- File paths like below, An example $boot_dir can be "/srv/os/openeuler/aarch64/20.03/boot" + +- $boot_dir/vmlinuz => $boot_dir/vmlinuz-$kernel_version + - default kernel file(vmlinuz) is a soft link + +- $boot_dir/modules => $boot_dir/modules-$kernel_version.cgz + - default modules is a soft link + +- $boot_dir/headers => $boot_dir/headers-$kernel_version.cgz + - default headers is a soft link + +- $boot_dir/vmlinuz-$kernel_version + - real kernel path + +- $boot_dir/modules-$kernel_version.cgz + - real modules path + +- $boot_dir/headers-$kernel_version.cgz + - real headers path + + +Usage example: +- submit iperf.yaml testbox=vm-hi1620-2p8g--xxj os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 -- 2.23.0

2 2

[PATCH compass-ci 1/2] container/assign-account: get_account_info add new function
by Luan Shengde 02 Nov '20

02 Nov '20

[why]: esaier for user to config the default yaml file [how]: when get request data, parse the data and write the user infos to default yaml file: ~/.config/compass-ci/default/${USER}.yaml include: - my_email - my_name - my_uuid - lab Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/get_account_info.rb | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..4e61bdc 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -111,9 +113,23 @@ class AccountStorage } setup_authorized_key(account, pub_key) + setup_default_yaml(account) if @data.key? 'my_email' + return jumper_account_info end + def setup_default_yaml(account) + default_yaml_dir = File.join('/home', account, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + File.open("#{default_yaml_dir}/#{account}.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" if @data.key? 'my_name' + file.puts "my_uuid: #{@data['my_uuid']}" if @data.key? 'my_uuid' + file.puts "lab: #{@data['lab']}" if @data.key? 'lab' + file.close + end + end + def setup_authorized_key(account, pub_key) ssh_dir = File.join('/home/', account, '.ssh') Dir.mkdir ssh_dir, 0o700 -- 2.23.0

6 10

[PATCH lkp-tests] tests/*: fix cci job dir layout
by Wang Yong 02 Nov '20

02 Nov '20

[Why] for multi version support, cgz file for cci-makepkg need change dir layout, and change cci-depends dir layout together to easily update scheduler code path: /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/ => /srv/initrd/pkg/${os_mount}/${os}/${os_arch}/${os_version}/${pkgname}/ filename: cci-makepkg ${pkgver}-${pkgrel}.cgz latest.cgz -> ${pkgver}-${pkgrel}.cgz filename: cci-depends ${benchmark}.cgz latest.cgz -> ${benchmark}.cgz E.g. output: cci-depends $ tree iperf ├── iperf.cgz └── latest.cgz -> iperf.cgz output: cci-makepkg $ tree sar ├── git-1.cgz └── latest.cgz -> git-1.cgz Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/aliyun | 7 +++---- distro/clear | 7 +++---- distro/debian | 8 +++----- tests/cci-depends | 4 ++-- tests/cci-makepkg | 13 ++++++------- 5 files changed, 17 insertions(+), 22 deletions(-) diff --git a/distro/aliyun b/distro/aliyun index 79656f0e4..a6ca49968 100755 --- a/distro/aliyun +++ b/distro/aliyun @@ -80,7 +80,6 @@ yum_download() yum_pack() { local target_dir=/opt/rpms - local date=$(date +"%Y%m%d") local downloaded_rpms=$(find /var/ -type f -name "*.rpm") @@ -90,10 +89,10 @@ yum_pack() mv $downloaded_rpms $target_dir - find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz + find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}.cgz - ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return - chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + ln -sf ${BM_NAME}.cgz $pack_to/latest.cgz || return + chown .lkp $pack_to/${BM_NAME}.cgz $pack_to/latest.cgz || return echo package installed to $pack_to/${BM_NAME}.cgz ls $target_dir/*.rpm > $pack_to/.${BM_NAME}.packages diff --git a/distro/clear b/distro/clear index f21de89c3..3b562aac6 100755 --- a/distro/clear +++ b/distro/clear @@ -121,7 +121,6 @@ list_bundles_files() pack_bundles_files() { - local date=$(date +"%Y%m%d") # allow empty ls $tmpdir | grep -q "\.list$" >/dev/null || { echo "empty PACKAGE_LIST for $BM_NAME" @@ -130,10 +129,10 @@ pack_bundles_files() cat $tmpdir/*.list | sort | uniq | cpio --quiet -o -H newc --owner=root.root | - gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz || return + gzip -n -9 >$pack_to/${BM_NAME}.cgz || return - ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return - chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + ln -sf ${BM_NAME}.cgz $pack_to/latest.cgz || return + chown .lkp $pack_to/${BM_NAME}.cgz $pack_to/latest.cgz || return echo package installed to $pack_to/${BM_NAME}.cgz } diff --git a/distro/debian b/distro/debian index 13c4e983a..270f77e01 100755 --- a/distro/debian +++ b/distro/debian @@ -99,8 +99,6 @@ fixup_perf() pack() { - local date=$(date +"%Y%m%d") - [[ $(type -t fixup_$BM_NAME) = 'function' ]] && fixup_$BM_NAME { @@ -112,12 +110,12 @@ pack() cat $LKP_SRC/distro/$DISTRO/$BM_NAME-allowlist } | cpio --quiet -o -H newc --owner=root.root | - gzip -n -9 > $pack_to/${BM_NAME}_$date.cgz || return + gzip -n -9 > $pack_to/${BM_NAME}.cgz || return - ln -sf ${BM_NAME}_$date.cgz \ + ln -sf ${BM_NAME}.cgz \ $pack_to/${BM_NAME}.cgz || return - chown .lkp $pack_to/${BM_NAME}_$date.cgz \ + chown .lkp $pack_to/${BM_NAME}.cgz \ $pack_to/${BM_NAME}.cgz || return echo package installed to $pack_to/${BM_NAME}.cgz diff --git a/tests/cci-depends b/tests/cci-depends index 36923ea4d..1c840a4ee 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -18,7 +18,7 @@ DEPS_MNT=/initrd/deps && mkdir -p "$DEPS_MNT" DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${DEPS_MNT}/${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} pack_arch=$os_arch [[ "$benchmark" = "${benchmark##*.}" ]] || @@ -38,7 +38,7 @@ umask 002 } check_shared_package "$benchmark" && { - ln -sf $BM_NAME.cgz $pack_to/$benchmark.cgz + ln -sf latest.cgz $pack_to/$benchmark.cgz echo "create symlink for shared package: $benchmark -> $BM_NAME" exit 0 diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9d5082629..115136f61 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -26,7 +26,7 @@ mkdir -p "$PKG_MNT" export DISTRO=${os} [[ "$os_mount" = "cifs" ]] && os_mount="nfs" -pack_to=${os_mount}/${os}/${os_arch}/${os_version} +pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} . $LKP_SRC/distro/${DISTRO} . $LKP_SRC/lib/install.sh @@ -65,8 +65,8 @@ update_shared_pkg() [ "$bm_name" = "$benchmark" ] && return # benchmark is a symlink - ln -sf "$bm_link" "$sync_dest/${benchmark}.cgz" || return - echo "update shared pkg link ${benchmark}.cgz -> $bm_link" + ln -sf latest.cgz "$sync_dest/${benchmark}.cgz" || return + echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } distro_install_depends lkp-dev @@ -82,11 +82,10 @@ distro_install_depends ${benchmark}-dev build_depends_pkg -i $benchmark $TMP sync_dest="$PKG_MNT/$pack_to" -date=$(date +"%Y%m%d") pkgver=$(get_pkg_info pkgver) pkgrel=$(get_pkg_info pkgrel) bm_name=$(check_shared_pkg) -cgz_name="${bm_name}-${pkgver:-0}-${pkgrel:-0}_${date}.cgz" +cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy @@ -98,8 +97,8 @@ update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return - ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/${bm_name}.cgz" || return - echo "create package: $sync_dest/${bm_name}.cgz -> $(realpath $sync_dest/$cgz_name)" + ln -sf "$(basename $(realpath $sync_dest/$cgz_name))" "$sync_dest/latest.cgz" || return + echo "create package: $sync_dest/latest.cgz -> $(realpath $sync_dest/$cgz_name)" update_shared_pkg "${bm_name}.cgz" -- 2.23.0

4 5

[PATCH compass-ci] area: fix src/lib/web_backend.rb "/compare_candidates" tbox_group regex error
by Zhang Yuhang 02 Nov '20

02 Nov '20

[error info] 1. /\d+$/ will be matched, such as "xxx123". 2. `index = "xxx123".index('--') || "xxx123".rindex('-')` - index equal nil. 3. `r = r[0, nil]` - raise a TypeError. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index c3f6bef..55c5e4a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -47,7 +47,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end -- 2.23.0

2 2

[PATCH v4 compass-ci] container/logging-es: optimize bulk write performance
by Wu Zhende 02 Nov '20

02 Nov '20

[Why] fluentd uses "bulk" to continuously write data to es. The es write capability needs to be optimized to adapt to high-concurrency scenarios. Before: fluent.info: POST http://172.17.0.50:9200/_bulk [status:200, request:0.116s, query:0.114s] After: fluent.info: POST http://172.17.0.50:9200/_bulk [status:200, request:0.060s, query:0.057s] [How] 1.Increase index.refresh_interval 2.Set index.translog.durability to Async 3.Increase the JVM memory 4.Increase write.queue_size 5.Increase index_buffer_size [Reference Link] https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-ind… https://ravishajava.com/2018/07/13/elastic-search-performance-tuning-parmet… https://thoughts.t37.net/designing-the-perfect-elasticsearch-cluster-the-al… https://thehoard.blog/techniques-to-achieve-high-write-throughput-with-elas… Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/logging-es/Dockerfile | 12 ++++++++---- container/logging-es/start | 12 ++++++++++++ 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 7f91f65..707299f 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -7,10 +7,14 @@ FROM gagara/elasticsearch-oss-arm64:7.6.2 MAINTAINER Wu Zhende <wuzhende666(a)163.com> -RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml; -RUN sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml; +RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a path.data: /srv/es/logging-es' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/start b/container/logging-es/start index 9be7a5b..25ed36b 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -19,3 +19,15 @@ cmd=( ) "${cmd[@]}" + +echo "Please wait 60s for setting index" +sleep 60 + +curl -H "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' +{ + "index": { + "refresh_interval": "120s", + "translog.durability": "async", + "translog.sync_interval": "120s", + "translog.flush_threshold_size": "1024mb"} +}' -- 2.23.0

1 0

[PATCH v7 compass-ci 1/5] container/ntp-server: Dockerfile
by Wang Chenglong 01 Nov '20

01 Nov '20

[What] The docker file for build a container to provide network time synchronization service for virtual and physical machines. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/ntp-server/Dockerfile | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/ntp-server/Dockerfile diff --git a/container/ntp-server/Dockerfile b/container/ntp-server/Dockerfile new file mode 100644 index 0000000..76f2a0e --- /dev/null +++ b/container/ntp-server/Dockerfile @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM centos:7 +MAINTAINER Wang Chenglong <18509160991(a)163.com> + +RUN yum repolist +RUN yum -y install ntp && \ + yum clean all +RUN ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime + +COPY ntp.conf /etc/ntp.conf +COPY start-server.sh / + +EXPOSE 123 + +ENTRYPOINT ["/start-server.sh"] +CMD ["sh"] -- 2.23.0

3 3

[PATCH v5 lkp-tests 2/2] tests/iozone: add specified parameter for iozone test
by Lu Kaiyi 01 Nov '20

01 Nov '20

[why] need to test write, rewrite, read, reread, random write, random read for different block_size. [how] add judement for variable $block_size to execute iozone command with specificed parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 40 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/tests/iozone b/tests/iozone index 3095ce80..88a92a18 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,13 +1,49 @@ #!/bin/sh - +# - block_size +# - file_size +# - write_rewrite +# - read_reread +# - random_read_write +# - read_backwards +# - rewrite_record +# - stride_read +# - fwrite_refwrite +# - fread_refread +# - random_mix +# - pwrite_repwrite +# - pread_repread +# - pwritev_repwritev +# - preadv_repreadv ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh + +args="iozone" +if [ -n "$block_size" ]; then + args+=" -r $block_size" + [ -n "$file_size" ] && args+=" -s $file_size" + [ -n "$write_rewrite" ] && args+=" -i 0" + [ -n "$read_reread" ] && args+=" -i 1" + [ -n "$random_read_write" ] && args+=" -i 2" + [ -n "$read_backwards" ] && args+=" -i 3" + [ -n "$rewrite_record" ] && args+=" -i 4" + [ -n "$stride_read" ] && args+=" -i 5" + [ -n "$fwrite_refwrite" ] && args+=" -i 6" + [ -n "$fread_refread" ] && args+=" -i 7" + [ -n "$random_mix" ] && args+=" -i 8" + [ -n "$pwrite_repwrite" ] && args+=" -i 9" + [ -n "$pread_repread" ] && args+=" -i 10" + [ -n "$pwritev_repwritev" ] && args+=" -i 11" + [ -n "$preadv_repreadv" ] && args+=" -i 12" +else + args+=" -a" +fi + for mnt in $mount_points do - cmd="iozone -a -f $mnt/iozonetest" + cmd="${args} -f $mnt/iozonetest" log_echo "$cmd &" $cmd > $TMP/iozone-$i & i=$((i+1)) -- 2.23.0

2 1

2025

2024

2023

2022

2021

2020

Compass-ci November 2020