- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 05 Nov '20

05 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe..bb20a845 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -37,6 +37,21 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +[ -n "$PKGBUILD_TAG" ] && { + PKGBUILD_TAG="-$PKGBUILD_TAG" + PKGBUILD_TAG_FILE="PKGBUILD$PKGBUILD_TAG" + get_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 -- 2.23.0

2 2

[PATCH v2 compass-ci 05/10] container/mail-robot: send-apply-account-email.rb
by Luan Shengde 05 Nov '20

05 Nov '20

SendApplyAccountEmail send uuid email build email message send email send error email build error email message send error email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/send-apply-account-email.rb | 72 +++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100755 container/mail-robot/send-apply-account-email.rb diff --git a/container/mail-robot/send-apply-account-email.rb b/container/mail-robot/send-apply-account-email.rb new file mode 100755 index 0000000..f29ff1e --- /dev/null +++ b/container/mail-robot/send-apply-account-email.rb @@ -0,0 +1,72 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# build mail data and send mail +class SendApplyAccountEmail + def initialize(send_mail_info) + @send_mail_host = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + @send_mail_info = send_mail_info + end + + def send_uuid_email + email_msg = <<~EMAIL_MESSAGE + To: #{@send_mail_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{@send_mail_info['my_name']}, + + Thank you for joining us. + You can use the following info to submit jobs: + + 1) setup default config + run the following command to add the below setup to default config file + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/\\${USER}.yaml <<-EOF + my_email: #{@send_mail_info['my_email']} + my_name: #{@send_mail_info['my_name']} + my_uuid: #{@send_mail_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + find how to write job yaml + submit jobs/netperf.yaml + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_error_email + email_msg = <<~EMAIL_MESSAGE + To: #{@send_mail_info['my_email']} + Subject: apply account failed + + Dear #{@send_mail_info['my_name']} + + Your application for account failed with following errors + + #{@send_mail_info['error_message']} + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_mail(email_msg) + %x(curl -XPOST "#{@send_mail_host}:#{ENV['SEND_MAIL_PORT']}/send_mail_text" -d "#{email_msg}") + end +end -- 2.23.0

3 5

[PATCH v4 lkp-tests] job2sh.rb: define on_fail
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh export_top_envexport_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7e6a5d7c..370c7899 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -358,6 +360,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

1 0

[PATCH compass-ci] qcow2rootfs: modify the location of the vmlinuz soft link of rootfs
by Yu Chuan 05 Nov '20

05 Nov '20

[Why] Due to the changes in the layout of rootfs, so modify the location of the vmlinuz soft link when generating rootfs. [How] Before: ${ROOTFS_DIR} └── vmlinuz -> boot/vmlinuz-xxx After: ${ROOTFS_DIR}/boot/ └── vmlinuz -> vmlinuz-xxx In addition: ROOTFS_VMLINUZ=$(basename "$ROOTFS_VMLINUZ_FILE"). Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/qcow2rootfs/bin/common | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index 0d109e39c3fc..da39f523e2c7 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -174,8 +174,9 @@ create_links_vmlinuz_initrd() { bash -c " cd $EXTRACT_ROOT/rootfs-dir chmod a+r $ROOTFS_INITRD_LKP - ln -sf $ROOTFS_VMLINUZ_FILE vmlinuz ln -sf $ROOTFS_INITRD_LKP initrd.lkp + cd $EXTRACT_ROOT/rootfs-dir/boot + ln -sf $ROOTFS_VMLINUZ vmlinuz " ) if "${cmds[@]}" ; then -- 2.23.0

1 0

[PATCH lkp-tests] fio: fix error when test with write mode
by Wei Jihui 05 Nov '20

05 Nov '20

[Why] fio need test on disk, so mount a disk by: "mount -t xfs -o inode64 /dev/sda /fs/sda" when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 1 + 1 file changed, 1 insertion(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..3ea79fa2 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -95,6 +95,7 @@ fallocate=$fallocate io_size=$io_size file_service_type=$file_service_type random_distribution=$random_distribution +allow_mounted_write=1 group_reporting create_only=1 pre_read=$pre_read -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: change the way get hosts file
by Li Ping 05 Nov '20

05 Nov '20

[why] we can get hosts file from the repo lab-z9 or get hosts file from lkp-tests/hosts before: get hosts file from lab-z9, but the hosts file not exist in lkp-tests/hosts the job will puts the error message: hosts_file not exist ... but there is no need to puts such error message, if we have got the hosts file from lab-z9 --- lib/job.rb | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 995b38bf..254562f7 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,20 +334,14 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + return lab_hosts_file if lab_hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.file?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + puts("hosts_file not exist: #{hosts_file_name}, maybe need check testbox field") end def include_files -- 2.23.0

3 3

[PATCH v2 compass-ci 2/2] add template compare mode
by Lu Weitao 05 Nov '20

05 Nov '20

add template compare mode in sbin/compare(command-line tool) by: - load and parse compare.yaml - calls lib/es_query.rb, query jobs data set from ES - calls lib/matrix.rb, convert jobs data set to group matrices - calls lib/compare_matrixes.rb, use groups matrices to compare background: To support compare with user-defined template feature, the work-flow of user-defined template feature: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results example: compare -t/--template compare_template.yaml, the yaml like: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s output:{ "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "bs|test_size", "tables": { "4k|1G": { "fio.write_iops": { "average": { "openeuler 20.03": 308298.5524761111, "debian sid": 220289.07563 }, "standard_deviation": { "openeuler 20.03": null, "debian sid": null }, "change": { "debian sid vs openeuler 20.03": -28.5 } } }, "16k|1G": {...}, ... } } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++ sbin/compare | 10 +++++++++- 2 files changed, 60 insertions(+), 1 deletion(-) diff --git a/lib/compare.rb b/lib/compare.rb index 2837edc..9edf049 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -6,6 +6,7 @@ require_relative './es_query.rb' require_relative './matrix2.rb' require_relative './compare_matrixes.rb' require_relative './constants.rb' +require 'yaml' # ------------------------------------------------------------------------------------------- # compare_matrices_list @@ -66,3 +67,53 @@ def create_groups_matrices_list(conditions, dims) query_results = es.multi_field_query(conditions) combine_group_query_data(query_results, dims) end + +# ------------------------------------------------------------------------------------------- +# compare with user-defined compare_template.yaml +# the work-flow are: +# load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list +# ---> create groups_matrices ---> compare_values by each metrics ---> format/show results +# + +def compare_by_template(template) + template_params = load_template(template) + groups_matrices = create_groups_matrices(template_params) + compare_values = compare_metrics_values(groups_matrices) + chart_results = format_for_chart(compare_values, template_params) + show_chart_compare_result(chart_results) +end + +def load_template(template) + unless File.file?(template) + warn 'template does not exist' + exit + end + YAML.load_file(template) +end + +# input: template_params: Hash +# eg: +# { +# "compare_metrics"=>["fio.write_iops", "fio.read_iops"], +# "filter"=>[ +# {"suite"=>["fio-bisic"]}, +# {"os_arch"=>["aarch_64"]} +# ], +# "compare_dimensions"=>[ +# {"os"=>"openeuler", "os_version"=>20.03}, +# {"os"=>"centos", "os_version"=>7.6} +# ], +# "x_params"=>["block_size", "package_size"], +# "title"=>"Hackbench Performance Testing", +# "unit"=>"KB/s" +# } +def create_groups_matrices(template_params) + es = ESQuery.new + query_results = es.multi_field_query(template_params['filter']) + combine_group_jobs_list( + query_results, + template_params['x_params'], + template_params['compare_dimensions'], + template_params['compare_metrics'] + ) +end diff --git a/sbin/compare b/sbin/compare index cf3980c..b13682e 100755 --- a/sbin/compare +++ b/sbin/compare @@ -6,6 +6,7 @@ # Usage: # compare "conditions_1" "conditions_2" ... -c "common_conditions" # compare "conditions" -d "dimensions" +# compare -t template.yaml # Eg: # compare "id=6000,6001" "id=7000,7001" # compare "commit=a12d232e" "commit=b3bacc31" @@ -21,6 +22,7 @@ is_group = false dimensions = nil colorful = nil options = {} +template = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: compare "conditions" ... [option]' @@ -45,6 +47,10 @@ opt_parser = OptionParser.new do |opts| colorful = color end + opts.on('-t', '--template template', 'compare with user-defined template') do |t| + template = t + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -60,7 +66,9 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful -if is_group +if template + compare_by_template(template) +elsif is_group compare_group(argv, dimensions, options) else compare_matrices_list(argv, common_conditions, options) -- 2.23.0

3 4

[PATCH lkp-tests] fio: fix error when test with write mode
by Wei Jihui 05 Nov '20

05 Nov '20

[Why] fio need test on disk, so mount a disk by: "mount -t xfs -o inode64 /dev/sda /fs/sda" when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. [How] set allow_mounted_write=1 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 1 + 1 file changed, 1 insertion(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..3ea79fa2 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -95,6 +95,7 @@ fallocate=$fallocate io_size=$io_size file_service_type=$file_service_type random_distribution=$random_distribution +allow_mounted_write=1 group_reporting create_only=1 pre_read=$pre_read -- 2.23.0

2 1

[PATCH compass-ci 4/6] container/webhook: start
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 container/webhook/start diff --git a/container/webhook/start b/container/webhook/start new file mode 100755 index 0000000..50cbf20 --- /dev/null +++ b/container/webhook/start @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "webhook" + +cmd = %W[ + docker run + --restart=always + --name webhook + -d + -p 11301:11301 + node:webhook +] + +cmd += ['sh', '-c', 'node webhook.js'] +system(*cmd) -- 2.23.0

2 2

[PATCH v2 lkp-tests] libmicro: add PKGBUILD for libmicro
by Bai Jing 05 Nov '20

05 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/libmicro/PKGBUILD | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 pkg/libmicro/PKGBUILD diff --git a/pkg/libmicro/PKGBUILD b/pkg/libmicro/PKGBUILD new file mode 100644 index 00000000..61cd32f4 --- /dev/null +++ b/pkg/libmicro/PKGBUILD @@ -0,0 +1,19 @@ +pkgname=libMicro +pkgver=0.4 +pkgrel=0 +pkgdesc="A portable set of microbenchmarks focused on the syscall and system library interfaces." +arch=('i386' 'x86_64') +url="https://github.com/rzezeski/libMicro" +license=('CDDL') +source=("https://codeload.github.com/redhat-performance/libMicro/zip/0.4.0-rh") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname-$pkgver.$pkgrel-rh" + make +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/libMicro" + cp -r "$srcdir/$pkgname-$pkgver.$pkgrel-rh/"* "$pkgdir/lkp/benchmarks/libMicro" +} -- 2.23.0

4 7

[PATCH lkp-tests] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 05 Nov '20

05 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } PKGBUILD-1.1 contains only the changed part. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..dc8471b9 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -888,7 +888,7 @@ update_pkgver() { exit 1 fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" - source_safe "$BUILDFILE" + source_safe "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -1604,10 +1604,14 @@ cd_safe() { source_safe() { shopt -u extglob - if ! source "$@"; then - error "$(gettext "Failed to source %s")" "$1" - exit 1 - fi + local file + for file in "$@" + do + if ! source "$file"; then + error "$(gettext "Failed to source %s")" "$1" + exit 1 + fi + done shopt -s extglob } @@ -1626,7 +1630,6 @@ merge_arch_attrs() { source_buildfile() { source_safe "$@" - if (( !SOURCEONLY )); then merge_arch_attrs fi @@ -3661,6 +3664,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +PKGBUILD_TAG=$PKGBUILD_TAG if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then @@ -3691,7 +3695,7 @@ else if [[ ${BUILDFILE:0:1} != "/" ]]; then BUILDFILE="$startdir/$BUILDFILE" fi - source_buildfile "$BUILDFILE" + source_buildfile "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

4 4

[PATCH lkp-tests] add sysbench-threads-git.yaml
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] job.yaml transfer required parameters to testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/sysbench-threads-git.yaml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 jobs/sysbench-threads-git.yaml diff --git a/jobs/sysbench-threads-git.yaml b/jobs/sysbench-threads-git.yaml new file mode 100644 index 00000000..6b0f2c00 --- /dev/null +++ b/jobs/sysbench-threads-git.yaml @@ -0,0 +1,21 @@ +suite: sysbench-threads-git +category: benchmark + +nr_threads: +- 2 +- 4 +- 8 +- 16 +- 32 +- 64 +- 128 +- 256 +- 512 +- 1024 +- 2048 +- 4096 +- 8192 + +sysbench-threads-git: + thread_yields: 100 + thread_locks: 2 -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 05 Nov '20

05 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..6d24d8d9 --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,36 @@ +#!/bin/sh +# - runtime +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is an old version, It is both a benchmark and a stress test for the Linux kernel scheduler. +## It's main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes) which communicate via either sockets or pipes and time how long +## it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' + +[ -n "$runtime" ] || runtime=600 +[ -n "$iterations" ] || iterations=10 +[ -n "$loops" ] || loops=100 +[ -n "$nr_threads" ] || nr_threads=1 + +run_hackbench() +{ + # args: groups number, mode [process/thread], times + local hackbench_path + if [ -x /lkp/benchmarks/hackbench-git/hackbench/hackbench ]; then + hackbench_path=/lkp/benchmarks/hackbench-git/hackbench/hackbench + else + echo "The hackbench executable file cannot be found." && exit 1 + fi + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

2 3

[PATCH lkp-tests] add sysbench-threads-git.yaml
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] job.yaml transfer required parameters to testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/sysbench-threads-git.yaml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 jobs/sysbench-threads-git.yaml diff --git a/jobs/sysbench-threads-git.yaml b/jobs/sysbench-threads-git.yaml new file mode 100644 index 00000000..fe7552a4 --- /dev/null +++ b/jobs/sysbench-threads-git.yaml @@ -0,0 +1,21 @@ +suite: sysbench-threads-git +category: benchmark + +nr_threads: +- 2 +- 4 +- 8 +- 16 +- 32 +- 64 +- 128 +- 256 +- 512 +- 1024 +- 2048 +- 4096 +- 8192 + +sysbench-threads: + thread_yields: 100 + thread_locks: 2 -- 2.23.0

2 1

[PATCH v2 compass-ci 03/10] container/mail-robot: parse-apply-account-email.rb
by Luan Shengde 05 Nov '20

05 Nov '20

parse email for apply account check out commit url check commit url exists check base url in upstream-repos check commit available hub: gitee.com clone the repo and check the commit exists hub: non-gitee.com check the feedback of curl command to check the commit url available check out pub key extract the attachment of the pub_key Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/parse-apply-account-email.rb | 135 ++++++++++++++++++ 1 file changed, 135 insertions(+) create mode 100755 container/mail-robot/parse-apply-account-email.rb diff --git a/container/mail-robot/parse-apply-account-email.rb b/container/mail-robot/parse-apply-account-email.rb new file mode 100755 index 0000000..1325c8f --- /dev/null +++ b/container/mail-robot/parse-apply-account-email.rb @@ -0,0 +1,135 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check whether there is a commit url and ssh pub_key in the email +# apply account on junper server +# the entry point is parse_commit_url_pub_key +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + @my_email = mail_content.from[0] + @my_name = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + end + + def parse_commit_url_pub_key + commit_url = check_out_commit_url + pub_key = check_out_pub_key + + return commit_url, pub_key + end + + def send_error_email(error_message) + send_error_email_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'error_message' => error_message + } + send_err_mail = SendApplyAccountEmail.new(send_error_email_info) + send_err_mail.send_error_email + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = 'No matched commit url found' + send_error_email(error_message) + + raise error_message + end + + def check_out_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + check_base_url_in_upstream_repos('/c/upstream-repos', base_url) + check_commit_available(url, base_url) + + return url + end + + def check_base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = 'The url is not in upstream-repo list' + send_error_email(error_message) + + raise error_message + end + + def check_commit_available(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + check_gitee_commit(url, base_url) + else + check_non_gitee_commit(url) + end + end + + def check_non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_email + + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_gitee_commit(url, base_url) + repo_dir = url.split('/')[-3] + repo_url = [base_url, 'git'].join('.') + commit_id = url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_email + + FileUtils.rm_rf repo_dir + + check_gitee_commit_exist(email_index) + end + + def check_gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = 'Your commit url is not a valid commit url.' + send_error_email(error_message) + + raise error_message + end + + def check_out_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + check_pub_key(pub_key) + + return pub_key + end + + def check_pub_key(pub_key) + return unless pub_key.nil? + + error_message = 'No pub_key found' + send_error_email(error_message) + + raise error_message + end +end -- 2.23.0

4 7

[PATCH v2 compass-ci 02/10] container/mail-robot: apply-account.rb
by Luan Shengde 05 Nov '20

05 Nov '20

check to assign account invoke ParseApplyAccountEmail to parse commit url and pubkey generate uuid and invoke AssignJumperAccount to apply account store account info to es send uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/apply-account.rb | 65 +++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) create mode 100755 container/mail-robot/apply-account.rb diff --git a/container/mail-robot/apply-account.rb b/container/mail-robot/apply-account.rb new file mode 100755 index 0000000..9a8d706 --- /dev/null +++ b/container/mail-robot/apply-account.rb @@ -0,0 +1,65 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '../../lib/es_client' + +# assign uuid/account +class AssignAccount + def initialize(mail_content) + @mail_content = mail_content + @my_email = mail_content.from[0] + @my_name = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + @send_mail_host = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + end + + def check_to_send_account + my_commit_url, my_ssh_pubkey = check_out_commit_url_pub_key + my_uuid, acct_infos = check_to_apply_account(my_ssh_pubkey) + + my_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_commit_url' => my_commit_url, + 'my_login_name' => acct_infos['my_login_name'], + 'my_uuid' => my_uuid + } + store_account_info(my_info) + my_info['acct_infos'] = acct_infos + send_my_uuid(my_info) + end + + def check_out_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + my_commit_url, my_ssh_pubkey = parse_apply_account_email.parse_commit_url_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def check_to_apply_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + apply_account_info = { + 'my_email' => @my_email, + 'my_name' => @my_name, + 'my_uuid' => my_uuid, + 'my_ssh_pubkey' => my_ssh_pubkey + } + apply_account = AssignJumperAccount.new(apply_account_info) + acct_infos = apply_account.apply_jumper_account + + return my_uuid, acct_infos + end + + def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) + end + + def send_my_uuid(my_info) + send_uuid = SendApplyAccountEmail.new(my_info) + send_uuid.send_uuid_email + end +end -- 2.23.0

5 8

[PATCH v3 lkp-tests] add PKGBUILD for sysbench-threads-git
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] Adapting sysbench-threads needs old version to user Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..41ed9f46 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=1 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="http://github.com/dallasmarlow/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() +{ + cd "$srcdir/sysbench" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() +{ + cd "$srcdir/sysbench" + make DESTDIR=$pkgdir install +} -- 2.23.0

5 4

[PATCH v4 lkp-tests] add PKGBUILD for sysbench-threads-git
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] Adapting sysbench-threads needs old version to user Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..41ed9f46 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=1 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="http://github.com/dallasmarlow/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() +{ + cd "$srcdir/sysbench" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() +{ + cd "$srcdir/sysbench" + make DESTDIR=$pkgdir install +} -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 05 Nov '20

05 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..799d3c4d --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,36 @@ +#!/bin/sh +# - runtime +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is an old version, It is both a benchmark and a stress test for the Linux kernel scheduler. +## It's main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes) which communicate via either sockets or pipes and time how long +## it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' + +[ -n "$runtime" ] || runtime=600 +[ -n "$iterations" ] || iterations=10 + +run_hackbench() +{ + # args: groups number, mode [process/thread], times + local hackbench_path + if [ -f /lkp/benchmarks/hackbench-git/hackbench/hackbench ]; then + hackbench_path=/lkp/benchmarks/hackbench-git/hackbench/hackbench + elif [ -f /usr/local/bin/hackbench ]; then + hackbench_path=/usr/local/bin/hackbench + else + echo "The hackbench executable file cannot be found." + fi + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

3 3

[PATCH v2 compass-ci] sbin/es-repo-mapping.sh: create script that generates "repo" mapping
by Zhang Yuhang 05 Nov '20

05 Nov '20

[why] We need a script to create "repo" mapping in es. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-repo-mapping.sh | 47 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 sbin/es-repo-mapping.sh diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh new file mode 100755 index 0000000..3a3930e --- /dev/null +++ b/sbin/es-repo-mapping.sh @@ -0,0 +1,47 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Check if "repo" index already exists. +status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) +[ $status_code -eq 200 ] && echo '"repo" index already exists. Exit.' && exit + +# Create "repo" index. +echo 'Start to create "repo" index.' +curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' +{ + "mappings": { + "_doc": { + "properties": { + "git_repo": { + "type": "keyword" + }, + "pkgbuild_repo": { + "type": "keyword" + }, + "url": { + "type": "keyword" + }, + "fetch_time": { + "type": "keyword" + }, + "new_refs_time": { + "type": "keyword" + }, + "offset_fetch": { + "type": "long" + }, + "offset_new_refs": { + "type": "long" + }, + "priority": { + "type": "long" + }, + "queued": { + "type": "boolean" + } + } + } + } +}' +[ $? -ne 0 ] && echo 'Failed to create "repo" index.' -- 2.23.0

2 2

[PATCH v5 lkp-tests 2/2] tests/iozone: modify the way of parsing parameter
by Lu Kaiyi 05 Nov '20

05 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] modify the way of parsing parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 66 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..b6a1e206 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,52 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + # reset IFS to split $test by "," and then restore default + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write") args+=" -i 0" + ;; + "read") args+=" -i 1" + ;; + "rand_rw") args+=" -i 2" + ;; + "backwards") args+=" -i 3" + ;; + "record") args+=" -i 4" + ;; + "stride") args+=" -i 5" + ;; + "fwrite") args+=" -i 6" + ;; + "fread") args+=" -i 7" + ;; + "rand_mix") args+=" -i 8" + ;; + "pwrite") args+=" -i 9" + ;; + "pread") args+=" -i 10" + ;; + "pwritev") args+=" -i 11" + ;; + "preadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

3 3

[PATCH compass-ci 6/6] container/webhook: add scripts for webhook
by Li Yuanchao 05 Nov '20

05 Nov '20

webhook.js will listen to webhook message from 'gitee.com' or 'github.com' if catch 'push' message, call push_hook.rb to send message to git-mirror through message queue Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/root/push_hook.rb | 15 +++++++++++++++ container/webhook/root/webhook.js | 22 ++++++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/webhook/root/push_hook.rb create mode 100755 container/webhook/root/webhook.js diff --git a/container/webhook/root/push_hook.rb b/container/webhook/root/push_hook.rb new file mode 100755 index 0000000..e467ccd --- /dev/null +++ b/container/webhook/root/push_hook.rb @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'bunny' + +connection = Bunny.new('amqp://172.17.0.1:5672') +connection.start +channel = connection.create_channel + +queue = channel.queue('web_hook') +message = ARGV[0] +queue.publish(message) +connection.close diff --git a/container/webhook/root/webhook.js b/container/webhook/root/webhook.js new file mode 100755 index 0000000..a98576c --- /dev/null +++ b/container/webhook/root/webhook.js @@ -0,0 +1,22 @@ +const http = require('http') +const spawn = require('child_process').spawn +const createHandler = require('git-webhook-handler') +// if users want to use webhook, they need to config webhook on their code warehouse('gitee.com' or 'github.com'), +// and use url: http://183.134.196.212:11301/webhook, secret: webhook@git. +const handler = createHandler({ path: '/webhook', secret: 'webhook@git' }) + +handler.on('error', function(err){ + console.error('Error:', err.message) +}) + +handler.on('push', function(event){ + console.log(event.payload.repository.url) + spawn('ruby', ['push_hook.rb', event.payload.repository.url]) +}) + +http.createServer(function(req, res){ + handler(req, res, function(err){ + res.statusCode = 404 + res.end('no such location') + }) +}).listen(11301) -- 2.23.0

2 1

[PATCH compass-ci 3/6] container/webhook: build
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/build | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/webhook/build diff --git a/container/webhook/build b/container/webhook/build new file mode 100755 index 0000000..bb5c0a7 --- /dev/null +++ b/container/webhook/build @@ -0,0 +1,6 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t node:webhook . + -- 2.23.0

2 1

[PATCH compass-ci 2/6] container/webhook: Dockerfile
by Li Yuanchao 05 Nov '20

05 Nov '20

add container webhook Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/Dockerfile | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/webhook/Dockerfile diff --git a/container/webhook/Dockerfile b/container/webhook/Dockerfile new file mode 100644 index 0000000..a9f0356 --- /dev/null +++ b/container/webhook/Dockerfile @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM node:10.21.0-alpine + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +ADD root / + +RUN apk update && \ + apk upgrade && \ + apk add --no-cache git && \ + apk add npm ruby-dev + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc + +RUN npm install git-webhook-handler &&\ + gem install bunny + -- 2.23.0

3 2

[PATCH lkp-tests 2/2] nginx: add nginx PKGBUILD
by Wei Jihui 05 Nov '20

05 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/nginx/PKGBUILD | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 pkg/nginx/PKGBUILD diff --git a/pkg/nginx/PKGBUILD b/pkg/nginx/PKGBUILD new file mode 100644 index 00000000..b437c4b1 --- /dev/null +++ b/pkg/nginx/PKGBUILD @@ -0,0 +1,24 @@ +pkgname=nginx +pkgver=1.14 +pkgrel=2 +arch=('i386' 'x86_64' 'aarch64') +url="https://nginx.org" +license=('BSD') +depends=(pcre zlib openssl gcc) +source=("https://nginx.org/download/nginx-${pkgver}.#{pkgrel}.tar.gz") +sha256sums=('SKIP') + +build() +{ + benchmark_path="${pkgdir}/lkp/benchmarks/${pkgname}" + mkdir -p "${benchmark_path}" + cd "${srcdir}/nginx-${pkgver}.#{pkgrel}" + ./configure --with-http_ssl_module --prefix="${benchmark_path}" + make +} + +package() +{ + cd "$srcdir/$pkgname-$pkgver" + make install +} -- 2.23.0

3 2

[PATCH lkp-tests] fio-setup-basic: keep format when change create_task
by Xu Xijian 05 Nov '20

05 Nov '20

[why] When add time_based to create_task, add '\' first, keep this format with others in context. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- setup/fio-setup-basic | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..5b0d0fce 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -117,7 +117,7 @@ ioscheduler=$ioscheduler" fi if parse_bool -q "$time_based"; then - create_task=" + create_task="\ $create_task time_based " -- 2.23.0

1 0

[PATCH compass-ci 5/6] container/webhook: start-depends
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start-depends | 2 ++ 1 file changed, 2 insertions(+) create mode 100755 container/webhook/start-depends diff --git a/container/webhook/start-depends b/container/webhook/start-depends new file mode 100755 index 0000000..38fc696 --- /dev/null +++ b/container/webhook/start-depends @@ -0,0 +1,2 @@ +rabbitmq +es -- 2.23.0

1 0

[PATCH compass-ci] container/master-fluentd: fix buffer warn
by Wu Zhende 05 Nov '20

05 Nov '20

[warn] [warn]: suppressed same stacktrace failed to emit fluentd's log event tag="fluent.warn" event={"message"=>"failed to write data into buffer by buffer overflow"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" [warn]: failed to write data into buffer by buffer overflow [How] Set num_threads=10 Flush the cache with 10 thread pools， prevents logs from piling up Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index a57d4cd..62c0319 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -44,6 +44,7 @@ port "#{ENV['LOGGING_ES_PORT']}" suppress_type_name true flush_interval 1s + num_threads 10 index_name ${tag} ssl_verify false log_es_400_reason true -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] compare values by each metrics
by Lu Weitao 05 Nov '20

05 Nov '20

compare values by each metrics based on groups matrices, and format compare result as echart data_set background: To support compare with user-defined template feature, the work-flow of user-defined template feature: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: compare_values by each metrics ---> format/show results Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 107 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 078028a..ebe1cd2 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -399,8 +399,115 @@ def compare_group_matrices(group_matrices, options) result_str end +# input: groups_matrices +# { +# group_key_1 => { +# dimension_1 => matrix_1, (openeuler 20.03) +# dimension_2 => matrix_2, (openeuler 20.09) +# dimension_3 => matrix_3, (centos 7.6) +# }, +# group_key_2 => {...} +# } +# +# output: compare_metrics_values +# { +# group_key_1 => { +# metric_1 => { +# 'average' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'standard_deviation' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'change' => { +# 'dimension_2 vs dimension_1' => xxx +# 'dimension_3 vs dimension_1' => xxx +# 'dimension_3 vs dimension_2' => xxx +# } +# }, +# metric_2 => {...} +# } +# } +def compare_metrics_values(groups_matrices) + metrics_compare_values = {} + groups_matrices.each do |group_key, dimensions| + metrics_compare_values[group_key] = get_metric_values(dimensions) + end + metrics_compare_values +end + +def get_metric_values(dimensions) + metrics_values = {} + dimensions.each do |dim, matrix| + matrix.each do |metric, values| + assign_metric_values(metrics_values, dim, metric, values) + end + end + assign_metric_change(metrics_values) + metrics_values +end + +def assign_metric_values(metrics_values, dim, metric, values) + metrics_values[metric] ||= {} + metrics_values[metric]['average'] ||= {} + metrics_values[metric]['standard_deviation'] ||= {} + metric_value = get_values(values, true) + metrics_values[metric]['average'][dim] = metric_value[:average] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] +end + +def assign_metric_change(metrics_values) + metrics_values.each do |metric, values| + metrics_values[metric]['change'] = {} + next if values['average'].size < 2 + + dimension_list = values['average'].keys + change_titles = get_change_titles(dimension_list) + change_titles.each do |base_dimension, challenge_dimension| + change = get_compare_value(values['average'][base_dimension], values['average'][challenge_dimension], true) + values['change'] = { "#{challenge_dimension} vs #{base_dimension}" => change } + end + end +end + +# input: dimension_list +# eg: ['openeuler 20.03', 'debian sid', 'centos 7.6'] +# output: Array(base_dimension: String, challenge_dimension: String) +# [ +# ['openeuler 20.03', 'debian sid'], +# ['openeuler 20.03', 'centos 7.6'], +# ['debian sid', 'centos 7.6'] +# ] +def get_change_titles(dimension_list) + dims = [] + dimension_list_size = dimension_list.size + (1..dimension_list_size - 1).each do |i| + (i..dimension_list_size - 1).each do |j| + dims << [dimension_list[i - 1], dimension_list[j]] + end + end + dims +end + +def show_chart_compare_result(chart_result) + print JSON.pretty_generate(chart_result) +end + # Format Fields +def format_for_chart(metrics_compare_values, template_params) + chart_result = {} + chart_result['title'] = template_params['title'] || nil + chart_result['unit'] = template_params['unit'] || nil + chart_result['x_name'] = template_params['x_params'].join('|') || nil + chart_result['tables'] = metrics_compare_values + chart_result +end + def format_fails_runs(fails, runs) fails_width = (SUB_LONG_COLUMN_WIDTH * FAILS_PROPORTION).to_i runs_width = SUB_LONG_COLUMN_WIDTH - fails_width - 1 -- 2.23.0

2 1

[PATCH compass-ci] sbin/es-repo-mapping.sh: create script that generates "repo" mapping
by Zhang Yuhang 05 Nov '20

05 Nov '20

[why] We need a script to create "repo" mapping in es. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-repo-mapping.sh | 78 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 sbin/es-repo-mapping.sh diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh new file mode 100755 index 0000000..ce0ad74 --- /dev/null +++ b/sbin/es-repo-mapping.sh @@ -0,0 +1,78 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Determine whether curl is installed. If not, install curl. +if ! [ -x "$(command -v curl)" ] +then + echo "curl not exists, try to install." + if [ -x "$(command -v apk)" ] + then + apk add install -y curl + elif [ -x "$(command -v yum)" ] + then + yum install -y curl + elif [ -x "$(command -v apt-get)" ] + then + apt-get install -y curl + fi +else + echo "curl has install." +fi + +# Determine whether curl is installed successfully +if [ $? -ne 0 ] +then + echo "curl install failed, exit." + exit +fi + +# Determine whether repo index has created +status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) + +if [ $status_code -eq 200 ] +then + echo "repo index has create, exit." +else + echo "repo index not exists, begin create index." + curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' + { + "mappings": { + "_doc": { + "properties": { + "git_repo": { + "type": "keyword" + }, + "pkgbuild_repo": { + "type": "keyword" + }, + "url": { + "type": "keyword" + }, + "fetch_time": { + "type": "keyword" + }, + "new_refs_time": { + "type": "keyword" + }, + "offset_fetch": { + "type": "long" + }, + "offset_new_refs": { + "type": "long" + }, + "priority": { + "type": "long" + }, + "queued": { + "type": "boolean" + } + } + } + } + }' + if [ $? -ne 0 ] + then + echo "create repo index failed." + fi +fi -- 2.23.0

5 8

[PATCH v5 lkp-tests 1/2] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 05 Nov '20

05 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..026b97e2 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write,read,rand_rw block_size: - 64k -- 2.23.0

2 1

[PATCH compass-ci] container: delete invalid start depends fluentd
by Liu Yinsi 05 Nov '20

05 Nov '20

fluentd renamed master-fluentd, but scheduler and extract-stats have no dependency with master-fluentd now, so remove it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/extract-stats/start-depends | 1 - container/scheduler/start-depends | 1 - 2 files changed, 2 deletions(-) diff --git a/container/extract-stats/start-depends b/container/extract-stats/start-depends index 4a016e6..efbe5a2 100755 --- a/container/extract-stats/start-depends +++ b/container/extract-stats/start-depends @@ -1,2 +1 @@ taskqueue -fluentd diff --git a/container/scheduler/start-depends b/container/scheduler/start-depends index 3e28dc1..4a17b46 100755 --- a/container/scheduler/start-depends +++ b/container/scheduler/start-depends @@ -1,4 +1,3 @@ redis -fluentd es taskqueue -- 2.23.0

2 2

[PATCH lkp-tests 1/2] fio: fix error when test with write mode
by Wei Jihui 05 Nov '20

05 Nov '20

[Why] when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. [How] add allow_mounted_write when write mode Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..f90963aa 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -116,6 +116,12 @@ $create_task ioscheduler=$ioscheduler" fi +if [[ "$rw" =~ "write" ]]; then + create_task="\ +$create_task +allow_mounted_write=1" +fi + if parse_bool -q "$time_based"; then create_task=" $create_task -- 2.23.0

2 1

[PATCH compass-ci] jobfile_operate.cr: set lkp path as a constant var
by Xu Xijian 05 Nov '20

05 Nov '20

[why] "/c/lkp-tests" is twice used here, so set it as a constant var LKP_PATH. [how] ... Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/jobfile_operate.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index d3c26c5..68818ed 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -8,8 +8,9 @@ require "yaml" # require from '/c/lkp-tests/lib/' require "shellwords" -if ENV["LKP_SRC"] != "/c/lkp-tests" - raise "ENV LKP_SRC mismatch: #{ENV["LKP_SRC"]} '/c/lkp-tests'" +LKP_PATH = "/c/lkp-tests" +if ENV["LKP_SRC"] != LKP_PATH + raise "ENV LKP_SRC mismatch: #{ENV["LKP_SRC"]} #{LKP_PATH}" end module Jobfile::Operate -- 2.23.0

1 0

[PATCH v2 compass-ci] improve multi-qemu code to support queue parameter
by Xiao Shenwei 05 Nov '20

05 Nov '20

Usage: multi-qemu [-n] [-c] [-q] -n, --name HOSTNAME_PREFIX specify used hostname_prefix -c, --count count how much VM do you need -q, --queues queues requested queues, use "," to separate more than 2 values -h, --help show this message example: ./multi-qemu -n vm-2p8g.xsw -c 2 -q vm-2p8g~xsw,vm-2p8g.aarch64 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-qemu | 99 +++++++++++++++++++++++++++++++++----------- 1 file changed, 74 insertions(+), 25 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 23e0451..bddad08 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -4,46 +4,95 @@ # frozen_string_literal: true require 'fileutils' +require 'optparse' +require 'time' -PWD = Dir.pwd +opt = {} +options = OptionParser.new do |opts| + opts.banner = 'Usage: multi-qemu [-n] [-c] [-q]' + + opts.separator '' + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'specify used hostname_prefix') do |name| + opt['hostname_prefix'] = name + end + + opts.on('-c count', '--count count', 'how much VM do you need') do |num| + opt['nr_vm'] = num + end + + opts.on('-q queues', '--queues queues', 'requested queues, use "," to separate more than 2 values') do |queues| + opt['queues'] = queues + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.size.zero? + puts options + exit +end + +options.parse!(ARGV) # Run multiple QEMU in parallel -HOSTNAME = ARGV[0] || "vm-2p8g--#{ENV['USER']}" -NR_VM = ARGV[1] || 1 +PWD = Dir.pwd +HOSTNAME = opt['hostname_prefix'] || "vm-2p8g.#{ENV['USER']}" +NR_VM = opt['nr_vm'] || 1 +QUEUES = opt['queues'] || "vm-2p8g.#{RUBY_PLATFORM.split('-')[0]}" +LOG_DIR = '/srv/cci/serial/logs' -def run(seqno) - loop do - start_time = Time.new - hostname = "#{HOSTNAME}-#{seqno}" - log_file = "/srv/cci/serial/logs/#{hostname}" +def main(hostname) + start_time = Time.new.strftime('%Y-%m-%d %H:%M:%S') + log_file = "#{LOG_DIR}/#{hostname}" + record_runtime(log_file, start_time) + run_vm(hostname) + duration = ((Time.new - Time.parse(start_time)) / 60).round(2) + record_runtime(log_file, duration, is_start: false) +end +def record_runtime(log_file, message, is_start: true) + if is_start File.open(log_file, 'w') do |f| # fluentd refresh time is 1s # let fluentd to monitor this file first sleep(2) - f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting QEMU" + f.puts "\n#{message} starting QEMU" end + return + end + File.open(log_file, 'a') do |f| + f.puts "\nTotal QEMU duration: #{message} minutes" + end +end - pwd_hostname = File.join(PWD, hostname) - FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) - FileUtils.cd(pwd_hostname) - system( - { 'hostname' => hostname }, - ENV['CCI_SRC'] + '/providers/qemu.sh' - ) - - duration = ((Time.new - start_time) / 60).round(2) - File.open(log_file, 'a') do |f| - f.puts "\nTotal QEMU duration: #{duration} minutes" - end +def run_vm(hostname) + pwd_hostname = File.join(PWD, hostname) + FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) + FileUtils.cd(pwd_hostname) + system( + { 'hostname' => hostname, 'queues' => QUEUES }, + ENV['CCI_SRC'] + '/providers/qemu.sh' + ) +end - # sleep 5s is for fluentd to collect it's log - sleep(5) +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.backtrace + # if an exception happend, request the next time after 30 seconds + sleep 25 + ensure + sleep 5 + end end end def save_pid(arr) - FileUtils.rm('pid') if File.exist?('pid') f = File.new('pid', 'a') arr.each do |i| f.puts(i) @@ -55,7 +104,7 @@ def multiqemu pids = [] NR_VM.to_i.times do |i| pid = Process.fork do - run i + loop_main("#{HOSTNAME}-#{i}") end pids << pid end -- 2.23.0

3 8

[PATCH v2 lkp-tests] add PKGBUILD for sysbench-threads-git
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] Adapting sysbench-threads needs to old version to user Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..897799ff --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=0.4.8 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="http://github.com/dallasmarlow/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() +{ + cd "$srcdir/$pkgname" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() +{ + cd "$srcdir/$pkgname" + make DESTDIR=$pkgdir install +} -- 2.23.0

2 2

[PATCH v2 lkp-tests] add testcase for sysbench-threads-git
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] old version package needs to debug by new testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 tests/sysbench-threads-git diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git new file mode 100755 index 00000000..8b9e1c99 --- /dev/null +++ b/tests/sysbench-threads-git @@ -0,0 +1,17 @@ +#!/bin/sh +# - nr_threads +# - thread_yields +# - thread_locks + +. "$LKP_SRC/lib/reproduce-log.sh" + +: "${nr_threads:=2}" +: "${thread_yields:=100}" +: "${thread_locks:=2}" + +args="\ + --threads=$nr_threads\ + --thread-yields=$thread_yields\ + --thread-locks=$thread_locks +" +log_cmd sysbench --test=threads $args run -- 2.23.0

2 1

[PATCH v6 lkp-tests] tests/build-pkg: change location for *.cgz file
by Liu Shaofei 05 Nov '20

05 Nov '20

[why] the path of *.cgz file generated by build-pkg script cannot be obtained during test. So the path of cgz file need to be changed. [how] below is example for openeuler: - path: /srv/initrd/build-pkg/initramfs/openeuler/aarch64/20.03/nasm-git - filename: latest.cgz -> f71820d6055373c9b9381615e28a28f307bb2df4.cgz f71820d6055373c9b9381615e28a28f307bb2df4.cgz v1.8.1.cgz -> 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz v1.5.1.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz df0584e46db9e7b412626deab10b923504745b2c.cgz Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..7c494a15 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,15 +23,19 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount $LKP_SERVER:$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } @@ -75,12 +81,22 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 +} + +create_softlink() +{ + [ -n "$upstream_tag" ] || return 0 + [ -e "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" ] && return 0 + + ln -sf "$(basename $(realpath ${cgz_name}))" "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" || return 0 + echo "create soft link: $PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz -> $(basename $(realpath ${cgz_name}))" } check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

3 3

[PATCH lkp-tests 3/3] lib/job.rb: change the way get hostname
by Li Ping 05 Nov '20

05 Nov '20

[why] we can get hostname from the repo lab-z9 or get hostname from lkp-tests/hosts before: get hostname from lab-z9, but the hostname not exist in lkp-tests/hosts the job will puts the error message: hosts_file not exist ... but there is no need to puts such error message, if we have got the hostname from lab-z9 --- lib/job.rb | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 2c06a475..12391b39 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,20 +334,14 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + return lab_hosts_file if lab_hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.file?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + puts("hosts_file not exist: #{hosts_file_name}, maybe need check testbox field") end def include_files -- 2.23.0

4 5

[PATCH v3 lkp-tests 3/3] lib/job2sh.rb: fix bug: can not find the file:pkg/$program/PKGBUILD
by Li Ping 05 Nov '20

05 Nov '20

[why] when the $dir not exist, cat > $dir/$file << 'EOF' xxx EOF will echo error message: can not find the file $dir/$file Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job2sh.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..7e6a5d7c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -350,6 +350,7 @@ class Job2sh < Job out_line out_line "\texport define_files='#{define_files.keys.join ' '}'" define_files.each do |file, val| + out_line "\tmkdir -p $LKP_SRC/$(dirname #{file})" out_line "\tcat > $LKP_SRC/#{file} <<'EOF'" out_line val out_line "EOF" -- 2.23.0

2 3

[PATCH v5 compass-ci 1/3] container/assign-account: assign-account.rb
by Luan Shengde 05 Nov '20

05 Nov '20

disable assigning account for user if there is no: - my_email - my_name - my_uuid [why]: my_email, my_name, my_uuid is required when initialize the default config file [how]: check if the parsed data has keys: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/assign-account.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..834f666 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,17 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + check_to_assign_account(data) +end + +def check_to_assign_account(data) + lacked_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lacked_info.join(', ')}." + raise error_message unless lacked_info.empty? + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

2 2

[PATCH v4 lkp-tests 2/2] tests/iozone: modify the way of parsing parameter
by Lu Kaiyi 05 Nov '20

05 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] modify the way of parsing parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 66 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..e126948d 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,52 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + # reset IFS to split $test by "," and then restore default + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write_rewrite") args+=" -i 0" + ;; + "read_reread") args+=" -i 1" + ;; + "random_read_write") args+=" -i 2" + ;; + "read_backwards") args+=" -i 3" + ;; + "rewrite_record") args+=" -i 4" + ;; + "stride_read") args+=" -i 5" + ;; + "fwrite_refwrite") args+=" -i 6" + ;; + "fread_refread") args+=" -i 7" + ;; + "random_mix") args+=" -i 8" + ;; + "pwrite_repwrite") args+=" -i 9" + ;; + "pread_repread") args+=" -i 10" + ;; + "pwritev_repwritev") args+=" -i 11" + ;; + "preadv_repreadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

2 1

[PATCH v2 compass-ci 06/10] container/mail-robot: run.rb
by Luan Shengde 05 Nov '20

05 Nov '20

invoke: send-apply-account-email assign-jumper-account parse-apply-account-email apply-account mail-robot Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/run.rb | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100755 container/mail-robot/run.rb diff --git a/container/mail-robot/run.rb b/container/mail-robot/run.rb new file mode 100755 index 0000000..8ac1d6e --- /dev/null +++ b/container/mail-robot/run.rb @@ -0,0 +1,10 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative 'send-apply-account-email' +require_relative 'assign-jumper-account' +require_relative 'parse-apply-account-email' +require_relative 'apply-account' +require_relative '../../lib/mail-robot' -- 2.23.0

3 2

[PATCH lkp-tests 1/3] lib/job2sh.rb: fix bug: can not find the file:pkg/$program/PKGBUILD
by Li Ping 05 Nov '20

05 Nov '20

[why] when the $dir not exist, cat > $dir/$file << 'EOF' xxx EOF will echo error message: can not find the file $dir/$file Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job2sh.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..7e6a5d7c 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -350,6 +350,7 @@ class Job2sh < Job out_line out_line "\texport define_files='#{define_files.keys.join ' '}'" define_files.each do |file, val| + out_line "\tmkdir -p $LKP_SRC/$(dirname #{file})" out_line "\tcat > $LKP_SRC/#{file} <<'EOF'" out_line val out_line "EOF" -- 2.23.0

2 1

[PATCH v4 lkp-tests 1/2] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 05 Nov '20

05 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..f9ab305f 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write_rewrite,read_reread,random_read_write block_size: - 64k -- 2.23.0

1 0

[PATCH v1 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 05 Nov '20

05 Nov '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..bb66bd1d --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,33 @@ +#!/bin/bash +# - runtime +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is an old version, It is both a benchmark and a stress test for the Linux kernel scheduler. +## It's main job is to create a specified number of pairs of schedulable entities +## (either threads or traditional processes) which communicate via either sockets or +## pipes and time how long it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' + +[ -n "$runtime" ] || runtime=600 +[ -n "$iterations" ] || iterations=10 + + +run_hackbench() +{ + # args: groups number, mode [process/thread], times + local hackbench_path + [ -f /lkp/benchmarks/hackbench-git/hackbench/hackbench ] && hackbench_path=/lkp/benchmarks/hackbench-git/hackbench/hackbench + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench + + -- 2.23.0

2 2

[PATCH lkp-tests 2/3] lib/job.rb: fix bug: add_define_files not work
by Li Ping 05 Nov '20

05 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths actually don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..2c06a475 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -626,6 +626,7 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + program = $1 if program =~ /^(.*)-\d+?$/ # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

1 1

[PATCH lkp-tests] lib/job.rb: fix bug: add_define_files not work
by Li Ping 05 Nov '20

05 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths actually don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index fb826b8c..254562f7 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -620,6 +620,7 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + program = $1 if program =~ /^(.*)-\d+$/ # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

1 0

[PATCH v2 compass-ci 01/10] lib/mail-robot.rb
by Luan Shengde 05 Nov '20

05 Nov '20

mail robot for listening the mail dir monitor mailbox check new added email file in new email directory handle new email file check to apply account only if email's subject match 'apply account' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..ff1dbd8 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +MAILDIR = '/srv/cci/Maildir/.compass-ci' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + begin + handle_new_email(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + puts e.backtrace + end + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + check_to_apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def check_to_apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.check_to_send_account +end + +monitor_dir("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

2 1

[PATCH lkp-tests] lib/job.rb: fix bug: add_define_files not work
by Li Ping 05 Nov '20

05 Nov '20

[why] multi-task job.yaml like this: program-0: xxx: program-1: xxx: the add_define_files will not work, because the following paths actually don't exist: $LKP_SRC/*/$program-$d $LKP_SRC/*/*/$program-$d $LKP_SRC/*/$program-$d/* $LKP_SRC/*/*/$program-$d/* Signed-off-by: Li Ping <15396232681(a)163.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index fb826b8c..12391b39 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -620,6 +620,7 @@ class Job programs_params_hash = @job["pp"] || {} program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| + program = $1 if program =~ /^(.*)-\d+?$/ # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files -- 2.23.0

1 0