November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 05/10] container/mail-robot: send-apply-account-email.rb
by Luan Shengde 05 Nov '20

05 Nov '20

SendApplyAccountEmail send uuid email build email message send email send error email build error email message send error email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/send-apply-account-email.rb | 72 +++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100755 container/mail-robot/send-apply-account-email.rb diff --git a/container/mail-robot/send-apply-account-email.rb b/container/mail-robot/send-apply-account-email.rb new file mode 100755 index 0000000..f29ff1e --- /dev/null +++ b/container/mail-robot/send-apply-account-email.rb @@ -0,0 +1,72 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# build mail data and send mail +class SendApplyAccountEmail + def initialize(send_mail_info) + @send_mail_host = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + @send_mail_info = send_mail_info + end + + def send_uuid_email + email_msg = <<~EMAIL_MESSAGE + To: #{@send_mail_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{@send_mail_info['my_name']}, + + Thank you for joining us. + You can use the following info to submit jobs: + + 1) setup default config + run the following command to add the below setup to default config file + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/\\${USER}.yaml <<-EOF + my_email: #{@send_mail_info['my_email']} + my_name: #{@send_mail_info['my_name']} + my_uuid: #{@send_mail_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + find how to write job yaml + submit jobs/netperf.yaml + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_error_email + email_msg = <<~EMAIL_MESSAGE + To: #{@send_mail_info['my_email']} + Subject: apply account failed + + Dear #{@send_mail_info['my_name']} + + Your application for account failed with following errors + + #{@send_mail_info['error_message']} + + regards + compass-ci + EMAIL_MESSAGE + + send_mail(email_msg) + end + + def send_mail(email_msg) + %x(curl -XPOST "#{@send_mail_host}:#{ENV['SEND_MAIL_PORT']}/send_mail_text" -d "#{email_msg}") + end +end -- 2.23.0

3 5

[PATCH v4 lkp-tests] job2sh.rb: define on_fail
by Hu Xuejiao 05 Nov '20

05 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh export_top_envexport_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7e6a5d7c..370c7899 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -358,6 +360,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

1 0

[PATCH compass-ci] qcow2rootfs: modify the location of the vmlinuz soft link of rootfs
by Yu Chuan 05 Nov '20

05 Nov '20

[Why] Due to the changes in the layout of rootfs, so modify the location of the vmlinuz soft link when generating rootfs. [How] Before: ${ROOTFS_DIR} └── vmlinuz -> boot/vmlinuz-xxx After: ${ROOTFS_DIR}/boot/ └── vmlinuz -> vmlinuz-xxx In addition: ROOTFS_VMLINUZ=$(basename "$ROOTFS_VMLINUZ_FILE"). Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/qcow2rootfs/bin/common | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/qcow2rootfs/bin/common b/container/qcow2rootfs/bin/common index 0d109e39c3fc..da39f523e2c7 100755 --- a/container/qcow2rootfs/bin/common +++ b/container/qcow2rootfs/bin/common @@ -174,8 +174,9 @@ create_links_vmlinuz_initrd() { bash -c " cd $EXTRACT_ROOT/rootfs-dir chmod a+r $ROOTFS_INITRD_LKP - ln -sf $ROOTFS_VMLINUZ_FILE vmlinuz ln -sf $ROOTFS_INITRD_LKP initrd.lkp + cd $EXTRACT_ROOT/rootfs-dir/boot + ln -sf $ROOTFS_VMLINUZ vmlinuz " ) if "${cmds[@]}" ; then -- 2.23.0

1 0

[PATCH lkp-tests] fio: fix error when test with write mode
by Wei Jihui 05 Nov '20

05 Nov '20

[Why] fio need test on disk, so mount a disk by: "mount -t xfs -o inode64 /dev/sda /fs/sda" when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 1 + 1 file changed, 1 insertion(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..3ea79fa2 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -95,6 +95,7 @@ fallocate=$fallocate io_size=$io_size file_service_type=$file_service_type random_distribution=$random_distribution +allow_mounted_write=1 group_reporting create_only=1 pre_read=$pre_read -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: change the way get hosts file
by Li Ping 05 Nov '20

05 Nov '20

[why] we can get hosts file from the repo lab-z9 or get hosts file from lkp-tests/hosts before: get hosts file from lab-z9, but the hosts file not exist in lkp-tests/hosts the job will puts the error message: hosts_file not exist ... but there is no need to puts such error message, if we have got the hosts file from lab-z9 --- lib/job.rb | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 995b38bf..254562f7 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,20 +334,14 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + return lab_hosts_file if lab_hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.file?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + puts("hosts_file not exist: #{hosts_file_name}, maybe need check testbox field") end def include_files -- 2.23.0

3 3

[PATCH v2 compass-ci 2/2] add template compare mode
by Lu Weitao 05 Nov '20

05 Nov '20

add template compare mode in sbin/compare(command-line tool) by: - load and parse compare.yaml - calls lib/es_query.rb, query jobs data set from ES - calls lib/matrix.rb, convert jobs data set to group matrices - calls lib/compare_matrixes.rb, use groups matrices to compare background: To support compare with user-defined template feature, the work-flow of user-defined template feature: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results example: compare -t/--template compare_template.yaml, the yaml like: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s output:{ "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "bs|test_size", "tables": { "4k|1G": { "fio.write_iops": { "average": { "openeuler 20.03": 308298.5524761111, "debian sid": 220289.07563 }, "standard_deviation": { "openeuler 20.03": null, "debian sid": null }, "change": { "debian sid vs openeuler 20.03": -28.5 } } }, "16k|1G": {...}, ... } } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++ sbin/compare | 10 +++++++++- 2 files changed, 60 insertions(+), 1 deletion(-) diff --git a/lib/compare.rb b/lib/compare.rb index 2837edc..9edf049 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -6,6 +6,7 @@ require_relative './es_query.rb' require_relative './matrix2.rb' require_relative './compare_matrixes.rb' require_relative './constants.rb' +require 'yaml' # ------------------------------------------------------------------------------------------- # compare_matrices_list @@ -66,3 +67,53 @@ def create_groups_matrices_list(conditions, dims) query_results = es.multi_field_query(conditions) combine_group_query_data(query_results, dims) end + +# ------------------------------------------------------------------------------------------- +# compare with user-defined compare_template.yaml +# the work-flow are: +# load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list +# ---> create groups_matrices ---> compare_values by each metrics ---> format/show results +# + +def compare_by_template(template) + template_params = load_template(template) + groups_matrices = create_groups_matrices(template_params) + compare_values = compare_metrics_values(groups_matrices) + chart_results = format_for_chart(compare_values, template_params) + show_chart_compare_result(chart_results) +end + +def load_template(template) + unless File.file?(template) + warn 'template does not exist' + exit + end + YAML.load_file(template) +end + +# input: template_params: Hash +# eg: +# { +# "compare_metrics"=>["fio.write_iops", "fio.read_iops"], +# "filter"=>[ +# {"suite"=>["fio-bisic"]}, +# {"os_arch"=>["aarch_64"]} +# ], +# "compare_dimensions"=>[ +# {"os"=>"openeuler", "os_version"=>20.03}, +# {"os"=>"centos", "os_version"=>7.6} +# ], +# "x_params"=>["block_size", "package_size"], +# "title"=>"Hackbench Performance Testing", +# "unit"=>"KB/s" +# } +def create_groups_matrices(template_params) + es = ESQuery.new + query_results = es.multi_field_query(template_params['filter']) + combine_group_jobs_list( + query_results, + template_params['x_params'], + template_params['compare_dimensions'], + template_params['compare_metrics'] + ) +end diff --git a/sbin/compare b/sbin/compare index cf3980c..b13682e 100755 --- a/sbin/compare +++ b/sbin/compare @@ -6,6 +6,7 @@ # Usage: # compare "conditions_1" "conditions_2" ... -c "common_conditions" # compare "conditions" -d "dimensions" +# compare -t template.yaml # Eg: # compare "id=6000,6001" "id=7000,7001" # compare "commit=a12d232e" "commit=b3bacc31" @@ -21,6 +22,7 @@ is_group = false dimensions = nil colorful = nil options = {} +template = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: compare "conditions" ... [option]' @@ -45,6 +47,10 @@ opt_parser = OptionParser.new do |opts| colorful = color end + opts.on('-t', '--template template', 'compare with user-defined template') do |t| + template = t + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -60,7 +66,9 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful -if is_group +if template + compare_by_template(template) +elsif is_group compare_group(argv, dimensions, options) else compare_matrices_list(argv, common_conditions, options) -- 2.23.0

3 4

[PATCH lkp-tests] fio: fix error when test with write mode
by Wei Jihui 05 Nov '20

05 Nov '20

[Why] fio need test on disk, so mount a disk by: "mount -t xfs -o inode64 /dev/sda /fs/sda" when test fio with write mode, an error is reported: fio: /dev/sda appears mounted, and 'allow_mounted_write' isn't set. [How] set allow_mounted_write=1 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- setup/fio-setup-basic | 1 + 1 file changed, 1 insertion(+) diff --git a/setup/fio-setup-basic b/setup/fio-setup-basic index 58ebeb80..3ea79fa2 100755 --- a/setup/fio-setup-basic +++ b/setup/fio-setup-basic @@ -95,6 +95,7 @@ fallocate=$fallocate io_size=$io_size file_service_type=$file_service_type random_distribution=$random_distribution +allow_mounted_write=1 group_reporting create_only=1 pre_read=$pre_read -- 2.23.0

2 1

[PATCH compass-ci 4/6] container/webhook: start
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 container/webhook/start diff --git a/container/webhook/start b/container/webhook/start new file mode 100755 index 0000000..50cbf20 --- /dev/null +++ b/container/webhook/start @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "webhook" + +cmd = %W[ + docker run + --restart=always + --name webhook + -d + -p 11301:11301 + node:webhook +] + +cmd += ['sh', '-c', 'node webhook.js'] +system(*cmd) -- 2.23.0

2 2

[PATCH v2 lkp-tests] libmicro: add PKGBUILD for libmicro
by Bai Jing 05 Nov '20

05 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- pkg/libmicro/PKGBUILD | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 pkg/libmicro/PKGBUILD diff --git a/pkg/libmicro/PKGBUILD b/pkg/libmicro/PKGBUILD new file mode 100644 index 00000000..61cd32f4 --- /dev/null +++ b/pkg/libmicro/PKGBUILD @@ -0,0 +1,19 @@ +pkgname=libMicro +pkgver=0.4 +pkgrel=0 +pkgdesc="A portable set of microbenchmarks focused on the syscall and system library interfaces." +arch=('i386' 'x86_64') +url="https://github.com/rzezeski/libMicro" +license=('CDDL') +source=("https://codeload.github.com/redhat-performance/libMicro/zip/0.4.0-rh") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname-$pkgver.$pkgrel-rh" + make +} + +package() { + mkdir -p "$pkgdir/lkp/benchmarks/libMicro" + cp -r "$srcdir/$pkgname-$pkgver.$pkgrel-rh/"* "$pkgdir/lkp/benchmarks/libMicro" +} -- 2.23.0

4 7

[PATCH lkp-tests] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 05 Nov '20

05 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } PKGBUILD-1.1 contains only the changed part. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..dc8471b9 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -888,7 +888,7 @@ update_pkgver() { exit 1 fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" - source_safe "$BUILDFILE" + source_safe "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -1604,10 +1604,14 @@ cd_safe() { source_safe() { shopt -u extglob - if ! source "$@"; then - error "$(gettext "Failed to source %s")" "$1" - exit 1 - fi + local file + for file in "$@" + do + if ! source "$file"; then + error "$(gettext "Failed to source %s")" "$1" + exit 1 + fi + done shopt -s extglob } @@ -1626,7 +1630,6 @@ merge_arch_attrs() { source_buildfile() { source_safe "$@" - if (( !SOURCEONLY )); then merge_arch_attrs fi @@ -3661,6 +3664,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +PKGBUILD_TAG=$PKGBUILD_TAG if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then @@ -3691,7 +3695,7 @@ else if [[ ${BUILDFILE:0:1} != "/" ]]; then BUILDFILE="$startdir/$BUILDFILE" fi - source_buildfile "$BUILDFILE" + source_buildfile "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

4 4