- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/compare_data_format: optimize the format for echart
by Lu Weitao 18 Nov '20

18 Nov '20

[Why] With template compare mode, the result(data_set) isn't friendly for echart [How] Make the result more friendly for echart like: { "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "test_size|bs", "tables": { "fio.read_iops": { "average": { "dimensions": [ "compare_dimension", "openeuler 20.03" ], "source": [ ["1G|4k", "1G|16k", "1G|32k", "1G|64k", "1G|128k", "1G|256k", "1G|512k", "1G|1024k", "80G|4k"], ["openeuler 20.03", 144076.2903315, 37865.30472368628, 21145.10375497826, 14010.34254665909, 6701.240849466667, 3205.077255, 1367.476930860465, 673.3270888666667, 11601.099817 ] ] }, "standard_deviation": {...}, "change": {...} }, "fio.write_iops": {...} ... } } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_data_format.rb | 77 ++++++++++++++++++++++++++++++++++---- lib/compare_matrixes.rb | 3 +- 2 files changed, 71 insertions(+), 9 deletions(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index 3d82550..b4f26ee 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -5,14 +5,75 @@ # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # +class FormatEchartData + def initialize(metrics_compare_results, template_params) + @metrics_compare_results = metrics_compare_results + @template_params = template_params + @data_set = {} + end -def format_for_echart(metrics_compare_results, template_params) - echart_result = {} - echart_result['title'] = template_params['title'] - echart_result['unit'] = template_params['unit'] - x_params = template_params['x_params'] - echart_result['x_name'] = x_params.join('|') if x_params - echart_result['tables'] = metrics_compare_results + def format_for_echart + echart_result = {} + echart_result['title'] = @template_params['title'] + echart_result['unit'] = @template_params['unit'] + @x_name = @template_params['x_params'] + echart_result['x_name'] = @x_name.join('|') if @x_name + echart_result['tables'] = convert_to_echart_dataset - echart_result + echart_result + end + + def convert_to_echart_dataset + @x_params = sort_x_params((a)metrics_compare_results.keys) + @metrics_compare_results.each_value do |metrics_values| + metrics_values.each do |metric, metric_value| + assign_echart_data_set(metric, metric_value) + end + end + + @data_set + end + + def assign_echart_data_set(metric, metric_value) + @data_set[metric] = {} + metric_value.each do |value_type, values| # value_type can be: average, standard_deviation, change + @data_set[metric][value_type] = { + 'dimensions' => ['compare_dimension'] + } + + dimension_list = values.keys.sort + @data_set[metric][value_type]['dimensions'] += dimension_list + @data_set[metric][value_type]['source'] = assign_echart_source(metric, value_type, dimension_list) + end + end + + def assign_echart_source(metric, value_type, dimensions) + source = [] + source << @x_params + dimensions.each do |dimension| + dimension_values = [dimension] + @x_params.each do |x_param| + if @metrics_compare_results[x_param][metric] + dimension_values << @metrics_compare_results[x_param][metric][value_type][dimension] + end + end + source << dimension_values + end + + source + end +end + +# input: x_params_list +# eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] +# output: +# ["1G|4K", "1G|128k", "1G|1024k", "2G|4k"] +def sort_x_params(x_params_list) + x_params_hash = {} + x_params_list.each do |param| + params = param.gsub(/[a-zA-Z]+$/, '0').split('|').map(&:to_i) + x_params_hash[params] = param + end + + x_params_hash.sort.map { |h| h[1] } end diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..762c087 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -498,7 +498,8 @@ def get_dimentions_combination(dimension_list) end def show_compare_result(metrics_compare_results, template_params) - echart_results = format_for_echart(metrics_compare_results, template_params) + formatter = FormatEchartData.new(metrics_compare_results, template_params) + echart_results = formatter.format_for_echart print JSON.pretty_generate(echart_results) end -- 2.23.0

2 2

[PATCH compass-ci] dracut-initrd: delete unused kernel module iscsi from dracut
by Xu Xijian 18 Nov '20

18 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/run b/container/dracut-initrd/run index b0c9b30..6301ee2 100755 --- a/container/dracut-initrd/run +++ b/container/dracut-initrd/run @@ -43,7 +43,7 @@ cmd=( -v $root/lib/modules:/lib/modules debian:dracut bash -c - "dracut --force --kver $kver -k $kernel_modules $initrd_output && + "dracut -o iscsi --omit-drivers iscsi --force --kver $kver -k $kernel_modules $initrd_output && chmod 644 $initrd_output" # example: -- 2.23.0

1 0

[PATCH v6 lkp-tests] tests/cci-makepkg: update gcc for centos7 when use initramfs os
by Lin Jiaxin 18 Nov '20

18 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 530c9fef..905ad689 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -88,13 +88,22 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +centos_update_gcc() +{ + if grep -sqF "CentOS Linux release 7" /etc/centos-release; then + [ "$os_mount" = "initramfs" ] && { + yum install -y centos-release-scl-rh + yum install -y "devtoolset-8-gcc*" + } + source /opt/rh/devtoolset-8/enable + fi +} + [ -n "$PKGBUILD_TAG" ] && download_pkgfile distro_install_depends lkp-dev -[ $os = "centos" ] && -grep -sqF "CentOS Linux release 7" /etc/centos-release && -source /opt/rh/devtoolset-8/enable +[ "$os" = "centos" ] && centos_update_gcc distro_install_depends ${benchmark}-dev -- 2.23.0

2 1

[PATCH compass-ci] updata document for apply account
by Luan Shengde 18 Nov '20

18 Nov '20

internal user: CLI tool: answerback-email external user: send 'apply account' email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/apply-ssh-account.md | 49 ++++++++++++++++++++++++++++++---------- 1 file changed, 37 insertions(+), 12 deletions(-) diff --git a/doc/apply-ssh-account.md b/doc/apply-ssh-account.md index 69d8df6..d718d0d 100644 --- a/doc/apply-ssh-account.md +++ b/doc/apply-ssh-account.md @@ -1,18 +1,22 @@ -# apply for ssh account +# apply for jumper account ## purpose -The ssh account is used to login our jumper VM, where you can +The jumper account is used to login the jumper VM, where you can - submit a job - ssh into a testbox +- read output of the job ## steps overview -1. send email to apply for ssh account - If approved, you'll get a response email with: +### manually assign account +for internal user +1. run assign account tool + usage: + answerback-mail.rb [-e|--email email] [-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file] [-g|--gen-sshkey] - login command: - ssh -p port account@server_ip + login command: + ssh -p port account@server_ip 2. login authentication - use pub key @@ -20,18 +24,39 @@ The ssh account is used to login our jumper VM, where you can 3. submit jobs - prepare job.yaml - - run job + - submit job - command: submit job.yaml -## example apply email +### apply account with email +for external user +1. send apply account email + - send apply account email to: compass-ci(a)qq.com + attention: + email subject: + apply account + email body: + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + attachment: + ssh pubkey file + it will return email with my_email, my_name, my_uuid if successfully applied account + +2. prepare to submit job + execute on user's local host + - config default yaml file: + ~/.config/compass-ci/default/account.yaml + - prepare job.yaml + - submit job + +3. apply account email format + example: - To: team(a)crystal.ci - Subject: apply ssh account + To: compass-ci(a)qq.com + Subject: apply account # Show a commit URL that you contributed to an OSS project # We'll validate whether the URL contains your email. # for example, - commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… - # attach your ssh pub key as file (optionally but highly recommended) + # attach your ssh pub key to the email as attachment -- 2.23.0

1 0

[PATCH v5 lkp-tests] tests/cci-makepkg: update gcc for centos7 when use initramfs os
by Lin Jiaxin 18 Nov '20

18 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 530c9fef..865b2a9d 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -88,13 +88,22 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +centos_update_gcc() +{ + if $(grep -sqF "CentOS Linux release 7" /etc/centos-release ); then + [ "$os_mount" = "initramfs" ] && { + yum install -y centos-release-scl-rh + yum install -y "devtoolset-8-gcc*" + } + source /opt/rh/devtoolset-8/enable + fi +} + [ -n "$PKGBUILD_TAG" ] && download_pkgfile distro_install_depends lkp-dev -[ $os = "centos" ] && -grep -sqF "CentOS Linux release 7" /etc/centos-release && -source /opt/rh/devtoolset-8/enable +[ "$os" = "centos" ] && centos_update_gcc distro_install_depends ${benchmark}-dev -- 2.23.0

2 1

[PATCH compass-ci] container: add other group read permission
by Bai Jing 18 Nov '20

18 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- container/conserver/conserver | 1 + 1 file changed, 1 insertion(+) diff --git a/container/conserver/conserver b/container/conserver/conserver index 94b1909..6886127 100755 --- a/container/conserver/conserver +++ b/container/conserver/conserver @@ -2,4 +2,5 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +umask 0002 conserver -o -O1 -C /etc/conserver.cf -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/2] tests/cci-makepkg: update gcc for centos7 when use initramfs os
by Lin Jiaxin 18 Nov '20

18 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a62b5be0..b090559f 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -88,13 +88,22 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +centos_update_gcc() +{ + if $(grep -sqF "CentOS Linux release 7" /etc/centos-release ); then + [[ "$os_mount" = "initramfs" ]] && { + yum install -y centos-release-scl-rh + yum install -y "devtoolset-8-gcc*" + } + source /opt/rh/devtoolset-8/enable + fi +} + [ -n "$PKGBUILD_TAG" ] && get_pkgfile distro_install_depends lkp-dev -[ $os = "centos" ] && -grep -sqF "CentOS Linux release 7" /etc/centos-release && -source /opt/rh/devtoolset-8/enable +[ $os = "centos" ] && centos_update_gcc distro_install_depends ${benchmark}-dev -- 2.23.0

2 1

[PATCH compass-ci] remove actual ip in compass-ci code
by Xiao Shenwei 18 Nov '20

18 Nov '20

obtain the actual ip address from configuration file Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/assign-account/get_account_info.rb | 2 +- src/features/jobs/right_iperf.yaml | 2 +- src/scheduler/README.md | 2 +- user-client/helper/iperf.yaml | 2 +- user-client/jobs/iperf-pxe.yaml | 2 +- user-client/jobs/iperf-vm.yaml | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 137f5c6..ca86e88 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -39,7 +39,7 @@ the returned data for setup_jumper_account_info like: { "account" => "guest", "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", + "jumper_ip" => "0.0.0.0", "jumper_port" => "10000" } diff --git a/src/features/jobs/right_iperf.yaml b/src/features/jobs/right_iperf.yaml index f81af38..e239ccd 100644 --- a/src/features/jobs/right_iperf.yaml +++ b/src/features/jobs/right_iperf.yaml @@ -30,7 +30,7 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 +LKP_SERVER: LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x diff --git a/src/scheduler/README.md b/src/scheduler/README.md index fdc7027..8e3a197 100644 --- a/src/scheduler/README.md +++ b/src/scheduler/README.md @@ -6,7 +6,7 @@ - And set the environment variables needed for running (modify "/etc/compass-ci/defaults/crystal.yaml", and run $CCI_SRC/container/defconfig.rb | defconfig.sh) LKP_SRC: ~/code/lkp-tests # path to your lkp-tests repo CCI_SRC: ~/code/compass-ci # path to your compass-ci repo - SCHED_HOST: 172.168.131.131 # hostname or ip of the scheduler + SCHED_HOST: 172.17.0.1 # hostname or ip of the scheduler SCHED_PORT: 3000 # port of the scheduler JOB_INDEX_TYPE: "jobs/_doc" # es job document index and type - '$'{variable} is used at shell command, means the value of variable diff --git a/user-client/helper/iperf.yaml b/user-client/helper/iperf.yaml index 1509470..30e929f 100644 --- a/user-client/helper/iperf.yaml +++ b/user-client/helper/iperf.yaml @@ -30,7 +30,7 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 +LKP_SERVER: LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x diff --git a/user-client/jobs/iperf-pxe.yaml b/user-client/jobs/iperf-pxe.yaml index 209c5d6..cb5a513 100644 --- a/user-client/jobs/iperf-pxe.yaml +++ b/user-client/jobs/iperf-pxe.yaml @@ -53,7 +53,7 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 +LKP_SERVER: LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x diff --git a/user-client/jobs/iperf-vm.yaml b/user-client/jobs/iperf-vm.yaml index 434b964..c976bed 100644 --- a/user-client/jobs/iperf-vm.yaml +++ b/user-client/jobs/iperf-vm.yaml @@ -34,7 +34,7 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 +LKP_SERVER: LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x -- 2.23.0

2 1

[PATCH v5 compass-ci] container: fix failed to build archlinux images in x86_64 machine
by Liu Yinsi 18 Nov '20

18 Nov '20

[why] when build archlinux images in x86_64 machine error: failed retrieving file 'community.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : The requested URL returned error:404 error: failed to update community (failed to retrieve some files) error: failed to synchronize all databases The command '/bin/sh -c pacman --needed --noprogressbar --noconfirm -Syu && pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk' returned a non-zero code: 1 because archlinux mirrorlist not support x86_64 machine. [how] 1. add 2 folder root/aarch64/etc/pacman.d/mirrorlist and root/x86_64/etc/pacman.d/mirrorlist; In addition, 'COPY' in Dockerfile, the directory itself is not copied, only its contents. Please see: https://docs.docker.com/engine/reference/builder/#copy 2. use $(arch) to choose mirrorlist according to different system architecture. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/archlinux/Dockerfile | 6 +++++- container/archlinux/build | 2 +- .../archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist | 0 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist | 1 + 4 files changed, 7 insertions(+), 2 deletions(-) rename container/archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist (100%) create mode 100644 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/Dockerfile b/container/archlinux/Dockerfile index c0f05d3..1f80ae0 100644 --- a/container/archlinux/Dockerfile +++ b/container/archlinux/Dockerfile @@ -5,7 +5,11 @@ FROM lopsided/archlinux MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> -COPY root / +ARG ARCH + +COPY root/$ARCH / + RUN chmod 755 /etc + RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk diff --git a/container/archlinux/build b/container/archlinux/build index 81feda2..9749489 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,4 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t archlinux:testbed . +docker build --build-arg ARCH=$(arch) -t archlinux:testbed . diff --git a/container/archlinux/root/etc/pacman.d/mirrorlist b/container/archlinux/root/aarch64/etc/pacman.d/mirrorlist similarity index 100% rename from container/archlinux/root/etc/pacman.d/mirrorlist rename to container/archlinux/root/aarch64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist new file mode 100644 index 0000000..556fac8 --- /dev/null +++ b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist @@ -0,0 +1 @@ +Server = http://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch -- 2.23.0

2 1

[PATCH v2 lkp-tests] lib/job.rb: change the way to analyze the testbox field
by Li Ping 18 Nov '20

18 Nov '20

If user wants to reproduce the result directly in the result directory. [problem] /srv/result/iperf/vm-2p8g/2020-11-09/z9.147520% submit job.yaml [Error] Traceback (most recent call last): 9: from /home/liping/lkp-tests/sbin/submit:96:in `<main>' 8: from /home/liping/lkp-tests/sbin/submit:96:in `each' 7: from /home/liping/lkp-tests/sbin/submit:106:in `block in <main>' 6: from /home/liping/lkp-tests/lib/job.rb:699:in `each_jobs' 5: from /home/liping/lkp-tests/lib/job.rb:400:in `load_defaults' 4: from /home/liping/lkp-tests/lib/job.rb:400:in `each' 3: from /home/liping/lkp-tests/lib/job.rb:401:in `block in load_defaults' 2: from /home/liping/lkp-tests/lib/job.rb:381:in `load_one_defaults' 1: from /home/liping/lkp-tests/lib/job.rb:306:in `load_hosts_config' /home/liping/lkp-tests/lib/job.rb:344:in `get_hosts_file': hosts file not exist: vm-2p8g.taishan200-2280-2s64p-256g, maybe need check testbox field (ArgumentError) --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..2e94316c 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -337,7 +337,7 @@ class Job lab_hosts_file = get_lab_hosts_file return lab_hosts_file if lab_hosts_file - hosts_file_name = @job['tbox_group'].split('--')[0] + hosts_file_name = @job['tbox_group'].split(/\.|--/)[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" return hosts_file if File.file?(hosts_file) -- 2.23.0

2 2

Re: [PATCH v6 lkp-tests] add: sysbench-threads-git result parsing file the colon is not required. add sysbench-threads-git result parse file
by Li Ping 18 Nov '20

18 Nov '20

Reply-To: In-Reply-To: <20201118065209.1071046-1-1034502035(a)qq.com> On Wed, Nov 18, 2020 at 02:52:09PM +0800, Hu Xuejiao wrote: >[input] >General statistics: >total time: 0.3987s >total number of events: 10000 >total time taken by event execution: 0.3971 >response time: >min: 0.04ms > It has written in your patch. >[output] >total time: 0.3987s > >Signed-off-by: Hu XueJiao <1034502035(a)qq.com> >--- > stats/sysbench-threads-git | 14 ++++++++++++++ > 1 file changed, 14 insertions(+) > create mode 100755 stats/sysbench-threads-git > >diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git >new file mode 100755 >index 00000000..afe5d664 >--- /dev/null >+++ b/stats/sysbench-threads-git >@@ -0,0 +1,14 @@ >+#!/usr/bin/env ruby >+ >+# General statistics: >+# total time: 0.2671s >+# total number of events: 10000 >+# total time taken by event execution: 0.3971s >+# response time: >+# min: 0.04ms >+ >+$stdin.each_line do |line| >+ next unless line =~ /^\s+total time:\s+([0-9.]+)/ >+ >+ puts "total time: #{$1.to_f}s" >+end >-- >2.23.0 >

2 2

[PATCH v5 lkp-tests] pkg: add mysql
by Lin Jiaxin 18 Nov '20

18 Nov '20

sysbench-mysql need to test with mysql8.0.17 Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...00-mysql-add-fstack-protector-strong.patch | 20 ++++++ pkg/mysql/PKGBUILD | 60 +++++++++++++++++ pkg/mysql/my.cnf | 67 +++++++++++++++++++ 3 files changed, 147 insertions(+) create mode 100644 pkg/mysql/0000-mysql-add-fstack-protector-strong.patch create mode 100644 pkg/mysql/PKGBUILD create mode 100644 pkg/mysql/my.cnf diff --git a/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch new file mode 100644 index 00000000..23c07c87 --- /dev/null +++ b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch @@ -0,0 +1,20 @@ +diff --git a/cmake/build_configurations/compiler_options.cmake b/cmake/build_configurations/compiler_options.cmake +index fecf2a2..f3e012f 100644 +--- a/cmake/build_configurations/compiler_options.cmake ++++ b/cmake/build_configurations/compiler_options.cmake +@@ -43,6 +43,7 @@ IF(UNIX) + # Default GCC flags + IF(CMAKE_COMPILER_IS_GNUCC) + SET(COMMON_C_FLAGS "-fno-omit-frame-pointer") ++ SET(COMMON_C_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_C_FLAGS "-fno-inline ") +@@ -57,6 +58,7 @@ IF(UNIX) + ENDIF() + IF(CMAKE_COMPILER_IS_GNUCXX) + SET(COMMON_CXX_FLAGS "-std=c++14 -fno-omit-frame-pointer") ++ SET(COMMON_CXX_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_CXX_FLAGS "-fno-inline ") diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD new file mode 100644 index 00000000..8869e62b --- /dev/null +++ b/pkg/mysql/PKGBUILD @@ -0,0 +1,60 @@ +pkgname=mysql +pkgver=8.0.17 +pkgrel=3 +arch=('aarch64' 'x86_64') +url="http://www.mysql.com" +license=('GPLv2') +source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" + "0000-mysql-add-fstack-protector-strong.patch" + "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" + "my.cnf") +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' '7bb0372ec4e9281813a9afc67c6cfc20') + +prepare() +{ + cd $srcdir/$pkgname-$pkgver + patch -p1 < ../0000-mysql-add-fstack-protector-strong.patch +} + +build() +{ + build_getcpu + build_mysql +} + +build_mysql() +{ + cd $srcdir/$pkgname-$pkgver + + CMAKE=cmake + grep -sqF "CentOS Linux release 7" /etc/centos-release && CMAKE=cmake3 + + $CMAKE . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc \ + -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_PARTITION_STORAGE_ENGINE=1 -DWITH_FEDERATED_STORAGE_ENGINE=1 \ + -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_MYISAM_STORAGE_ENGINE=1 \ + -DENABLED_LOCAL_INFILE=1 -DENABLE_DTRACE=0 -DDEFAULT_CHARSET=utf8mb4 -DDEFAULT_COLLATION=utf8mb4_general_ci \ + -DWITH_EMBEDDED_SERVER=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=./boost -DFORCE_INSOURCE_BUILD=1 + make -j8 +} + +build_getcpu() +{ + cd $srcdir + tar -zxvf getcpu.tgz + + cd getcpu || exit + make +} + + +package() +{ + mkdir -p $pkgdir/getcpu + cp getcpu/getcpu-hi.ko $pkgdir/getcpu/ + + mkdir -p $pkgdir/etc + cp my.cnf $pkgdir/etc/ + + cd $srcdir/$pkgname-$pkgver + make DESTDIR=$pkgdir install +} diff --git a/pkg/mysql/my.cnf b/pkg/mysql/my.cnf new file mode 100644 index 00000000..9732af40 --- /dev/null +++ b/pkg/mysql/my.cnf @@ -0,0 +1,67 @@ +[mysqld_safe] +log-error=/data/mysql/log/mysql.log +pid-file=/data/mysql/run/mysqld.pid +[mysqldump] +quick +[mysql] +[client] +default-character-set=utf8mb4 +socket=/data/mysql/run/mysql.sock + +[mysqld] +socket=/data/mysql/run/mysql.sock +tmpdir=/data/mysql/tmp +datadir=/data/mysql/data +default_authentication_plugin=mysql_native_password +port=3309 +user=mysql +memlock=on +bind-address=0.0.0.0 +skip-grant-tables + +lc-messages=en_US +lc-messages-dir=/usr/local/mysql/share + +innodb_page_size=64k +max_connections=4000 +back_log=2048 +performance_schema=OFF +max_prepared_stmt_count=128000 + +#file +innodb_file_per_table +innodb_log_file_size=2048M +innodb_log_files_in_group=32 +innodb_open_files=10000 +table_open_cache_instances=64 + +#buffers +innodb_buffer_pool_size=128G +innodb_buffer_pool_instances=64 +innodb_log_buffer_size=2048M + +#tune +sync_binlog=0 +innodb_flush_log_at_trx_commit=0 +innodb_use_native_aio=1 +innodb_spin_wait_delay=160 +innodb_sync_spin_loops=20 +innodb_flush_method=O_DIRECT +innodb_io_capacity=30000 +innodb_io_capacity_max=40000 +innodb_lru_scan_depth=9000 +innodb_page_cleaners=1 +innodb_spin_wait_pause_multiplier=25 + +#perf special +innodb_flush_neighbors=0 +innodb_write_io_threads=1 +innodb_read_io_threads=1 +innodb_purge_threads=1 + +sql_mode=STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_AUTO_VALUE_ON_ZERO,STRICT_ALL_TABLES + +#skop_log_bin +ssl=0 + +table_open_cache=15000 -- 2.23.0

2 1

[PATCH v5 lkp-tests] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 18 Nov '20

18 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git from /srv/git/pkg/ usage: job yaml add new field: PKGBUILD_TAG About $PKGBUILD_TAG, it is from PKGBUILD file's $pkgver-$pkgrel. example: PKGBUILD_TAG: 1.0-20 You should create a git repo that name same as $benchmark in /srv/git/pkg/ before you add PKGBUILD_TAG to yaml. If not, cci-makepkg will report an error. And everything same as before if no PKGBUILD_TAG. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 44be44b3..530c9fef 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -39,12 +39,29 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +curl_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +download_pkgfile() +{ + local pkg_tag_file="PKGBUILD-$PKGBUILD_TAG" + + curl_pkgfile "pkg/$benchmark" "$pkg_tag_file" + + [ -f "$pkg_tag_file" ] || die "$pkg_tag_file not exist" + [ -s "$pkg_tag_file" ] || die "$pkg_tag_file is empty" +} + get_pkg_info() { var=$1 pkg_dir=${2:-.} ( . $pkg_dir/PKGBUILD + [ -n "$PKGBUILD_TAG" ] && . $pkg_dir/PKGBUILD-$PKGBUILD_TAG eval echo '$'$var ) } @@ -71,6 +88,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && download_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

2 1

[PATCH v2 lkp-tests] spec/submit_spec: change existing spec cases input and output directory
by Wei Jihui 18 Nov '20

18 Nov '20

[Why] we will add more spec cases, so it should give each type of cases for different input and output directories. [How] move existing spec case to link_jobs, they all from lkp-tests/jobs. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 0 .../ftq/ftq-10000-2-100000ss-add.yaml | 0 .../ftq/ftq-20-2-6000ss-cache.yaml | 0 ...-68000000-1-65535-set_get-never-never.yaml | 0 spec/submit/redis.yaml | 1 - spec/submit_spec.rb | 26 +++++++++---------- 7 files changed, 13 insertions(+), 15 deletions(-) delete mode 120000 spec/submit/ftq.yaml rename spec/submit/{ => link_jobs}/ftq/ftq-1000-2-100000ss-add.yaml (100%) rename spec/submit/{ => link_jobs}/ftq/ftq-10000-2-100000ss-add.yaml (100%) rename spec/submit/{ => link_jobs}/ftq/ftq-20-2-6000ss-cache.yaml (100%) rename spec/submit/{ => link_jobs}/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml (100%) delete mode 120000 spec/submit/redis.yaml diff --git a/spec/submit/ftq.yaml b/spec/submit/ftq.yaml deleted file mode 120000 index 0ba22f05..00000000 --- a/spec/submit/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/ftq/ftq-1000-2-100000ss-add.yaml rename to spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml diff --git a/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/ftq/ftq-10000-2-100000ss-add.yaml rename to spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml diff --git a/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml similarity index 100% rename from spec/submit/ftq/ftq-20-2-6000ss-cache.yaml rename to spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 100% rename from spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml diff --git a/spec/submit/redis.yaml b/spec/submit/redis.yaml deleted file mode 120000 index 196e182b..00000000 --- a/spec/submit/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 69bd58af..983d050e 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -21,19 +21,19 @@ def traverse_file(output_dir) end describe 'submit job spec' do - Dir.glob("#{LKP_SRC}/spec/submit/*.yaml").each do |yaml_file| - job_name = File.basename(yaml_file, '.yaml') - output_dir = "#{LKP_SRC}/spec/submit/#{job_name}" - it 'save atomic yaml' do - submit_cmd = [ - "#{LKP_SRC}/sbin/submit", - '-o', output_dir, - '-s', 'lab: spec_lab', - '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file - ] - system(*submit_cmd) - traverse_file(output_dir) + it 'link jobs spec' do + Dir.glob("#{LKP_SRC}/spec/submit/link_jobs/*.yaml").each do |yaml_file| + job_name = File.basename(yaml_file, '.yaml') + output_dir = "#{LKP_SRC}/spec/submit/link_jobs/#{job_name}" + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) end end end -- 2.23.0

1 0

[PATCH compass-ci] container: add other group read permission
by Bai Jing 18 Nov '20

18 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- container/conserver/start | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/conserver/start b/container/conserver/start index 58b2242..4354630 100755 --- a/container/conserver/start +++ b/container/conserver/start @@ -31,3 +31,5 @@ cmd=( ) "${cmd[@]}" + +chmod o+r /srv/cci/serial/logs/* -- 2.23.0

2 2

[PATCH v3 lkp-tests 2/2] spec/stats/iozone: add test case for iozone
by Lu Kaiyi 18 Nov '20

18 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- spec/stats/iozone/1 | 32 ++++++++++++++++++++++++++++++++ spec/stats/iozone/1.yaml | 14 ++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 spec/stats/iozone/1 create mode 100644 spec/stats/iozone/1.yaml diff --git a/spec/stats/iozone/1 b/spec/stats/iozone/1 new file mode 100644 index 00000000..cd579643 --- /dev/null +++ b/spec/stats/iozone/1 @@ -0,0 +1,32 @@ +2020-11-17 14:30:58 iozone -r 256k -s 4g -i 0 -i 1 -i 2 -f /fs/sda/iozonetest & +2020-11-17 14:30:58 wait + Iozone: Performance Test of File I/O + Version $Revision: 3.479 $ + Compiled for 64 bit mode. + Build: linux + + Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins + Al Slater, Scott Rhine, Mike Wisner, Ken Goss + Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, + Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, + Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, + Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, + Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, + Vangel Bojaxhi, Ben England, Vikentsi Lapa, + Alexey Skidanov. + + Run began: Tue Nov 17 14:30:58 2020 + + Record Size 256 kB + File size set to 4194304 kB + Command line used: iozone -r 256k -s 4g -i 0 -i 1 -i 2 -f /fs/sda/iozonetest + Output is in kBytes/sec + Time Resolution = 0.000001 seconds. + Processor cache size set to 1024 kBytes. + Processor cache line size set to 32 bytes. + File stride size set to 17 * record size. + random random bkwd record stride + kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread + 4194304 256 14257802 15088999 8516076 8375138 5221373 5382018 + +iozone test complete. diff --git a/spec/stats/iozone/1.yaml b/spec/stats/iozone/1.yaml new file mode 100644 index 00000000..441d76ef --- /dev/null +++ b/spec/stats/iozone/1.yaml @@ -0,0 +1,14 @@ +4194304KB_256reclen.write: 14257802 +4194304KB_256reclen.rewrite: 15088999 +4194304KB_256reclen.read: 8516076 +4194304KB_256reclen.reread: 8375138 +4194304KB_256reclen.random_read: 5221373 +4194304KB_256reclen.random_write: 5382018 +write_KBps: 1096754.0 +rewrite_KBps: 1160692.2307692308 +read_KBps: 655082.7692307692 +reread_KBps: 644241.3846153846 +random_read_KBps: 401644.07692307694 +random_write_KBps: 414001.3846153846 +4194304KB_256reclen: 56841406.0 +average_KBps: 4372415.846153846 -- 2.23.0

2 1

[PATCH v7 compass-ci] make the container/initrd-cifs to share /srv/initrd
by Xiao Shenwei 18 Nov '20

18 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result in order to reduce the coupling, need split them [how] one folder mount by one container [usage] Two ways for the client to mount initrd: 1. initrd mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage: for lkp compatibility mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage --- container/initrd-cifs/Dockerfile | 24 ++++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 91 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..d4496bf --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,24 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN adduser -u 1090 -D lkp + +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash +RUN rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..4b93647 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..77558a1 --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

3 4

[PATCH v6 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 18 Nov '20

18 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..b7a88534 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,17 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + + +$stdin.each_line do |line| + next unless line =~ /^\s+total time:\s+([0-9.]+)/ + + puts "total time: #{$1.to_f}s" +end + + -- 2.23.0

2 1

[PATCH compass-ci] sparrow/3-code: add ENV INITRD_HTTP_HOST and OS_HTTP_HOST default values
by Liu Yinsi 18 Nov '20

18 Nov '20

Using the default value server_ip is convenient for customers to set up compass CI locally. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 77027d9..8928816 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -6,6 +6,8 @@ server_name=$(hostname | cut -f1 -d.) server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${sched_host:=$server_ip} : ${sched_port:=3000} +: ${os_http_host:=$server_ip} +: ${initrd_http_host:=$server_ip} : ${initrd_http_port:=8800} : ${MASTER_FLUENTD_HOST:=$server_ip} : ${MASTER_FLUENTD_PORT:=24224} -- 2.23.0

1 0

[PATCH v4 compass-ci] sbin/docker-pull: fix failed to pull image from local repository
by Liu Yinsi 18 Nov '20

18 Nov '20

[why] lys@crystal ~/compass-ci/sbin% ./docker-pull centos:7 pull image from local repository Error response from daemon: manifest for 127.0.0.1:5001/centos:7 not found pull image from docker hub 7: Pulling from library/centos Digest: sha256:0f4ec88e21daf75124b8a9e5ca03c37a5e937e0e108a255d890492430789b60e Status: Image is up to date for centos:7 [how] if pull image failed from local repository, it will pull from docker hub, so hide error info for first failed pull. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/docker-pull | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/docker-pull b/sbin/docker-pull index 5972913..d69af64 100755 --- a/sbin/docker-pull +++ b/sbin/docker-pull @@ -14,7 +14,7 @@ image_name=$1 local_repository() { echo "pull image from local repository" - docker pull $DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/$image_name && { + docker pull $DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/$image_name 2> /dev/null && { docker tag $DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/$image_name $image_name } } -- 2.23.0

1 0

[PATCH v3 compass-ci] providers/docker: fix failed to pull image from local repository
by Liu Yinsi 18 Nov '20

18 Nov '20

[why] lys@crystal ~/compass-ci/providers% ./my-docker Donehttp://$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/boot.container/hostname/dc-1g.lys-501708 pull image from local repository Error response from daemon: manifest for 127.0.0.1:5001/centos:7 not found pull image from docker hub 7: Pulling from library/centos [how] if pull image failed from local repository, it will pull from docker hub, so hide error info for first failed pull. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/docker/docker.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 8e489c6..849ea22 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -80,7 +80,7 @@ end def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] - system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" + system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image} 2> /dev/null" system( { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, ENV['CCI_SRC'] + '/providers/docker/run.sh' -- 2.23.0

3 4

[PATCH v9 lkp-tests] job2sh.rb: define on_state
by Hu Xuejiao 18 Nov '20

18 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By on_fail(sh_on_state), it will appear in job.sh: on_fail() { . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh run_test $LKP_SRC/tests/wrapper sleep 300 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c8b578d..14bf7159 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -363,14 +363,14 @@ class Job2sh < Job out_line "}\n\n" end - def sh_on_state(job = nil) + def sh_on_state(state = nil, job = nil) @script_lines = [] @programs = available_programs(:workload_elements) job ||= (@jobx || @job).clone - if job.include?('on_fail') + if state == 'on_fail' && job.include?('on_fail') @cur_func = :run_job -- 2.23.0

1 0

[PATCH v6 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 18 Nov '20

18 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..785aca73 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +$stdin.each_line do |line| + next unless line =~ /^s\+total time:\s+([0-9.]+)/ + + puts "total time: #{$1.to_f}s" +end -- 2.23.0

2 1

[PATCH v3 lkp-tests 1/2] stats/iozone: fix iozone cannot match the data line of part test
by Lu Kaiyi 18 Nov '20

18 Nov '20

[why] iozone can match the line like: 64 4 1599680 3057153 4274062 4897948 3588436 2892445 2772930 3057153 3958892 3057153 3022727 3363612 4564786 but cannot match the line as below(there are extra spaces after '2892445'): 64 4 1599680 3057153 4274062 4897948 3588436 2892445 [how] Add line.strip! before match Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/iozone | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/stats/iozone b/stats/iozone index 9f7ee72f..32dce7ba 100755 --- a/stats/iozone +++ b/stats/iozone @@ -8,7 +8,8 @@ per_record = {} all_sum = 0 nr_records = 0 while (line = STDIN.gets) - next unless line =~ /^\s*\d+.*\d+$/ + line.strip! + next unless line =~ /^\d+.*\d+$/ data = line.split data.each_index do |i| -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/docker-pull: fix failed to pull image from local repository
by Liu Yinsi 18 Nov '20

18 Nov '20

[why] lys@crystal ~/compass-ci/providers% ./my-docker Donehttp://$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/boot.container/hostname/dc-1g.lys-501708 pull image from local repository Error response from daemon: manifest for 127.0.0.1:5001/centos:7 not found pull image from docker hub 7: Pulling from library/centos [how] if pull image failed from local repository, it will pull from docker hub, so hide error info for first failed pull. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/docker/docker.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 8e489c6..849ea22 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -80,7 +80,7 @@ end def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] - system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" + system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image} 2> /dev/null" system( { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, ENV['CCI_SRC'] + '/providers/docker/run.sh' -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats/libmicro: fix rubocop offenses
by Lu Kaiyi 18 Nov '20

18 Nov '20

[errors] Inspecting 1 file C Offenses: libmicro:7:1: C: Layout/IndentationWidth: Use 2 (not 4) spaces for indentation. next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ ^^^^ libmicro:7:5: C: Layout/EmptyLineAfterGuardClause: Add empty line after guard clause. next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1 file inspected, 2 offenses detected Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/libmicro | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/stats/libmicro b/stats/libmicro index 530d8e2e..b7e895bf 100755 --- a/stats/libmicro +++ b/stats/libmicro @@ -4,6 +4,7 @@ # close_tmp 1 1 0.61880 201 0 640 $stdin.each_line do |line| - next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ - puts "#{$1}: #{$2}" + next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ + + puts "#{$1}: #{$2}" end -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/libmicro: fix rubocop warning
by Lu Kaiyi 18 Nov '20

18 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/libmicro | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/stats/libmicro b/stats/libmicro index 530d8e2e..b7e895bf 100755 --- a/stats/libmicro +++ b/stats/libmicro @@ -4,6 +4,7 @@ # close_tmp 1 1 0.61880 201 0 640 $stdin.each_line do |line| - next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ - puts "#{$1}: #{$2}" + next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ + + puts "#{$1}: #{$2}" end -- 2.23.0

2 1

[PATCH v3 lkp-tests] update command parameters
by Hu Xuejiao 18 Nov '20

18 Nov '20

[why] The current result: nr_threads: 2 "total_time" is 0.3927 nr_threads: 4 "total_time" is 0.3927 ... nr_threads: 8192 "total_time" is 0.3927 [how] After updating testcase: nr_threads: 2 "total_time" is 0.3927 nr_threads: 4 "total_time" is 0.5643 ... nr_threads: 8192 "total_time" is 1.678 Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git index df1bb5cf..be83b441 100755 --- a/tests/sysbench-threads-git +++ b/tests/sysbench-threads-git @@ -10,7 +10,7 @@ : "${thread_locks:=2}" args=( - --threads=$nr_threads + --num-threads=$nr_threads --thread-yields=$thread_yields --thread-locks=$thread_locks ) -- 2.23.0

2 1

[PATCH v5 compass-ci 4/4] assign-account: build send account email
by Luan Shengde 18 Nov '20

18 Nov '20

build email message for successfuly assign account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../build-send-account-email.rb | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/assign-account/build-send-account-email.rb diff --git a/container/assign-account/build-send-account-email.rb b/container/assign-account/build-send-account-email.rb new file mode 100755 index 0000000..3c41404 --- /dev/null +++ b/container/assign-account/build-send-account-email.rb @@ -0,0 +1,32 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_message(email, account_info) + message = <<~EMAIL_MESSAGE + To: #{email} + Subject: [compass-ci] jumper account is ready + + Dear user: + + Thank you for joining us. + You can use the following command to login the jumper server: + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + + You can use your pub_key to login if you have offered one. + + Suggest: + If you use the password to login, change it in time. + + regards + compass-ci + EMAIL_MESSAGE + + return message +end -- 2.23.0

2 4

[PATCH v8 compass-ci 04/11] mail-robot: check commit url for gitee
by Luan Shengde 18 Nov '20

18 Nov '20

GiteeCommitUrlCheck used to check commit for hub gitee.com when execute commit_url_availability in ParseApplyAccountEmail input: my_commit_url gitee_commit_check clone the repo and check commit exists and availalbe Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 container/mail-robot/lib/gitee-commit-url-check.rb diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb new file mode 100755 index 0000000..23b91f1 --- /dev/null +++ b/container/mail-robot/lib/gitee-commit-url-check.rb @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# used to check commit available for hub gitee.com +# it is called when: +# - ParseApplyAccountEmail execute commit_url_availability +# - hub is gitee.com +# gitee_commit_check +# clone the repo +# check commit available +class GiteeCommitUrlCheck + def initialize(my_info, url, base_url) + @my_info = my_info + @url = url + @base_url = base_url + end + + def gitee_commit_check + repo_dir = @url.split('/')[-3] + repo_url = [@base_url, 'git'].join('.') + commit_id = @url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] + + FileUtils.rm_rf repo_dir + + gitee_commit_exist(email_index) + end + + def gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = "We can not conform whether the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with your email.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH lkp-tests] lib/job.rb: change the way to analyze the testbox field
by Li Ping 18 Nov '20

18 Nov '20

If user wants to reproduce the result directly in the result directory. [problem] /srv/result/iperf/vm-2p8g/2020-11-09/z9.147520% submit job.yaml output: /home/liping/lkp-tests/lib/job.rb:344:in `get_hosts_file': hosts file not exist: vm-2p8g.taishan200-2280-2s64p-256g, maybe need check testbox field (ArgumentError) --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..6dd93f01 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -337,7 +337,7 @@ class Job lab_hosts_file = get_lab_hosts_file return lab_hosts_file if lab_hosts_file - hosts_file_name = @job['tbox_group'].split('--')[0] + hosts_file_name = @job['tbox_group'].split(/(\.|--)/)[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" return hosts_file if File.file?(hosts_file) -- 2.23.0

2 2

[PATCH compass-ci] provide my-docker script for user
by Xiao Shenwei 18 Nov '20

18 Nov '20

Different users can run it concurrently while avoiding hostname collision. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/my-docker | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100755 providers/my-docker diff --git a/providers/my-docker b/providers/my-docker new file mode 100755 index 0000000..c8420c7 --- /dev/null +++ b/providers/my-docker @@ -0,0 +1,13 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './docker/docker' + +tbox_group = 'dc-1g' +hostname = "#{tbox_group}.#{ENV['USER']}-#{Process.pid}" +# specify which queues will be request, use "," to separate more than 2 values +queues = "#{tbox_group}~#{ENV['USER']}" + +main(hostname, queues) -- 2.23.0

4 4

[PATCH compass-ci] sbin/docker-pull: fix failed to pull image from local repository
by Liu Yinsi 18 Nov '20

18 Nov '20

[why] lys@crystal ~/compass-ci/providers% ./my-docker dc-1g.lys dc-1g~lys Donehttp://$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/boot.container/hostname/dc-1g.lys-501708 pull image from local repository Error response from daemon: manifest for 127.0.0.1:5001/centos:7 not found pull image from docker hub 7: Pulling from library/centos because lacking of judgment before pull image from local repository, if local repository not found images, it will show error message. [how] if no images in local repository, not pull from local repository, just pull from docker hub. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/docker-pull | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/sbin/docker-pull b/sbin/docker-pull index 5972913..d5b90dc 100755 --- a/sbin/docker-pull +++ b/sbin/docker-pull @@ -25,9 +25,16 @@ docker_hub() docker pull $image_name } +query_local_repository() +{ + local image_array=($(echo "$image_name"|tr ':' ' ')) + image=$(curl -s -X GET http://$DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/v2/${image_array[0]}/manifests/${image_array[1]}) + echo $image| grep -q ${image_array[0]} && local_repository +} + main() { - local_repository || docker_hub + query_local_repository || docker_hub } main -- 2.23.0

2 2

[PATCH compass-ci 2/2] sched: fix taskqueue hand over cluster job 409
by Cao Xueliang 18 Nov '20

18 Nov '20

[why] When add a cluster job to taskqueue use host, but we don't change the queue of job to the host. So when finished job we hand over the queue of job to extract queue will get 409 error. [how] Set the queue of job to the host. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/submit_job.cr | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 8871464..8838e40 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -49,8 +49,8 @@ class Sched # steps for each host cluster_config["nodes"].as_h.each do |host, config| - tbox_group = host.to_s - job_id = add_task(tbox_group, lab) + queue = host.to_s + job_id = add_task(queue, lab) # return when job_id is '0' # 2 Questions: @@ -58,15 +58,16 @@ class Sched # - may consume job before all jobs done return job_messages << { "job_id" => "0", - "message" => "add task queue sched/#{tbox_group} failed", + "message" => "add task queue sched/#{queue} failed", "job_state" => "submit", } unless job_id job_ids << job_id # add to job content when multi-test - job["testbox"] = tbox_group - job.update_tbox_group(tbox_group) + job["testbox"] = queue + job["queue"] = queue + job.update_tbox_group(queue) job["node_roles"] = config["roles"].as_a.join(" ") direct_macs = config["macs"].as_a direct_ips = [] of String -- 2.23.0

3 3

[PATCH compass-ci] scheduler: fix: check whether a job is cluster job or not
by Ren Wen 18 Nov '20

18 Nov '20

How to check whether a job is cluster job or not at LKP? should_wait_cluster() { [ -z "$LKP_SERVER" ] && return 1 [ -z "$node_roles" ] && return 1 [ "$cluster" = "cs-localhost" ] && return 1 return 0 } So when you need a cluster job, the value of cluster field should not be "cs-localhost". Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 8838e40..1297b57 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -9,10 +9,10 @@ class Sched job = Job.new(job_content, job_content["id"]?) job["commit_date"] = get_commit_date(job) - # it is not a cluster job if cluster field is empty or - # field's prefix is 'cs-localhost' + # how to execute a cluster job: + # has a cluster field, and the value is not "cs-localhost" cluster_file = job["cluster"] - if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") + if cluster_file.empty? || cluster_file == "cs-localhost" return submit_single_job(job) else cluster_config = get_cluster_config(cluster_file, -- 2.23.0

2 3

[PATCH v2 lkp-tests] lib/reboot.sh: fix archlinux reboot problem
by Wang Yong 18 Nov '20

18 Nov '20

[why] my-qemu.sh can't exit after job done, because archlinux don't use 'reboot' command to reboot, it use 'systemctl reboot' to reboot [how] judge os, use 'systemctl reboot' command when ${os} is archlinux Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/reboot.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/lib/reboot.sh b/lib/reboot.sh index 8557c5972..afe511ca1 100644 --- a/lib/reboot.sh +++ b/lib/reboot.sh @@ -2,5 +2,9 @@ reboot_tbox() { - reboot + if [ "${os}" = "archlinux" ]; then + systemctl reboot + else + reboot + fi } -- 2.23.0

1 0

[PATCH lkp-tests] lib/reboot.sh: fix archlinux reboot problem
by Wang Yong 18 Nov '20

18 Nov '20

[why] my-qemu.sh can't exit after job done, because archlinux don't use 'reboot' command to reboot, it use 'systemctl reboot' to reboot [how] judge os, use 'systemctl reboot' command when ${os} is archlinux Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/reboot.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/reboot.sh b/lib/reboot.sh index 8557c5972..5551dc586 100644 --- a/lib/reboot.sh +++ b/lib/reboot.sh @@ -2,5 +2,9 @@ reboot_tbox() { + [ ${os} == "archlinux" ] && { + systemctl reboot + } + reboot } -- 2.23.0

2 2

[PATCH v3 lkp-tests 2/2] job: fix each_jobs to fit multi-args
by Bai Jing 17 Nov '20

17 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash={"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03","os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20","os_arch"=>"aarch64"} h1 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} We need to merge the h2 to @job Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index f3b7a092..f8608e94 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -727,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] lib: add multi-args in job yaml
by Bai Jing 17 Nov '20

17 Nov '20

[original style] os: openeuler os_version: 20.03 os_arch: aarch64 os: openeuler os_version: 20 os_arch: aarch64 [now style] os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 [point] Currently, only one key of this style(xx xxx xx) can be contained in YAML files. Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..f3b7a092 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -250,7 +250,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) - @jobs << hash + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e log_error "#{jobfile}: " + e.message @@ -280,6 +280,30 @@ class Job @jobfile = jobfile end + def multi_args(hash) + jobs_array = [] + hash.each { |key, value| + next unless key =~ /^\w.*\s+.*\w$/ + + next unless value.class == Array + + key_array = key.split + value.each do |v| + v_array = v.split + next unless key_array.size == v_array.size + + hash_job = hash.clone + key_array.size.times do |i| + hash_job[key_array[i]] = v_array[i] + end + hash_job.delete(key) + jobs_array << hash_job + end + } + jobs_array = [hash] if jobs_array.empty? + return jobs_array + end + def delete_keys_from_spec(spec_file) return unless File.exist? spec_file -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit_spec: change existing spec cases input and output directory
by Wei Jihui 17 Nov '20

17 Nov '20

[Why] we will add more spec cases, so it should give each type of cases for different input and output directories. [How] move existing spec case to link_jobs, they all from lkp-tests/jobs. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 0 .../ftq/ftq-10000-2-100000ss-add.yaml | 0 .../ftq/ftq-20-2-6000ss-cache.yaml | 0 ...-68000000-1-65535-set_get-never-never.yaml | 0 spec/submit/redis.yaml | 1 - spec/submit_spec.rb | 27 ++++++++++--------- 7 files changed, 14 insertions(+), 15 deletions(-) delete mode 120000 spec/submit/ftq.yaml rename spec/submit/{ => link_jobs}/ftq/ftq-1000-2-100000ss-add.yaml (100%) rename spec/submit/{ => link_jobs}/ftq/ftq-10000-2-100000ss-add.yaml (100%) rename spec/submit/{ => link_jobs}/ftq/ftq-20-2-6000ss-cache.yaml (100%) rename spec/submit/{ => link_jobs}/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml (100%) delete mode 120000 spec/submit/redis.yaml diff --git a/spec/submit/ftq.yaml b/spec/submit/ftq.yaml deleted file mode 120000 index 0ba22f05..00000000 --- a/spec/submit/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/ftq/ftq-1000-2-100000ss-add.yaml rename to spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml diff --git a/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml similarity index 100% rename from spec/submit/ftq/ftq-10000-2-100000ss-add.yaml rename to spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml diff --git a/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml similarity index 100% rename from spec/submit/ftq/ftq-20-2-6000ss-cache.yaml rename to spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 100% rename from spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml diff --git a/spec/submit/redis.yaml b/spec/submit/redis.yaml deleted file mode 120000 index 196e182b..00000000 --- a/spec/submit/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 69bd58af..67e05749 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -21,19 +21,20 @@ def traverse_file(output_dir) end describe 'submit job spec' do - Dir.glob("#{LKP_SRC}/spec/submit/*.yaml").each do |yaml_file| - job_name = File.basename(yaml_file, '.yaml') - output_dir = "#{LKP_SRC}/spec/submit/#{job_name}" - it 'save atomic yaml' do - submit_cmd = [ - "#{LKP_SRC}/sbin/submit", - '-o', output_dir, - '-s', 'lab: spec_lab', - '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file - ] - system(*submit_cmd) - traverse_file(output_dir) + it 'link jobs spec' do + Dir.glob("#{LKP_SRC}/spec/submit/link_jobs/*.yaml").each do |yaml_file| + p yaml_file + job_name = File.basename(yaml_file, '.yaml') + output_dir = "#{LKP_SRC}/spec/submit/link_jobs/#{job_name}" + submit_cmd = [ + "#{LKP_SRC}/sbin/submit", + '-o', output_dir, + '-s', 'lab: spec_lab', + '-s', 'testbox: vm-2p8g--spec_submit', + yaml_file + ] + system(*submit_cmd) + traverse_file(output_dir) end end end -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests/sysbench-mysql: adapt to the sysbench test of a specified version
by Zhang Yu 17 Nov '20

17 Nov '20

[why] use the special version of sysbench test mysql required by the customer. so need to use different sysbench script. [how] add parameters to test, and use different sysbench script. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 88 ++++++++++++++++++++++++++++++++------------ 1 file changed, 65 insertions(+), 23 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 4ab5b453..106ee0b9 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,52 +1,94 @@ #!/bin/sh -# mysql_user -# mysql_host -# mysql_port -# mysql_db -# db_driver # - oltp_test_mode # - oltp_tables_count # - oltp_table_size +# - max_requests +# - mysql_table_engine # - nr_threads +# - rand_type +# - rand_spec_pct # - runtime # - report_interval +# - oltp_read_only +# - events +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +# db_driver : "${mysql_user:=root}" -: "${mysql_host:=localhost}" -: "${mysql_port:=3306}" -: "${mysql_db:=test_1000}" +: "${mysql_host:=$direct_server_ips}" +: "${mysql_port:=$mysql_port}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=$mysql_password}" : "${db_driver:=mysql}" -: "${oltp_tables_count:=3}" -: "${oltp_table_size:=1000}" -: "${nr_threads:=64}" -: "${runtime:=120}" -: "${report_interval:=10}" +: "${oltp_test_mode:=complex}" +: "${oltp_tables_count:=1000}" +: "${oltp_table_size:=100000}" +: "${max_requests:=0}" +: "${mysql_table_engine:=innodb}" +: "${rand_type:=special}" +: "${rand_spec_pct:=100}" +: "${nr_threads:=256}" +: "${runtime:=7200}" +: "${report_interval:=1}" +: "${oltp_read_only:=off}" +: "${events:=2000000000}" -args=( +args1=( --mysql-user=$mysql_user --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db + --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver - --tables=$oltp_tables_count - --table-size=$oltp_table_size + --mysql-password=$mysql_password + --max-requests=$max_requests + --mysql-table-engine=$mysql_table_engine + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --rand-type=$rand_type + --rand-spec-pct=$rand_spec_pct --threads=$nr_threads --time=$runtime - --report-interval=$report_interval ) +args2=( + --mysql-user=$mysql_user + --mysql-password=$mysql_password + --mysql-host=$mysql_host + --mysql-port=$mysql_port + --mysql-db=$mysql_db + --threads=$nr_threads + --oltp-read-only=$oltp_read_only + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --report-interval=$report_interval + --time=$runtime + --events=$events +) + +stop_firewalld() +{ + systemctl stop firewalld + iptables -F +} + run_sysbench_step() { - sysbench /usr/share/sysbench/$1 "${args[@]}" $2 + lua_script=$1 + shift + sysbench /usr/local/share/sysbench/tests/include/oltp_legacy/$lua_script "$@" } run_sysbench_mysql() { - systemctl start mysqld - mysql -e "create database test_1000;" - run_sysbench_step oltp_common.lua prepare - run_sysbench_step oltp_read_write.lua run - run_sysbench_step oltp_common.lua cleanup + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + run_sysbench_step oltp.lua ${args2[@]} run + run_sysbench_step oltp.lua ${args2[@]} cleanup } +stop_firewalld run_sysbench_mysql -- 2.23.0

1 0

[PATCH v5 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 17 Nov '20

17 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..1e395231 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +$stdin.each_line do |line| + next unless line =~ /^s\+total time:\s+([0-9.]+)/ + time = $1.to_f + + puts "total time: #{time}s" +end -- 2.23.0

1 0

[PATCH v5 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 17 Nov '20

17 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..28e039c3 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +$stdin.each_line do |line| + next unless line =~ /^s\+total time:\s+([0-9.]+)/ + + results_total['total_time'] = $1.to_f +end + +results_total.each do |key, vals| + puts "#{key}: #{vals}s" +end -- 2.23.0

1 0

[PATCH v5 lkp-tests] add: sysbench-threads-git result parsing file [input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms
by Hu Xuejiao 17 Nov '20

17 Nov '20

[output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..df8181bd --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,21 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +$stdin.each_line do |line| + next unless line =~ /^s\+total time:\s+([0-9.]+)/ + + results_total['total_time'] = $1.to_f +end + +results_total.each do |key, vals| + puts "#{key}: #{vals}s" +end + -- 2.23.0

1 0

[PATCH v4 lkp-tests 4/4] test/cci-makepkg: fix a softlink will be established if the package fails
by Sun Yukui 17 Nov '20

17 Nov '20

After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 3914f542..44be44b3 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -111,7 +111,8 @@ build_source_pkg() echo "$makepkg" PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + + [ "$?" == 0 ] && update_softlink } build_source_pkg -update_softlink && echo "Makepkg finished successfully" -- 2.23.0

1 0

[PATCH v4 lkp-tests 3/4] test/cci-makepkg: Refactor encapsulate the statement as a function
by Sun Yukui 17 Nov '20

17 Nov '20

[why] Encapsulating statements as functions is important for a better syntax. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..3914f542 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,13 @@ update_softlink() rm $sync_dest/$cgz_name } +build_source_pkg() +{ + local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" + + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg +} + +build_source_pkg update_softlink && echo "Makepkg finished successfully" -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/4] test/build-pkg: fix a softlink will be established if the package fails
by Sun Yukui 17 Nov '20

17 Nov '20

After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 6426f2da..4aaa4fef 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -101,10 +101,11 @@ build_source_pkg() cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 + + [ "$?" == 0 ] && create_softlink } check_vars mount_dest request_pkg build_source_pkg -create_softlink -- 2.23.0

1 0

[PATCH v4 lkp-tests 1/4] test/build-pkg: Refactor move create_softlink function position
by Sun Yukui 17 Nov '20

17 Nov '20

[why] The create_softlink function needs to be called in build_source_pkg function. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 1eed784f..6426f2da 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,21 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + [ -e "$cgz_name" ] || return 0 + + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -88,22 +103,6 @@ build_source_pkg() $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 } -create_softlink() -{ - [ -e "$cgz_name" ] || return 0 - - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" -} - check_vars mount_dest request_pkg -- 2.23.0

1 0

[PATCH v3 lkp-tests 4/4] test/cci-makepkg: fix a softlink will be established if the package fails
by Sun Yukui 17 Nov '20

17 Nov '20

After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 43ba04ad..a3b5e960 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -110,7 +110,8 @@ build_source_pkg(){ echo "$makepkg" PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + + [ "$?" == 0 ] && update_softlink } build_source_pkg -update_softlink && echo "Makepkg finished successfully" -- 2.23.0

1 0

[PATCH v3 lkp-tests 3/4] test/cci-makepkg: Refactor encapsulate the statement as a function
by Sun Yukui 17 Nov '20

17 Nov '20

[why] Encapsulating statements as functions is important for a better syntax. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..43ba04ad 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,12 @@ update_softlink() rm $sync_dest/$cgz_name } +build_source_pkg(){ + local makepkg="$LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf" + echo "$makepkg" + + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg +} + +build_source_pkg update_softlink && echo "Makepkg finished successfully" -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/4] test/build-pkg: fix a softlink will be established if the package fails
by Sun Yukui 17 Nov '20

17 Nov '20

After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 6426f2da..4aaa4fef 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -101,10 +101,11 @@ build_source_pkg() cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 + + [ "$?" == 0 ] && create_softlink } check_vars mount_dest request_pkg build_source_pkg -create_softlink -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/4] test/build-pkg: Refactor move create_softlink function position
by Sun Yukui 17 Nov '20

17 Nov '20

[why] The create_softlink function needs to be called in build_source_pkg function. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 1eed784f..6426f2da 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,21 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + [ -e "$cgz_name" ] || return 0 + + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -88,22 +103,6 @@ build_source_pkg() $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 } -create_softlink() -{ - [ -e "$cgz_name" ] || return 0 - - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" -} - check_vars mount_dest request_pkg -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/4] test/build-pkg: Refactor move create_softlink function position
by Sun Yukui 17 Nov '20

17 Nov '20

[why] The create_softlink function needs to be called in build_source_pkg function. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 1eed784f..6426f2da 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,21 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + [ -e "$cgz_name" ] || return 0 + + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -88,22 +103,6 @@ build_source_pkg() $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 } -create_softlink() -{ - [ -e "$cgz_name" ] || return 0 - - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" -} - check_vars mount_dest request_pkg -- 2.23.0

2 1

[PATCH v8 lkp-tests 3/3] execute on_fail when exit-code isn't 0
by Hu Xuejiao 17 Nov '20

17 Nov '20

[why] When a job-task fails(exit_code != 0), the on_fail function needs to be called to keep the program in the waiting state to user adjustment Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- bin/run-lkp | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/bin/run-lkp b/bin/run-lkp index 26669331..b422a7eb 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -97,6 +97,10 @@ elif [ "$run_job_exit_code" != 0 ]; then else set_job_state 'failed' fi + + source $job_script + [ "$(type -t on_fail)" = "function" ] && $LKP_DEBUG_PREFIX $job_script on_fail + elif [ -f '/var/log/wtmp' ] && has_cmd last && last -n1 | grep -q -E '(pts/|tty)[0-9].*(still logged in)'; then set_job_state 'disturbed' disturbed=1 -- 2.23.0

2 1

[PATCH v4 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 17 Nov '20

17 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..08faf4e0 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +$stdin.each_line do |line| + next unless line =~ /total time:\s+([0-9.]+)/ + + results_total['total_time'] = $1.to_f +end + +results_total.each do |key, vals| + puts "#{key}: #{vals}s" +end -- 2.23.0

2 1

[PATCH v8 compass-ci 11/11] mail-robot: documentation for service
by Luan Shengde 17 Nov '20

17 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 98 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..ec1a866 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,98 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## usage + +build email as the following section: allowed email format +send mail to: compass-ci(a)qq.com + +## allowed email format: + +subject: apply account +mail content + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. apply-account + init my_info + - my_email + - my_name + check_to_send_account + - parse_mail_content + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there has an attachment file to the email file + attachment: + first attachment + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + - no more available account + build_error_email with raised error message + + - send_mail + - build success email + - build failed email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

1 0

[PATCH v8 compass-ci 10/11] mail-robot: start file
by Luan Shengde 17 Nov '20

17 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..5827e18 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +defaults['MAILDIR'] = '/srv/cci/Maildir/.compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v8 compass-ci 09/11] mail-robot: build file
by Luan Shengde 17 Nov '20

17 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v8 compass-ci 08/11] mail-robot: Dockerfile
by Luan Shengde 17 Nov '20

17 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v8 compass-ci 07/11] mail-robot: build fail message
by Luan Shengde 17 Nov '20

17 Nov '20

build fail email message with raised error message - lack commit url - repo url for the commit url not in upstream - commit not exist or not available - lack pubkey - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 container/mail-robot/lib/assign-account-fail-eamil.rb diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..769853f --- /dev/null +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following error: + + #{message} + + In order to successfully apply an account, please pay attention to the following points: + + 1. mail subject + The subject should exactly: apply account + + 2. commit url + When you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + Attention: + Ensure you commit url exist and available to access. + + 3. ssh pubkey + You need to add a pubkey as an attachment to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v8 compass-ci 06/11] mail-robot: build success email
by Luan Shengde 17 Nov '20

17 Nov '20

build email message for successfully assigning account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/assign-account-email.rb | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 container/mail-robot/lib/assign-account-email.rb diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb new file mode 100755 index 0000000..e882bb4 --- /dev/null +++ b/container/mail-robot/lib/assign-account-email.rb @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_email(my_info) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{my_info['my_name']}, + + Thank you for joining us. + + You can use the following info to submit jobs: + + notice: + the followings are ONE-TIME setup + + 1) setup default config + run the following command to add the below setup to default config file + + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF + my_email: #{my_info['my_email']} + my_name: #{my_info['my_name']} + my_uuid: #{my_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + + reference to 'how to write job yaml' section to write the job yaml + you can also reference to files in compass-ci/jobs as example. + + submit jobs, for example: + + submit -m iperf.yaml testbox=vm-2p8g queue=vm-2p8g~\\$USER + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v8 compass-ci 05/11] mail-robot: apply new account for user
by Luan Shengde 17 Nov '20

17 Nov '20

ApplyJumperAccount used to apply account for user when successfully parsed my_commit_url and my_ssh_pubkey input: my_info apply_jumper_account apply account with my_info my_info: - my_email - my_name - my_uuid - my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/apply-jumper-account.rb | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100755 container/mail-robot/lib/apply-jumper-account.rb diff --git a/container/mail-robot/lib/apply-jumper-account.rb b/container/mail-robot/lib/apply-jumper-account.rb new file mode 100755 index 0000000..44f412b --- /dev/null +++ b/container/mail-robot/lib/apply-jumper-account.rb @@ -0,0 +1,42 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 + +# used to apply account +# be called after AssignAccount successfully parsed my_commit_url and my_ssh_pubkey +# apply_jumper_account +# apply jumper account with my_info and my_ssh_pubkey +# account_info_exist +# check account exists +class ApplyJumperAccount + def initialize(my_info) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + end + + def apply_jumper_account + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = 'No more available jumper account.' + error_message += 'You may try again later or consulting the manager for a solution.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v8 compass-ci 03/11] mail-robot: parse apply account email
by Luan Shengde 17 Nov '20

17 Nov '20

ParseApplyAccountEmail used to parse my_email, my_name, my_commit_url and my_ssh_pubkey when execute check_to_send_account in ApplyAccount input: mail_content output: my_info init my_info parse_commit_url extract_commit_url extract my_commit_url from mail_content base_url_in_upstream_repos check whether the repo's url in upstream-repos commit_url_availability check the commit available - gitee.com call GiteeCommitUrlCheck to check commit available - non-gitee.com execute curl and check feedback for the commit parse_pub_key check attachment for my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 134 ++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100755 container/mail-robot/lib/parse-apply-account-email.rb diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb new file mode 100755 index 0000000..5467728 --- /dev/null +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -0,0 +1,134 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'gitee-commit-url-check' + +# used to parse mail_content for my_commit_url and my_ssh_pubkey +# be called by AssignAccount when it needs to extract required data: +# - my_commit_url +# - my_ssh_pubkey +# +# input: mail_content +# output: my_commit_url, my_ssh_pubkey +# +# parse_commit_url +# parse my_commit_url from the mail_content and return it +# extract my_commit_url +# extract my_coomit_url from the mail_content +# check commit url exists +# base_url_in_upstream_repos +# check whether the base url in upstream-repos +# commit_url_availability +# check commit available +# gitee.com: +# call GiteeCommitUrlCheck to check the commit +# non-gitee.com: +# execute curl to check the commit +# parse_pub_key +# check my_ssh_pubkey exists and return it +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_ssh_pubkey' => [] + } + end + + def build_my_info + @my_info['my_commit_url'] = parse_commit_url + @my_info['my_ssh_pubkey'] << parse_pub_key + + return @my_info + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + + # the commit url should be headed with a prefix: my oss commit + # the commit url must be in a standart format, example: + # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = "No matched commit url found.\n" + error_message += 'Ensure that you have add a right commit url, ' + error_message += "and with prefix 'my oss commit:'." + + raise error_message + end + + def parse_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + base_url_in_upstream_repos('/c/upstream-repos', base_url) + commit_url_availability(url, base_url) + + return url + end + + def base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = "The repo url for your commit is not in our upstream-repo list.\n" + error_message += 'Use a new one, or consulting the manager for available repo list.' + + raise error_message + end + + def commit_url_availability(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit information + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + gitee_commit(url, base_url) + else + non_gitee_commit(url) + end + end + + def gitee_commit(url, base_url) + my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) + my_gitee_commit.gitee_commit_check + end + + def non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_info['my_email'] + + return unless email_index.nil? + + error_message = "We can not confirm the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end + + def parse_pub_key + error_message = "No pub_key found.\n" + error_message += 'Please add a pub_key as an attachment to your email.' + + raise error_message if @mail_content.attachments.empty? + raise error_message unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ + + pub_key = @mail_content.attachments[0].body.decoded + + return pub_key + end +end -- 2.23.0

1 0

[PATCH v8 compass-ci 02/11] mail-robot: check to apply account
by Luan Shengde 17 Nov '20

17 Nov '20

ApplyAccount check to apply account for user when listened new email with subject 'apply account' input: mail_content check_to_send_account parse_mail_content update my_info apply_my_account generate uuid apply account with my_info my_info: - my_email - my_name - my_uuid - my_ssh_pubkey store account info to es send email send success email send fail email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 110 ++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100755 container/mail-robot/lib/apply-account.rb diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb new file mode 100755 index 0000000..bf40429 --- /dev/null +++ b/container/mail-robot/lib/apply-account.rb @@ -0,0 +1,110 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '../../../lib/es_client' +require_relative 'assign-account-email' +require_relative 'assign-account-fail-eamil' +require_relative 'apply-jumper-account' +require_relative 'parse-apply-account-email' + +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +# assign uuid/account for user +# when mail-robot listened new email, and the email's subject +# exactly equal 'apply account', mail-robot will call this class +# entry point: send_account +# input: email's content +# +# send_account +# parse_mail_content +# call ParseApplyAccountEmail to parse: +# - my_commit_url +# - my_ssh_pubkey +# apply_my_account +# call ApplyJumperAccount to apply new account +# required data: +# my_info: +# - my_email +# - my_name +# - my_uuid +# my_ssh_pubkey +# store_account_info +# call ESClient to store my_info +# my_info: +# - my_email +# - my_name +# - my_uuid +# - my_commit_url +# - my_login_name +# - my_ssh_pubkey +# send_mail +# when successfully applied an account +# call build_apply_account_email to send a successful email +# when rescued error message +# call build_apply_account_fail_email to send fail email +class ApplyAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = {} + end + + def check_to_send_account + # in case of failed parsing, parse_mail_content will return none info + # in order to successfully send email for failed parsing + # firstly get my_email before execute parse_mail_content is needed + @my_info['my_email'] = @mail_content.from[0] + parse_mail_content + acct_info = apply_my_account + + @my_info['my_login_name'] = acct_info['my_login_name'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_mail_content + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + parsed_email_info = parse_apply_account_email.build_my_info + + @my_info.update parsed_email_info + end + + def apply_my_account + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(error_message) + email_message = if error_message.empty? + build_apply_account_email(@my_info) + else + build_apply_account_fail_email(@my_info, error_message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

1 0

[PATCH v8 compass-ci 01/11] mail-robot: listen and handle new email
by Luan Shengde 17 Nov '20

17 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/mail-robot.rb | 41 ++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..a809eae --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'lib/apply-account' + +MAILDIR = ENV['MAILDIR'] + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = ApplyAccount.new(mail_content) + assign_uuid.check_to_send_account +end + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] spec/stats/iozone: add test case for iozone
by Lu Kaiyi 17 Nov '20

17 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- spec/stats/iozone/1 | 61 ++++++++++++++++++++++++++++++++++++++++ spec/stats/iozone/1.yaml | 14 +++++++++ 2 files changed, 75 insertions(+) create mode 100644 spec/stats/iozone/1 create mode 100644 spec/stats/iozone/1.yaml diff --git a/spec/stats/iozone/1 b/spec/stats/iozone/1 new file mode 100644 index 00000000..58f2fcb0 --- /dev/null +++ b/spec/stats/iozone/1 @@ -0,0 +1,61 @@ +==> /tmp/stdout <== + +==> /tmp/stderr <== + +==> /tmp/stdout <== +RESULT_ROOT=/result/iozone/taishan200-2280-2s64p-256g--a9/2020-11-17/z9.151579 +job=/lkp/scheduled/job.yaml +result_service=raw_upload, RESULT_MNT=/172.168.131.2/result, RESULT_ROOT=/172.168.131.2/result/iozone/taishan200-2280-2s64p-256g--a9/2020-11-17/z9.151579 +run-job /lkp/scheduled/job.yaml +/usr/bin/wget -q --timeout=1800 --tries=1 --local-encoding=UTF-8 http://172.168.131.2:3000/~lkp/cgi-bin/lkp-jobfile-append-var?job_file=/lkp… -O /dev/null +target ucode: +2020-11-17 14:30:52 dmsetup remove_all +2020-11-17 14:30:52 wipefs -a --force /dev/sda +/dev/sda: 4 bytes were erased at offset 0x00000000 (xfs): 58 46 53 42 +2020-11-17 14:30:52 mkfs -t xfs -f /dev/sda +meta-data=/dev/sda isize=512 agcount=4, agsize=73257062 blks + = sectsz=512 attr=2, projid32bit=1 + = crc=1 finobt=1, sparse=1, rmapbt=0 + = reflink=0 +data = bsize=4096 blocks=293028246, imaxpct=5 + = sunit=0 swidth=0 blks +naming =version 2 bsize=4096 ascii-ci=0, ftype=1 +log =internal log bsize=4096 blocks=143080, version=2 + = sectsz=512 sunit=0 blks, lazy-count=1 +realtime =none extsz=4096 blocks=0, rtextents=0 +2020-11-17 14:30:55 mkdir -p /fs/sda +2020-11-17 14:30:55 modprobe xfs +2020-11-17 14:30:56 mount -t xfs -o inode64 /dev/sda /fs/sda +2020-11-17 14:30:58 iozone -r 256k -s 4g -i 0 -i 1 -i 2 -f /fs/sda/iozonetest & +2020-11-17 14:30:58 wait +perf version 4.19.90-2003.4.0.0036.oe1.aarch64 + Iozone: Performance Test of File I/O + Version $Revision: 3.479 $ + Compiled for 64 bit mode. + Build: linux + + Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins + Al Slater, Scott Rhine, Mike Wisner, Ken Goss + Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, + Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, + Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, + Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, + Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer, + Vangel Bojaxhi, Ben England, Vikentsi Lapa, + Alexey Skidanov. + + Run began: Tue Nov 17 14:30:58 2020 + + Record Size 256 kB + File size set to 4194304 kB + Command line used: iozone -r 256k -s 4g -i 0 -i 1 -i 2 -f /fs/sda/iozonetest + Output is in kBytes/sec + Time Resolution = 0.000001 seconds. + Processor cache size set to 1024 kBytes. + Processor cache line size set to 32 bytes. + File stride size set to 17 * record size. + random random bkwd record stride + kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread + 4194304 256 14257802 15088999 8516076 8375138 5221373 5382018 + +iozone test complete. diff --git a/spec/stats/iozone/1.yaml b/spec/stats/iozone/1.yaml new file mode 100644 index 00000000..3ddec7fb --- /dev/null +++ b/spec/stats/iozone/1.yaml @@ -0,0 +1,14 @@ +suite: iozone +category: benchmark +file_size: 4g +test: write, read, rand_rw +block_size: 256k +disk: 1HDD + +fs: +- xfs + +iosched: +- kyber + +iozone: -- 2.23.0

1 0

[PATCH v5 lkp-tests] distro/archlinux: add archlinux distro file
by Wang Yong 17 Nov '20

17 Nov '20

add archlinux pack method Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 111 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100755 distro/archlinux diff --git a/distro/archlinux b/distro/archlinux new file mode 100755 index 000000000..408c5fa23 --- /dev/null +++ b/distro/archlinux @@ -0,0 +1,111 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +shopt -s nullglob + +. $LKP_SRC/lib/install.sh +. $LKP_SRC/distro/common +. $LKP_SRC/lib/debug.sh + +pack_benchmark_deps() +{ + setup_proxy + + for BM_NAME in $benchmark + do + + cleanup_downloaded_pkgs + + check_shared_package $BM_NAME + + packages=$(echo $(get_dependency_packages $DISTRO $BM_NAME)) + + if pacman_download $packages; then + pacman_pack + else + echo "failed to pack-deps $BM_NAME" >&2 + fi + done +} + +cleanup_downloaded_pkgs() +{ + [[ -d /var/cache/pacman/pkg/ ]] && { + find_pacman_package | xargs rm -f + } +} + +find_pacman_package() +{ + find /var/cache/pacman/pkg/ -type f -name "*.pkg.tar*" +} + +pacman_download() +{ + echo "pacman -Sw --noconfirm $*" + pacman -Sw --noconfirm "$@" +} + +pacman_pack() +{ + local target_dir=/opt/pkgs + local date=$(date +"%Y%m%d") + + local downloaded_pkgs=$(find_pacman_package) + + [ "$downloaded_pkgs" ] || return + + mkdir -p $target_dir + + mv $downloaded_pkgs $target_dir + + find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz + + ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + echo package installed to $pack_to/${BM_NAME}.cgz + + ls $target_dir/*.pkg.tar* > $pack_to/.${BM_NAME}.packages + + rm -rf $target_dir +} + +distro_install_depends() +{ + local script + local bm="$1" + local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") + + for script in $scripts + do + script=$(basename $script) + packages=$(get_dependency_packages $DISTRO $script) + + [ -z "$packages" ] && continue + + echo install packages for $script: $packages + + for pkg in $packages + do + pacman -Qqs --noconfirm $pkg || pacman -Sy --noconfirm $pkg + done + done +} + +pack_benchmark() +{ + setup_proxy + + distro_install_depends lkp-dev + + # Process each benchmark + for BM_NAME in $benchmark + do + distro_install_depends $BM_NAME-dev || continue + + echo $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ( + $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ) + done +} -- 2.23.0

1 0

[PATCH v4 lkp-tests] distro/archlinux: add archlinux distro file
by Wang Yong 17 Nov '20

17 Nov '20

add archlinux pack method Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 111 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100755 distro/archlinux diff --git a/distro/archlinux b/distro/archlinux new file mode 100755 index 000000000..e8e950726 --- /dev/null +++ b/distro/archlinux @@ -0,0 +1,111 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +shopt -s nullglob + +. $LKP_SRC/lib/install.sh +. $LKP_SRC/distro/common +. $LKP_SRC/lib/debug.sh + +distro_install_depends() +{ + local script + local bm="$1" + local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") + + for script in $scripts + do + script=$(basename $script) + packages=$(get_dependency_packages $DISTRO $script) + + [ -z "$packages" ] && continue + + echo install packages for $script: $packages + + for pkg in $packages + do + pacman -Qqs --noconfirm $pkg || pacman -Sy --noconfirm $pkg + done + done +} + +pack_benchmark() +{ + setup_proxy + + distro_install_depends lkp-dev + + # Process each benchmark + for BM_NAME in $benchmark + do + distro_install_depends $BM_NAME-dev || continue + + echo $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ( + $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ) + done +} + +pacman_download() +{ + echo "pacman -Sw --noconfirm $*" + pacman -Sw --noconfirm "$@" +} + +pacman_pack() +{ + local target_dir=/opt/pkgs + local date=$(date +"%Y%m%d") + + local downloaded_pkgs=$(find_pacman_package) + + [ "$downloaded_pkgs" ] || return + + mkdir -p $target_dir + + mv $downloaded_pkgs $target_dir + + find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz + + ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + echo package installed to $pack_to/${BM_NAME}.cgz + + ls $target_dir/*.pkg.tar* > $pack_to/.${BM_NAME}.packages + + rm -rf $target_dir +} + +cleanup_downloaded_pkgs() +{ + [[ -d /var/cache/pacman/pkg/ ]] && { + find_pacman_package | xargs rm -f + } +} + +pack_benchmark_deps() +{ + setup_proxy + + for BM_NAME in $benchmark + do + + cleanup_downloaded_pkgs + + check_shared_package $BM_NAME + + packages=$(echo $(get_dependency_packages $DISTRO $BM_NAME)) + + if pacman_download $packages; then + pacman_pack + else + echo "failed to pack-deps $BM_NAME" >&2 + fi + done +} + +find_pacman_package() +{ + find /var/cache/pacman/pkg/ -type f -name "*.pkg.tar*" +} -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/2] lib: add multi-args in job yaml
by Bai Jing 17 Nov '20

17 Nov '20

[original style] os: openeuler os_version: 20.03 os_arch: aarch64 os: openeuler os_version: 20 os_arch: aarch64 [now style] os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 [point] Currently, only one key of this style(xx xxx xx) can be contained in YAML files. Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..f3b7a092 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -250,7 +250,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) - @jobs << hash + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e log_error "#{jobfile}: " + e.message @@ -280,6 +280,30 @@ class Job @jobfile = jobfile end + def multi_args(hash) + jobs_array = [] + hash.each { |key, value| + next unless key =~ /^\w.*\s+.*\w$/ + + next unless value.class == Array + + key_array = key.split + value.each do |v| + v_array = v.split + next unless key_array.size == v_array.size + + hash_job = hash.clone + key_array.size.times do |i| + hash_job[key_array[i]] = v_array[i] + end + hash_job.delete(key) + jobs_array << hash_job + end + } + jobs_array = [hash] if jobs_array.empty? + return jobs_array + end + def delete_keys_from_spec(spec_file) return unless File.exist? spec_file -- 2.23.0

2 3

[PATCH v2 lkp-tests 3/4] test/cci-makepkg: Refactor encapsulate the statement as a function
by Sun Yukui 17 Nov '20

17 Nov '20

[why] Encapsulating statements as functions is important for a better syntax. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..a87cf01f 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,11 @@ update_softlink() rm $sync_dest/$cgz_name } +build_source_pkg(){ + echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ + $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf +} + +build_source_pkg update_softlink && echo "Makepkg finished successfully" -- 2.23.0

2 2

[PATCH lkp-tests] update command parameters
by Hu Xuejiao 17 Nov '20

17 Nov '20

[why] The test date is incorrect and the parameters need to be adjusted Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git index df1bb5cf..be83b441 100755 --- a/tests/sysbench-threads-git +++ b/tests/sysbench-threads-git @@ -10,7 +10,7 @@ : "${thread_locks:=2}" args=( - --threads=$nr_threads + --num-threads=$nr_threads --thread-yields=$thread_yields --thread-locks=$thread_locks ) -- 2.23.0

2 1

[PATCH v2 lkp-tests 2/2] job: fix each_jobs to fit multi-args
by Bai Jing 17 Nov '20

17 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash = {"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03", "os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20", "os_arch"=>"aarch64"} h1 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} We need to merge the h2 to @job Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index f3b7a092..f8608e94 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -727,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH lkp-tests] job: fix each_jobs to fit multi-args
by Bai Jing 17 Nov '20

17 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash = {"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03", "os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20", "os_arch"=>"aarch64"} h1 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} We need to merge the h2 to @job Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index f3b7a092..f8608e94 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -727,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 1

[PATCH v2 lkp-tests] update command parameters
by Hu Xuejiao 17 Nov '20

17 Nov '20

[why] The current result: value of "total_time" is the same for different threads Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git index df1bb5cf..be83b441 100755 --- a/tests/sysbench-threads-git +++ b/tests/sysbench-threads-git @@ -10,7 +10,7 @@ : "${thread_locks:=2}" args=( - --threads=$nr_threads + --num-threads=$nr_threads --thread-yields=$thread_yields --thread-locks=$thread_locks ) -- 2.23.0

2 1

[PATCH v2 lkp-tests] pkg: add mysql
by Lin Jiaxin 17 Nov '20

17 Nov '20

sysbench-mysql need to test with mysql8.0.17 Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...00-mysql-add-fstack-protector-strong.patch | 20 ++++++ pkg/mysql/PKGBUILD | 50 ++++++++++++++ pkg/mysql/my.cnf | 67 +++++++++++++++++++ 3 files changed, 137 insertions(+) create mode 100644 pkg/mysql/0000-mysql-add-fstack-protector-strong.patch create mode 100644 pkg/mysql/PKGBUILD create mode 100644 pkg/mysql/my.cnf diff --git a/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch new file mode 100644 index 00000000..23c07c87 --- /dev/null +++ b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch @@ -0,0 +1,20 @@ +diff --git a/cmake/build_configurations/compiler_options.cmake b/cmake/build_configurations/compiler_options.cmake +index fecf2a2..f3e012f 100644 +--- a/cmake/build_configurations/compiler_options.cmake ++++ b/cmake/build_configurations/compiler_options.cmake +@@ -43,6 +43,7 @@ IF(UNIX) + # Default GCC flags + IF(CMAKE_COMPILER_IS_GNUCC) + SET(COMMON_C_FLAGS "-fno-omit-frame-pointer") ++ SET(COMMON_C_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_C_FLAGS "-fno-inline ") +@@ -57,6 +58,7 @@ IF(UNIX) + ENDIF() + IF(CMAKE_COMPILER_IS_GNUCXX) + SET(COMMON_CXX_FLAGS "-std=c++14 -fno-omit-frame-pointer") ++ SET(COMMON_CXX_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_CXX_FLAGS "-fno-inline ") diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD new file mode 100644 index 00000000..2fd32ead --- /dev/null +++ b/pkg/mysql/PKGBUILD @@ -0,0 +1,50 @@ +pkgname=mysql +pkgver=8.0.17 +pkgrel=3 +arch=('aarch64' 'x86_64') +url="http://www.mysql.com" +license=('GPLv2') +source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" + "0000-mysql-add-fstack-protector-strong.patch" + "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" + "my.cnf") +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' '7bb0372ec4e9281813a9afc67c6cfc20') + +prepare() +{ + cd $srcdir/$pkgname-$pkgver + patch -p1 < ../0000-mysql-add-fstack-protector-strong.patch +} + +build() +{ + cd $srcdir/$pkgname-$pkgver + + CMAKE=cmake + grep -sqF "CentOS Linux release 7" /etc/centos-release && CMAKE=cmake3 + + $CMAKE . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc \ + -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_PARTITION_STORAGE_ENGINE=1 -DWITH_FEDERATED_STORAGE_ENGINE=1 \ + -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_MYISAM_STORAGE_ENGINE=1 \ + -DENABLED_LOCAL_INFILE=1 -DENABLE_DTRACE=0 -DDEFAULT_CHARSET=utf8mb4 -DDEFAULT_COLLATION=utf8mb4_general_ci \ + -DWITH_EMBEDDED_SERVER=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=./boost -DFORCE_INSOURCE_BUILD=1 + make -j8 + + cd $srcdir + tar -zxvf getcpu.tgz + + cd getcpu || exit + make +} + +package() +{ + mkdir -p $pkgdir/getcpu + cp getcpu/getcpu-hi.ko $pkgdir/getcpu/ + + mkdir -p $pkgdir/etc + cp my.cnf $pkgdir/etc/ + + cd $srcdir/$pkgname-$pkgver + make DESTDIR=$pkgdir install +} diff --git a/pkg/mysql/my.cnf b/pkg/mysql/my.cnf new file mode 100644 index 00000000..9732af40 --- /dev/null +++ b/pkg/mysql/my.cnf @@ -0,0 +1,67 @@ +[mysqld_safe] +log-error=/data/mysql/log/mysql.log +pid-file=/data/mysql/run/mysqld.pid +[mysqldump] +quick +[mysql] +[client] +default-character-set=utf8mb4 +socket=/data/mysql/run/mysql.sock + +[mysqld] +socket=/data/mysql/run/mysql.sock +tmpdir=/data/mysql/tmp +datadir=/data/mysql/data +default_authentication_plugin=mysql_native_password +port=3309 +user=mysql +memlock=on +bind-address=0.0.0.0 +skip-grant-tables + +lc-messages=en_US +lc-messages-dir=/usr/local/mysql/share + +innodb_page_size=64k +max_connections=4000 +back_log=2048 +performance_schema=OFF +max_prepared_stmt_count=128000 + +#file +innodb_file_per_table +innodb_log_file_size=2048M +innodb_log_files_in_group=32 +innodb_open_files=10000 +table_open_cache_instances=64 + +#buffers +innodb_buffer_pool_size=128G +innodb_buffer_pool_instances=64 +innodb_log_buffer_size=2048M + +#tune +sync_binlog=0 +innodb_flush_log_at_trx_commit=0 +innodb_use_native_aio=1 +innodb_spin_wait_delay=160 +innodb_sync_spin_loops=20 +innodb_flush_method=O_DIRECT +innodb_io_capacity=30000 +innodb_io_capacity_max=40000 +innodb_lru_scan_depth=9000 +innodb_page_cleaners=1 +innodb_spin_wait_pause_multiplier=25 + +#perf special +innodb_flush_neighbors=0 +innodb_write_io_threads=1 +innodb_read_io_threads=1 +innodb_purge_threads=1 + +sql_mode=STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_AUTO_VALUE_ON_ZERO,STRICT_ALL_TABLES + +#skop_log_bin +ssl=0 + +table_open_cache=15000 -- 2.23.0

2 1

[PATCH v6 compass-ci] add the container/initrd-cifs to share /srv/initrd
by Xiao Shenwei 17 Nov '20

17 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result in order to reduce the coupling, need split them [how] one folder mount by one container [usage] Two ways for the client to mount initrd: 1. initrd mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 24 ++++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 91 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..9590701 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,24 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN adduser -u 1090 -D lkp + +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash && \ + rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..4b93647 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..77558a1 --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

4 9

[PATCH v4 lkp-tests 1/2] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 17 Nov '20

17 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git from /srv/git/pkg/ usage: job yaml add new field: PKGBUILD_TAG About $PKGBUILD_TAG, it is from PKGBUILD file's $pkgver-$pkgrel. example: PKGBUILD_TAG: 1.0-20 You should create a git repo that name same as $benchmark in /srv/git/pkg/ before you add PKGBUILD_TAG to yaml. If not, cci-makepkg will report an error. And everything same as before if no PKGBUILD_TAG. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..a62b5be0 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -39,12 +39,29 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +download_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +get_pkgfile() +{ + local pkg_tag_file="PKGBUILD-$PKGBUILD_TAG" + + download_pkgfile "pkg/$benchmark" "$pkg_tag_file" + + [ -f "$pkg_tag_file" ] || die "$pkg_tag_file not exist" + [ -s "$pkg_tag_file" ] || die "$pkg_tag_file is empty" +} + get_pkg_info() { var=$1 pkg_dir=${2:-.} ( . $pkg_dir/PKGBUILD + [ -n "$PKGBUILD_TAG" ] && . $pkg_dir/PKGBUILD-$PKGBUILD_TAG eval echo '$'$var ) } @@ -71,6 +88,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && get_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

2 1

[PATCH v3 compass-ci 2/4] assign-account: add new option for ssh key
by Luan Shengde 17 Nov '20

17 Nov '20

add -r|--gen-sshkey option for answerback-email [why] enable one click login testbox from jumper server [how] add -r|--rsa-key option to enable generate private/public key on jumper server register the public key to es Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 77 ++++++++++---------- 1 file changed, 38 insertions(+), 39 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 686a327..39091c7 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -7,12 +7,23 @@ # samba mount # ssh logshn (huawei, ) (install pubkey / send password) +# 1) external user: email apply => mail robot +# 2) ./answerback-email: huawei user: cli apply +# => params +# => REST api (params different) +# => robot email notify + +# user_type: +# apply_from: mail/cli +# allow_login: + require 'json' require 'mail' require 'set' require 'optparse' require_relative '../defconfig' require_relative '../../lib/es_client' +require_relative 'send-account-email' names = Set.new %w[ JUMPER_HOST @@ -31,17 +42,18 @@ SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 my_info = { 'my_email' => nil, 'my_name' => nil, + 'my_commit_url' => nil, 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => nil + 'my_ssh_pubkey' => [], + 'gen_sshkey' => false } -def init_info(email_file, my_info) - mail_content = Mail.read(email_file) +def init_info(mail_content, my_info) my_info['my_email'] = mail_content.from[0] my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') - my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded - end + my_info['my_ssh_pubkey'] << if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end end options = OptionParser.new do |opts| @@ -55,17 +67,23 @@ options = OptionParser.new do |opts| opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| my_info['my_email'] = email_address - # when apply account with email address, will get no user name + # when apply account with email address, will get no user name and commit url my_info['my_name'] = '' + my_info['my_commit_url'] = '' end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| - my_info['my_ssh_pubkey'] = File.read(pub_key_file) + my_info['my_ssh_pubkey'] << File.read(pub_key_file).chomp end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| - init_info(email_file, my_info) + mail_content = Mail.read(email_file) + init_info(mail_content, my_info) + end + + opts.on('-g', '--gen-sshkey', 'generate jumper ras public/private key and return pubkey') do + my_info['gen_sshkey'] = true end opts.on_tail('-h', '--help', 'show this message') do @@ -76,32 +94,6 @@ end options.parse!(ARGV) -def build_message(email, account_info) - message = <<~EMAIL_MESSAGE - To: #{email} - Subject: [compass-ci] jumper account is ready - - Dear user: - - Thank you for joining us. - You can use the following command to login the jumper server: - - Login command: - ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} - - Account password: - #{account_info['my_password']} - - Suggest: - If you use the password to login, change it in time. - - regards - compass-ci - EMAIL_MESSAGE - - return message -end - def apply_account(my_info) account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str @@ -114,12 +106,19 @@ def send_account(my_info) raise message if my_info['my_email'].nil? account_info = apply_account(my_info) - # for manually assign account, there will be no my_commit_url - # but the key my_commit_url is required for es - my_info['my_commit_url'] = '' my_info['my_login_name'] = account_info['my_login_name'] - my_info.delete 'my_ssh_pubkey' + + unless account_info['my_jumper_pubkey'].nil? + my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'].chomp + end + + my_info.delete 'gen_sshkey' store_account_info(my_info) + + send_mail(my_info, account_info) +end + +def send_mail(my_info, account_info) message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") -- 2.23.0

2 12

[PATCH v8 lkp-tests 2/3] submit on_fail to the scheduler
by Hu Xuejiao 17 Nov '20

17 Nov '20

[why] When the task is failed(state), it need in the waiting state [how] When job2sh.rb defined sh_on_state, it will be invoked by the scheduler Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..a325e912 100755 --- a/sbin/submit +++ b/sbin/submit @@ -115,12 +115,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_state(state='on_fail') sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

1 0

[PATCH v8 lkp-tests 1/3] job2sh.rb: define on_state
by Hu Xuejiao 17 Nov '20

17 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By on_fail(sh_on_state), it will appear in job.sh: on_fail() { . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh run_test $LKP_SRC/tests/wrapper sleep 300 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c8b578d..9d0a7bb2 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -363,14 +363,14 @@ class Job2sh < Job out_line "}\n\n" end - def sh_on_state(job = nil) + def sh_on_state(state = nil, job = nil) @script_lines = [] @programs = available_programs(:workload_elements) job ||= (@jobx || @job).clone - if job.include?('on_fail') + if state == "on_fail" && job.include?('on_fail') @cur_func = :run_job -- 2.23.0

1 0

[PATCH lkp-tests] depends/sysbench-mysql: add depends
by Zhang Yu 17 Nov '20

17 Nov '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/sysbench-mysql | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/sysbench-mysql b/distro/depends/sysbench-mysql index a248fd07..b0d33ffe 100644 --- a/distro/depends/sysbench-mysql +++ b/distro/depends/sysbench-mysql @@ -1,3 +1,4 @@ libnuma1 xfsprogs ethtool +libmariadb3 -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/4] test/build-pkg: fix a softlink will be established if the package fails
by Sun Yukui 17 Nov '20

17 Nov '20

After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 6426f2da..8d5fc7d4 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -101,10 +101,11 @@ build_source_pkg() cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 + + [ "$?" == 0 ] || create_softlink } check_vars mount_dest request_pkg build_source_pkg -create_softlink -- 2.23.0

3 3

[PATCH v2 lkp-tests 2/3] iso2rootfs: add step: config something of rootfs
by Yu Chuan 17 Nov '20

17 Nov '20

[Why] The initial rootfs have the follow two problems: - no dns resolver; - selinux is opened; Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 2e7bcd47ef4f..87a51acaca5f 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -338,6 +338,27 @@ run_qcow2rootfs() ./run "${qcow2_path}" "${ROOTFS_DES_DIR}" } + +############ config rootfs ############ +config_dns_resolver() +{ + cat <<-EOF > "${ROOTFS_DES_DIR}/etc/resolv.conf" + nameserver 114.114.114.114 + nameserver 8.8.8.8 + EOF +} + +disable_selinux() +{ + sed -i 's/^SELINUX=.*/SELINUX=disabled/g' "${ROOTFS_DES_DIR}/etc/selinux/config" +} + +config_rootfs() +{ + config_dns_resolver + disable_selinux +} + ############ test rootfs ############ get_qemu_efi_fd() { @@ -497,6 +518,8 @@ main() config_qcow2rootfs run_qcow2rootfs + config_rootfs + test_rootfs [ -z ${test_yaml} ] || submit_test_job_yaml -- 2.23.0

3 5

[PATCH v2 lkp-tests 1/3] iso2rootfs: support not submit test job
by Yu Chuan 17 Nov '20

17 Nov '20

[Why] If user only want generate a rootfs without testing, he can not write submit related fields in the job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 2 ++ tests/iso2rootfs | 22 ++++++++++++++-------- 2 files changed, 16 insertions(+), 8 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 644c64be06b4..d4f0ffc0e80c 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -11,6 +11,8 @@ iso2rootfs: rootfs_path: initrd_http_host: initrd_http_port: + + # submit test yaml related fields test_yaml: test_os: test_os_arch: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index e8a027b0dad2..2e7bcd47ef4f 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -53,15 +53,8 @@ check_yaml_vars() "rootfs_path" "initrd_http_host" "initrd_http_port" - "test_yaml" - "test_os" - "test_os_arch" - "test_os_version" - "test_os_mount" - "test_testbox" ) - local yaml_t for yaml_t in "${yaml_vars[@]}" do [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." @@ -443,6 +436,19 @@ submit_test_job_yaml() [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + local test_yaml_vars=( + "test_os" + "test_os_arch" + "test_os_version" + "test_os_mount" + "test_testbox" + ) + + for yaml_t in "${test_yaml_vars[@]}" + do + [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." + done + cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh ./sbin/install-dependencies.sh @@ -492,7 +498,7 @@ main() run_qcow2rootfs test_rootfs - submit_test_job_yaml + [ -z ${test_yaml} ] || submit_test_job_yaml post_works } -- 2.23.0

3 4

[PATCH lkp-tests] stats/libmicro: fix rubocop warn
by Lu Kaiyi 17 Nov '20

17 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/libmicro | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/stats/libmicro b/stats/libmicro index 530d8e2e..b7e895bf 100755 --- a/stats/libmicro +++ b/stats/libmicro @@ -4,6 +4,7 @@ # close_tmp 1 1 0.61880 201 0 640 $stdin.each_line do |line| - next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ - puts "#{$1}: #{$2}" + next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ + + puts "#{$1}: #{$2}" end -- 2.23.0

1 0

[PATCH v2 lkp-tests] distro/archlinux: add archlinux distro file
by Wang Yong 17 Nov '20

17 Nov '20

add archlinux pack method Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 104 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100755 distro/archlinux diff --git a/distro/archlinux b/distro/archlinux new file mode 100755 index 000000000..e6d7dce50 --- /dev/null +++ b/distro/archlinux @@ -0,0 +1,104 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +shopt -s nullglob + +. $LKP_SRC/lib/install.sh +. $LKP_SRC/distro/common +. $LKP_SRC/lib/debug.sh + +distro_install_depends() +{ + local script + local bm="$1" + local scripts=$(find $LKP_SRC/distro/depends/ -name "$bm" -o -name "${bm}.[0-9]") + + for script in $scripts + do + script=$(basename $script) + packages=$(get_dependency_packages $DISTRO $script) + + [ -z "$packages" ] && continue + + echo install packages for $script: $packages + + for pkg in $packages + do + pacman -Qqs ----noconfirm $pkg || pacman -Sy --noconfirm $pkg + done + done +} + +pack_benchmark() +{ + setup_proxy + + distro_install_depends lkp-dev + + # Process each benchmark + for BM_NAME in $benchmark + do + distro_install_depends $BM_NAME-dev || continue + echo $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ( + $LKP_SRC/sbin/pack -d $DISTRO -f -c -s $PKG_MNT/$pack_to $BM_NAME + ) + done +} + +pacman_download() +{ + echo "pacman -Sw --noconfirm $*" + pacman -Sw --noconfirm "$@" +} + +pacman_pack() +{ + local target_dir=/opt/pkgs + local date=$(date +"%Y%m%d") + + local downloaded_pkgs=$(find /var/ -type f -name "*.pkg.tar.xz") + + [ "$downloaded_pkgs" ] || return + + mkdir -p $target_dir + + mv $downloaded_pkgs $target_dir + + find $target_dir | cpio --quiet -o -H newc --owner=root.root | gzip -n -9 >$pack_to/${BM_NAME}_$date.cgz + + ln -sf ${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + chown .lkp $pack_to/${BM_NAME}_$date.cgz $pack_to/${BM_NAME}.cgz || return + echo package installed to $pack_to/${BM_NAME}.cgz + + ls $target_dir/*.pkg.tar.gz > $pack_to/.${BM_NAME}.packages + + rm -rf $target_dir +} + +cleanup_downloaded_pkgs() +{ + find /var/ -type f -name "*.pkg.tar.xz" -exec rm -f -- '{}' \; +} + +pack_benchmark_deps() +{ + setup_proxy + + for BM_NAME in $benchmark + do + + cleanup_downloaded_pkgs + + check_shared_package $BM_NAME + + packages=$(echo $(get_dependency_packages $DISTRO $BM_NAME)) + + if pacman_download $packages; then + pacman_pack + else + echo "failed to pack-deps $BM_NAME" >&2 + echo $? + fi + done +} -- 2.23.0

3 8

[PATCH v3 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 17 Nov '20

17 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..fdc51b58 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby + +#LKP_SRC ||= ENV['LKP_SRC'] + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +$stdin.each_line do |line| + next unless line =~ /total time:\s+([0-9.]+)/ + + results_total['total_time'] = $1.to_f +end + +results_total.each do |key, vals| + puts "#{key}: #{vals}s" +end + -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests: adapt to the sysbench test of a specified version
by Zhang Yu 17 Nov '20

17 Nov '20

[why] use the special version of sysbench test mysql required by the customer. so need to use different sysbench script. [how] add parameters to test, and use different sysbench script. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 88 ++++++++++++++++++++++++++++++++------------ 1 file changed, 65 insertions(+), 23 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 4ab5b453..106ee0b9 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,52 +1,94 @@ #!/bin/sh -# mysql_user -# mysql_host -# mysql_port -# mysql_db -# db_driver # - oltp_test_mode # - oltp_tables_count # - oltp_table_size +# - max_requests +# - mysql_table_engine # - nr_threads +# - rand_type +# - rand_spec_pct # - runtime # - report_interval +# - oltp_read_only +# - events +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +# db_driver : "${mysql_user:=root}" -: "${mysql_host:=localhost}" -: "${mysql_port:=3306}" -: "${mysql_db:=test_1000}" +: "${mysql_host:=$direct_server_ips}" +: "${mysql_port:=$mysql_port}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=$mysql_password}" : "${db_driver:=mysql}" -: "${oltp_tables_count:=3}" -: "${oltp_table_size:=1000}" -: "${nr_threads:=64}" -: "${runtime:=120}" -: "${report_interval:=10}" +: "${oltp_test_mode:=complex}" +: "${oltp_tables_count:=1000}" +: "${oltp_table_size:=100000}" +: "${max_requests:=0}" +: "${mysql_table_engine:=innodb}" +: "${rand_type:=special}" +: "${rand_spec_pct:=100}" +: "${nr_threads:=256}" +: "${runtime:=7200}" +: "${report_interval:=1}" +: "${oltp_read_only:=off}" +: "${events:=2000000000}" -args=( +args1=( --mysql-user=$mysql_user --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db + --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver - --tables=$oltp_tables_count - --table-size=$oltp_table_size + --mysql-password=$mysql_password + --max-requests=$max_requests + --mysql-table-engine=$mysql_table_engine + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --rand-type=$rand_type + --rand-spec-pct=$rand_spec_pct --threads=$nr_threads --time=$runtime - --report-interval=$report_interval ) +args2=( + --mysql-user=$mysql_user + --mysql-password=$mysql_password + --mysql-host=$mysql_host + --mysql-port=$mysql_port + --mysql-db=$mysql_db + --threads=$nr_threads + --oltp-read-only=$oltp_read_only + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --report-interval=$report_interval + --time=$runtime + --events=$events +) + +stop_firewalld() +{ + systemctl stop firewalld + iptables -F +} + run_sysbench_step() { - sysbench /usr/share/sysbench/$1 "${args[@]}" $2 + lua_script=$1 + shift + sysbench /usr/local/share/sysbench/tests/include/oltp_legacy/$lua_script "$@" } run_sysbench_mysql() { - systemctl start mysqld - mysql -e "create database test_1000;" - run_sysbench_step oltp_common.lua prepare - run_sysbench_step oltp_read_write.lua run - run_sysbench_step oltp_common.lua cleanup + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + run_sysbench_step oltp.lua ${args2[@]} run + run_sysbench_step oltp.lua ${args2[@]} cleanup } +stop_firewalld run_sysbench_mysql -- 2.23.0

2 2

[PATCH v3 compass-ci] remove actual ip in compass-ci code
by Xiao Shenwei 17 Nov '20

17 Nov '20

obtain the actual ip address from configuration file Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- doc/develop-environment.md | 28 ---------------------------- src/features/jobs/right_iperf.yaml | 1 - src/scheduler/README.md | 2 +- user-client/helper/iperf.yaml | 1 - 4 files changed, 1 insertion(+), 31 deletions(-) diff --git a/doc/develop-environment.md b/doc/develop-environment.md index 8760878..5fda691 100644 --- a/doc/develop-environment.md +++ b/doc/develop-environment.md @@ -1,31 +1,3 @@ -# ssh - -## setup in your local laptop - - cat >> ~/.ssh/config <<-EOF - Host crystal - Hostname 124.90.34.227 - Port 22113 - EOF - - # password-less login - [ -f ~/.ssh/id_rsa.pub ] || ssh-keygen - ssh-copy-id crystal - -## setup in crystal server - - cat >> ~/.ssh/config <<-EOF - Host alpine - Hostname localhost - Port 2200 - User team - - Host debian - Hostname localhost - Port 2201 - User team - EOF - # git ## user setting diff --git a/src/features/jobs/right_iperf.yaml b/src/features/jobs/right_iperf.yaml index f81af38..c28b569 100644 --- a/src/features/jobs/right_iperf.yaml +++ b/src/features/jobs/right_iperf.yaml @@ -30,7 +30,6 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x diff --git a/src/scheduler/README.md b/src/scheduler/README.md index fdc7027..8e3a197 100644 --- a/src/scheduler/README.md +++ b/src/scheduler/README.md @@ -6,7 +6,7 @@ - And set the environment variables needed for running (modify "/etc/compass-ci/defaults/crystal.yaml", and run $CCI_SRC/container/defconfig.rb | defconfig.sh) LKP_SRC: ~/code/lkp-tests # path to your lkp-tests repo CCI_SRC: ~/code/compass-ci # path to your compass-ci repo - SCHED_HOST: 172.168.131.131 # hostname or ip of the scheduler + SCHED_HOST: 172.17.0.1 # hostname or ip of the scheduler SCHED_PORT: 3000 # port of the scheduler JOB_INDEX_TYPE: "jobs/_doc" # es job document index and type - '$'{variable} is used at shell command, means the value of variable diff --git a/user-client/helper/iperf.yaml b/user-client/helper/iperf.yaml index 1509470..50c9ac6 100644 --- a/user-client/helper/iperf.yaml +++ b/user-client/helper/iperf.yaml @@ -30,7 +30,6 @@ proc-vmstat: proc-stat: meminfo: -LKP_SERVER: 172.168.131.113 LKP_CGI_PORT: 3000 result_root: /result/iperf LKP_DEBUG_PREFIX: bash -x -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/build-pkg: add download PKGBUILD-$tag function
by Liu Shaofei 17 Nov '20

17 Nov '20

[why] Support more version's PKGBUILD to build different version software. [how] Download the PKGBUILD-$tag file through remote-git server from "/srv/git/pkg/" usage: 1) append upstream_tag field from build-pkg.yaml 2) create git repo that named by software name of $upstream_repo in "/srv/git/pkg/", and add PKGBUILD-$tag file to the directory of "/srv/git/pkg/$upstream_repo". if not PKGBUILD-$tag, the software is built with PKGBUILD. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 1eed784f..9fd03bab 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -7,7 +7,7 @@ # - upstream_repo # - upstream_url # - upstream_commit -# - upstream_tag +# - PKGBUILD_TAG ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -61,6 +61,15 @@ request_pkg() get_pkgfile "${pkgrepo}" "$pkgfile" done + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" + } + [ -f PKGBUILD ] || die "PKGBUILD not exist" [ -s PKGBUILD ] || die "PKGBUILD is empty" } @@ -81,6 +90,9 @@ build_source_pkg() upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\")" sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD + [ -n "$PKGBUILD_TAG" ] && { + sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" + } } cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" @@ -95,9 +107,9 @@ create_softlink() local soft_path="$PKG_MNT/${pack_to}/${pkgname}" local bm_name=$(basename $(realpath ${cgz_name})) - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + if [ -n "$PKGBUILD_TAG" ]; then + echo "create soft link: ${PKGBUILD_TAG}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${PKGBUILD_TAG}.cgz" fi echo "create soft link: latest.cgz -> ${bm_name}" -- 2.23.0

1 0

[PATCH v2 lkp-tests] [why] Support more version's PKGBUILD to build different version software.
by Liu Shaofei 17 Nov '20

17 Nov '20

[how] Download the PKGBUILD-$tag file through remote-git server from "/srv/git/pkg/" usage: 1) append upstream_tag field from build-pkg.yaml 2) create git repo that named by software name of $upstream_repo in "/srv/git/pkg/", and add PKGBUILD-$tag file to the directory of "/srv/git/pkg/$upstream_repo". if not PKGBUILD-$tag, the software is built with PKGBUILD. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 1eed784f..9fd03bab 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -7,7 +7,7 @@ # - upstream_repo # - upstream_url # - upstream_commit -# - upstream_tag +# - PKGBUILD_TAG ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -61,6 +61,15 @@ request_pkg() get_pkgfile "${pkgrepo}" "$pkgfile" done + # download PKGBUILD-$tag + benchmark="${upstream_repo##*/}" + benchmark="${benchmark%.git}" + [ -n "$PKGBUILD_TAG" ] && { + get_pkgfile "pkg/$benchmark" "PKGBUILD-$PKGBUILD_TAG" + [ -f "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG not exist" + [ -s "PKGBUILD-$PKGBUILD_TAG" ] || die "PKGBUILD-$PKGBUILD_TAG is empty" + } + [ -f PKGBUILD ] || die "PKGBUILD not exist" [ -s PKGBUILD ] || die "PKGBUILD is empty" } @@ -81,6 +90,9 @@ build_source_pkg() upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\")" sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD + [ -n "$PKGBUILD_TAG" ] && { + sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" + } } cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" @@ -95,9 +107,9 @@ create_softlink() local soft_path="$PKG_MNT/${pack_to}/${pkgname}" local bm_name=$(basename $(realpath ${cgz_name})) - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + if [ -n "$PKGBUILD_TAG" ]; then + echo "create soft link: ${PKGBUILD_TAG}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${PKGBUILD_TAG}.cgz" fi echo "create soft link: latest.cgz -> ${bm_name}" -- 2.23.0

1 1

[PATCH v2 lkp-tests 4/4] test/cci-makepkg: fix a softlink will be established if the package fails
by Sun Yukui 17 Nov '20

17 Nov '20

After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a87cf01f..37c5dc08 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -109,7 +109,8 @@ build_source_pkg(){ echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + + [ "$?" == 0 ] || update_softlink } build_source_pkg -update_softlink && echo "Makepkg finished successfully" -- 2.23.0

1 0

[PATCH v2 lkp-tests 8/8] tests/cci-makepkg: update gcc for centos7 when use initramfs os
by Lin Jiaxin 17 Nov '20

17 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 67d055a4..bdb2128d 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -88,13 +88,22 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +centos_update_gcc() +{ + if $(grep -sqF "CentOS Linux release 7" /etc/centos-release ); then + [[ "$os_mount" = "initramfs" ]] && { + yum install -y centos-release-scl-rh + yum install -y "devtoolset-8-gcc*" + } + source /opt/rh/devtoolset-8/enable + fi +} + [ -n "$PKGBUILD_TAG" ] && get_pkgfile distro_install_depends lkp-dev -[ $os = "centos" ] && -grep -sqF "CentOS Linux release 7" /etc/centos-release && -source /opt/rh/devtoolset-8/enable +[ $os = "centos" ] && centos_update_gcc distro_install_depends ${benchmark}-dev -- 2.23.0

1 1

[PATCH v2 lkp-tests 6/8] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 17 Nov '20

17 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git from /srv/git/pkg/ usage: job yaml: PKGBUILD_TAG: 1.0-20 About $PKGBUILD_TAG, it is from PKGBUILD file's $pkgver-$pkgrel. You should create a git repo that name same as $benchmark in /srv/git/pkg/ before you add PKGBUILD_TAG to yaml. If not, cci-makepkg will report an error. And everything same as before if no PKGBUILD_TAG. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..67d055a4 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -39,12 +39,29 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +download_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +get_pkgfile() +{ + PKGBUILD_TAG_FILE="PKGBUILD-$PKGBUILD_TAG" + + download_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 pkg_dir=${2:-.} ( . $pkg_dir/PKGBUILD + [ -n "$PKGBUILD_TAG" ] && . $pkg_dir/PKGBUILD-$PKGBUILD_TAG eval echo '$'$var ) } @@ -71,6 +88,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && get_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

1 1

[PATCH v2 lkp-tests 5/8] stats: add libmicro result parsing file
by Lin Jiaxin 17 Nov '20

17 Nov '20

From: Bai Jing <799286817(a)qq.com> [input] prc thr usecs/call samples errors cnt/samp getpid 1 1 0.15629 193 0 100000 [output] getpid: 0.15629 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/libmicro | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100755 stats/libmicro diff --git a/stats/libmicro b/stats/libmicro new file mode 100755 index 00000000..530d8e2e --- /dev/null +++ b/stats/libmicro @@ -0,0 +1,9 @@ +#!/usr/bin/env ruby + +# prc thr usecs/call samples errors cnt/samp +# close_tmp 1 1 0.61880 201 0 640 + +$stdin.each_line do |line| + next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ + puts "#{$1}: #{$2}" +end -- 2.23.0

1 1

[PATCH v2 lkp-tests 4/8] job2sh.rb: define on_state
by Lin Jiaxin 17 Nov '20

17 Nov '20

From: Hu Xuejiao <1034502035(a)qq.com> [why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By on_fail(sh_on_state), it will appear in job.sh: on_fail() { . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh run_test $LKP_SRC/tests/wrapper sleep 300 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7e6a5d7c..9c8b578d 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_state(job) out_line '"$@"' @@ -358,6 +360,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_state(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + if job.include?('on_fail') + + @cur_func = :run_job + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + parse_hash [], job['on_fail'] + out_line "}\n\n" + end end end -- 2.23.0

1 1

[PATCH v2 lkp-tests 3/8] sysbench-mysql/PKGBUILD: fix "No such file or directory"
by Lin Jiaxin 17 Nov '20

17 Nov '20

From: Zhang Yu <2134782174(a)qq.com> [why] only openeuler have dir "/usr/local/mysql/lib/" Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- pkg/sysbench-mysql/PKGBUILD | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/pkg/sysbench-mysql/PKGBUILD b/pkg/sysbench-mysql/PKGBUILD index d9f342a5..c214d8a2 100644 --- a/pkg/sysbench-mysql/PKGBUILD +++ b/pkg/sysbench-mysql/PKGBUILD @@ -18,6 +18,8 @@ build() { package() { cd "$pkgname-$pkgver.$pkgrel" make DESTDIR=$pkgdir install - [[ $os == openeuler ]] && mkdir -p $pkgdir/usr/lib64 - cp -rpf /usr/local/mysql/lib/* $pkgdir/usr/lib64/ + [[ $os == openeuler ]] && { + mkdir -p $pkgdir/usr/lib64 + cp -rpf /usr/local/mysql/lib/* $pkgdir/usr/lib64/ + } } -- 2.23.0

1 1

[PATCH v2 lkp-tests 2/8] depends/sysbench-mysql: add depends
by Lin Jiaxin 17 Nov '20

17 Nov '20

From: Zhang Yu <2134782174(a)qq.com> [why] when run with initramfs, there occurred an error: /lkp/lkp/src/daemon/mysql: line 57: ethtool: command not found Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/sysbench-mysql | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/sysbench-mysql b/distro/depends/sysbench-mysql index cfbc4518..a248fd07 100644 --- a/distro/depends/sysbench-mysql +++ b/distro/depends/sysbench-mysql @@ -1,2 +1,3 @@ libnuma1 xfsprogs +ethtool -- 2.23.0

1 1

[PATCH v2 lkp-tests 1/8] sbin/makepkg: download submodules code from LKP server
by Lin Jiaxin 17 Nov '20

17 Nov '20

From: Liu Shaofei <370072077(a)qq.com> Some repositories have submodules. So when build packages, we need download submodules from LKP server, avoid the waste of time in downloading files from the remote end. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index fde82212..6f8b95af 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -623,6 +623,11 @@ extract_git() { cd_safe "${dir##*/}" + # modify submodules address to local server address. + if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then + echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig + fi + local ref=origin/HEAD if [[ -n $fragment ]]; then case ${fragment%%=*} in -- 2.23.0

1 1

[PATCH v7 compass-ci 01/11] mail-robot: listen and handle new email
by Luan Shengde 17 Nov '20

17 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/mail-robot.rb | 41 ++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..a809eae --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,41 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'lib/apply-account' + +MAILDIR = ENV['MAILDIR'] + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = ApplyAccount.new(mail_content) + assign_uuid.check_to_send_account +end + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

2 4