November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v4 lkp-tests] netperf/PKGBUILD: download new config.guess, config.sub if guess failed
by Lin Jiaxin 13 Nov '20

13 Nov '20

error: ./config.guess: unable to guess system type This script, last modified 2005-08-03, has failed to recognize the operating system you are using. It is advised that you download the most up to date version of the config scripts. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- pkg/netperf/PKGBUILD | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/pkg/netperf/PKGBUILD b/pkg/netperf/PKGBUILD index 1b724948..e4747dd5 100755 --- a/pkg/netperf/PKGBUILD +++ b/pkg/netperf/PKGBUILD @@ -10,6 +10,13 @@ md5sums=('e0d45b5bca1eee2aef0155de82366202') build() { cd "$srcdir/$pkgname-$pkgname-$pkgver.$pkgrel" + + # the orginal config.guess is too old to guess new system type + # so download latest version if it guess failed + ./config.guess > /dev/null 2>&1 || { + curl -s -o config.guess 'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.gues…' + curl -s -o config.sub 'http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;…' + } # sendfile_tcp_stream() misses assignment of len and it is possible # for random len to trigger the following issue, so fix it. -- 2.23.0

1 0

[PATCH lkp-tests] stats: add libmicro result parsing file
by Bai Jing 13 Nov '20

13 Nov '20

[input] prc thr usecs/call samples errors cnt/samp getpid 1 1 0.15629 193 0 100000 [output] getpid: 0.15629 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/libmicro | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 stats/libmicro diff --git a/stats/libmicro b/stats/libmicro new file mode 100755 index 00000000..2e33ffc9 --- /dev/null +++ b/stats/libmicro @@ -0,0 +1,21 @@ +#!/usr/bin/env ruby + +# prc thr usecs/call samples errors cnt/samp +# close_tmp 1 1 0.61880 201 0 640 + +def line_fun(line) + hash = {} + test = line.split[0] + usecs_call = line.split[3] + hash[test] = usecs_call + hash.each do |key, value| + puts "#{key}: #{value}" + end +end + +$stdin.each_line do |line| + case line + when /^\w+\s+\d/ + line_fun(line) + end +end -- 2.23.0

1 1

[PATCH v4 compass-ci] container/srv-http: support display *.log file content in browser
by Lu Weitao 13 Nov '20

13 Nov '20

[Why] Haven't define some file(*.log) behavior of browser in mime.types, so download this file on browser, but we expect display the file content [How] 1. Add correspondence of file in mime.types 2. Set default type in default.conf of nginx, other type file that haven't define in mime.types, will display the file content in browser Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 1 + container/srv-http/root/etc/nginx/mime.types | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..91371f1 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -13,6 +13,7 @@ server { } location / { + default_type text/html; autoindex on; autoindex_localtime on; allow all; diff --git a/container/srv-http/root/etc/nginx/mime.types b/container/srv-http/root/etc/nginx/mime.types index e7f2f1c..a4ee74b 100644 --- a/container/srv-http/root/etc/nginx/mime.types +++ b/container/srv-http/root/etc/nginx/mime.types @@ -4,8 +4,8 @@ types { text/css css; text/xml xml; text/yaml yaml; - text/plain txt time data sh; + text/plain txt time data sh log; application/json json; - application/x-gzip gz tgz; + application/x-gzip gz tgz cgz; } -- 2.23.0

1 0

[PATCH v7 lkp-tests] stats: add hackbench-git result parsing script
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..aafe6dde --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,23 @@ +#!/usr/bin/env ruby + +# --------socket thread num=20-------- +# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 +# Running with 20*40 (== 800) tasks. +# Time: 0.495 +# ... + +time = [] + +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/3] iso2rootfs: allow generate a rootfs without testing it
by Yu Chuan 13 Nov '20

13 Nov '20

[Why] If user only want generate a rootfs without testing it, he can not write submit related fields in the job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 2 ++ tests/iso2rootfs | 22 +++++++++++++++------- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 644c64be06b4..d4f0ffc0e80c 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -11,6 +11,8 @@ iso2rootfs: rootfs_path: initrd_http_host: initrd_http_port: + + # submit test yaml related fields test_yaml: test_os: test_os_arch: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index e8a027b0dad2..4f7843d7f348 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -53,15 +53,8 @@ check_yaml_vars() "rootfs_path" "initrd_http_host" "initrd_http_port" - "test_yaml" - "test_os" - "test_os_arch" - "test_os_version" - "test_os_mount" - "test_testbox" ) - local yaml_t for yaml_t in "${yaml_vars[@]}" do [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." @@ -439,10 +432,25 @@ generate_submit_append_str() submit_test_job_yaml() { + [ -n ${test_yaml} ] || return 0 + log_info "starting submit ${test_yaml} ..." [ -d "${LKP_SRC}" ] || die "cannot find value of var: LKP_SRC." + local test_yaml_vars=( + "test_os" + "test_os_arch" + "test_os_version" + "test_os_mount" + "test_testbox" + ) + + for yaml_t in "${test_yaml_vars[@]}" + do + [ -n "$(eval echo "\$${yaml_t}")" ] || die "cannot fine value of var: ${yaml_t}." + done + cd "${LKP_SRC}" && { chmod +x ./sbin/install-dependencies.sh ./sbin/install-dependencies.sh -- 2.23.0

1 0

[PATCH v6 compass-ci 09/12] mail-robot: documentation for service
by Luan Shengde 13 Nov '20

13 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..2eb0dd1 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. assign-account + init my_info + - my_email + - my_name + send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

2 4

[PATCH v5 compass-ci] add the container/initrd-cifs to share /srv/initrd
by Xiao Shenwei 13 Nov '20

13 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result it may cause: 1. our cifs service becomes slow 2. if the container exit, none of the three folders can be accessed. in order to reduce the coupling, need split them [how] one folder mount by one container [Usage]: Two ways for the client to mount initrd: 1. initrd. mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage. mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 22 ++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 89 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..341bfc9 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +RUN adduser -u 1090 -D lkp +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash && \ + rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..ca9de39 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t alpine/initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..7232bdf --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + alpine/initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

2 2

[PATCH v2 lkp-tests 3/3] iso2rootfs: solve the problem of multiple vmlinuz failures during test
by Yu Chuan 13 Nov '20

13 Nov '20

[Why] We use compass-ci/container/qcow2rootfs to generate rootfs from qcow2, but compass-ci updated the code, qcow2rootfs create a vmlinuz soft link at {rootfs_dir}/boot. As a result, when getting var(kernel_path), the vmlinuz soft link is also included, and our var(kernel_path) gets two identical vmlinuz. Error msg: + sed -i 's|KERNEL|"{rootfs_dir}/boot/vmlinuz-4.19.90-2010.2.0.0046.oe1.aarch64 {rootfs_dir}/boot/vmlinuz-4.19.90-2010.2.0.0046.oe1.aarch64"|g' {i2r_src}/env/iso2rootfs/kvm.sh sed: -e expression #1, char 133: unterminated `s' command [How] 1. When filtering vmlinuz: Before: realpath {rootfs_dir}/boot/vmlinuz* | grep -v rescue After: realpath {rootfs_dir}/boot/vmlinuz-* | grep -v rescue 2. Add a judgement, if the kernel num of result rootfs is not equal to 1, exit. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 87a51acaca5f..754638c95b39 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -375,8 +375,12 @@ test_rootfs() log_info "starting test rootfs ..." local test_script="${CGZ_PATH}/kvm.sh" - local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz*|grep -v rescue) + local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz-*|grep -v rescue) local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/boot/initramfs.lkp*) + + local kernel_num=$(echo "$kernel_path" | wc -l) + [ "$kernel_num" == "1" ] || die "result rootfs have multiple vmlinuz, num: $kernel_num" + local root_path case ${rootfs_protocol} in "nfs") -- 2.23.0

3 4

[PATCH v2 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 13 Nov '20

13 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971s response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..8eaac442 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,28 @@ +#!/usr/bin/env ruby + +LKP_SRC ||= ENV['LKP_SRC'] + +# General statistics: +# total time: 0.3987s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +def add_result(results, key, val) + results[key] ||= [] + results[key] << val +end + +$stdin.each_line do |line| + case line + when /total time:\s+([0-9.]+)/ + add_result(results_total, 'total time', $1.to_f) + end +end + +results_total.each do |key, vals| + puts "#{key}: #{vals.inject(0.0, :+)s}" +end -- 2.23.0

3 2

[PATCH v3 compass-ci] container/srv-http: support display *.log file content in browser
by Lu Weitao 13 Nov '20

13 Nov '20

[why] haven't define some file(*.log) behavior of browser in mime.types, so download this file on browser now. [how] 1. add correspondence of file in mime.types 2. set default type in default.conf of nginx, other type file that haven't define in mime.types, will display the file content in browser Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 1 + container/srv-http/root/etc/nginx/mime.types | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..91371f1 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -13,6 +13,7 @@ server { } location / { + default_type text/html; autoindex on; autoindex_localtime on; allow all; diff --git a/container/srv-http/root/etc/nginx/mime.types b/container/srv-http/root/etc/nginx/mime.types index e7f2f1c..a4ee74b 100644 --- a/container/srv-http/root/etc/nginx/mime.types +++ b/container/srv-http/root/etc/nginx/mime.types @@ -4,8 +4,8 @@ types { text/css css; text/xml xml; text/yaml yaml; - text/plain txt time data sh; + text/plain txt time data sh log; application/json json; - application/x-gzip gz tgz; + application/x-gzip gz tgz cgz; } -- 2.23.0

4 7