- Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci] add the container/initrd-cifs to share /srv/initrd
by Xiao Shenwei 13 Nov '20

13 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result it may cause: 1. our cifs service becomes slow 2. if the container exit, none of the three folders can be accessed. in order to reduce the coupling, need split them [how] one folder mount by one container [Usage]: Two ways for the client to mount initrd: 1. initrd. mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage. mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 22 ++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 89 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..341bfc9 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +RUN adduser -u 1090 -D lkp +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash && \ + rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..ca9de39 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t alpine/initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..7232bdf --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + alpine/initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

2 2

[PATCH v2 lkp-tests 3/3] iso2rootfs: solve the problem of multiple vmlinuz failures during test
by Yu Chuan 13 Nov '20

13 Nov '20

[Why] We use compass-ci/container/qcow2rootfs to generate rootfs from qcow2, but compass-ci updated the code, qcow2rootfs create a vmlinuz soft link at {rootfs_dir}/boot. As a result, when getting var(kernel_path), the vmlinuz soft link is also included, and our var(kernel_path) gets two identical vmlinuz. Error msg: + sed -i 's|KERNEL|"{rootfs_dir}/boot/vmlinuz-4.19.90-2010.2.0.0046.oe1.aarch64 {rootfs_dir}/boot/vmlinuz-4.19.90-2010.2.0.0046.oe1.aarch64"|g' {i2r_src}/env/iso2rootfs/kvm.sh sed: -e expression #1, char 133: unterminated `s' command [How] 1. When filtering vmlinuz: Before: realpath {rootfs_dir}/boot/vmlinuz* | grep -v rescue After: realpath {rootfs_dir}/boot/vmlinuz-* | grep -v rescue 2. Add a judgement, if the kernel num of result rootfs is not equal to 1, exit. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 87a51acaca5f..754638c95b39 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -375,8 +375,12 @@ test_rootfs() log_info "starting test rootfs ..." local test_script="${CGZ_PATH}/kvm.sh" - local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz*|grep -v rescue) + local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz-*|grep -v rescue) local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/boot/initramfs.lkp*) + + local kernel_num=$(echo "$kernel_path" | wc -l) + [ "$kernel_num" == "1" ] || die "result rootfs have multiple vmlinuz, num: $kernel_num" + local root_path case ${rootfs_protocol} in "nfs") -- 2.23.0

3 4

[PATCH v2 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 13 Nov '20

13 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971s response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..8eaac442 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,28 @@ +#!/usr/bin/env ruby + +LKP_SRC ||= ENV['LKP_SRC'] + +# General statistics: +# total time: 0.3987s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +results_total = {} + +def add_result(results, key, val) + results[key] ||= [] + results[key] << val +end + +$stdin.each_line do |line| + case line + when /total time:\s+([0-9.]+)/ + add_result(results_total, 'total time', $1.to_f) + end +end + +results_total.each do |key, vals| + puts "#{key}: #{vals.inject(0.0, :+)s}" +end -- 2.23.0

3 2

[PATCH v3 compass-ci] container/srv-http: support display *.log file content in browser
by Lu Weitao 13 Nov '20

13 Nov '20

[why] haven't define some file(*.log) behavior of browser in mime.types, so download this file on browser now. [how] 1. add correspondence of file in mime.types 2. set default type in default.conf of nginx, other type file that haven't define in mime.types, will display the file content in browser Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 1 + container/srv-http/root/etc/nginx/mime.types | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..91371f1 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -13,6 +13,7 @@ server { } location / { + default_type text/html; autoindex on; autoindex_localtime on; allow all; diff --git a/container/srv-http/root/etc/nginx/mime.types b/container/srv-http/root/etc/nginx/mime.types index e7f2f1c..a4ee74b 100644 --- a/container/srv-http/root/etc/nginx/mime.types +++ b/container/srv-http/root/etc/nginx/mime.types @@ -4,8 +4,8 @@ types { text/css css; text/xml xml; text/yaml yaml; - text/plain txt time data sh; + text/plain txt time data sh log; application/json json; - application/x-gzip gz tgz; + application/x-gzip gz tgz cgz; } -- 2.23.0

4 7

[PATCH lkp-tests] lib: add multi-args in job yaml
by Bai Jing 13 Nov '20

13 Nov '20

[original style] os: openeuler os_verison: 20.03 os_arch: aarch64 os: centos os_version: 8.7 os_arch: aarch64 [now style] os os_version os_arch: - openeuler 20.03 aarch64 - centos 20 aarch64 [point] Currently, only one key of this style(xx xxx xx) can be contained in YAML files. Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 01719c2e..f8608e94 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -250,7 +250,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) - @jobs << hash + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e log_error "#{jobfile}: " + e.message @@ -280,6 +280,30 @@ class Job @jobfile = jobfile end + def multi_args(hash) + jobs_array = [] + hash.each { |key, value| + next unless key =~ /^\w.*\s+.*\w$/ + + next unless value.class == Array + + key_array = key.split + value.each do |v| + v_array = v.split + next unless key_array.size == v_array.size + + hash_job = hash.clone + key_array.size.times do |i| + hash_job[key_array[i]] = v_array[i] + end + hash_job.delete(key) + jobs_array << hash_job + end + } + jobs_array = [hash] if jobs_array.empty? + return jobs_array + end + def delete_keys_from_spec(spec_file) return unless File.exist? spec_file @@ -703,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

4 7

[PATCH v4 lkp-tests] stats: add sysbench-mysql result parsing file
by Zhang Yu 13 Nov '20

13 Nov '20

[input] SQL statistics: queries performed: total: 298163456 transactions: 14908172 (25024.26 per sec.) [output] transactions: 25024.26 Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/sysbench-mysql | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 stats/sysbench-mysql diff --git a/stats/sysbench-mysql b/stats/sysbench-mysql new file mode 100755 index 00000000..836ab6c9 --- /dev/null +++ b/stats/sysbench-mysql @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) + +while (line = STDIN.gets) + case line.chomp! + when /transactions:.*\(\s*(\S+)/ + puts "transactions: #{$1}" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: fix LAB -> lab
by Ren Wen 13 Nov '20

13 Nov '20

1) Global config uses "lab" not "LAB". 2) Do a crystal-format. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1b37333..c263478 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -10,7 +10,7 @@ JOB_ES_PORT = 9200 JOB_ES_PORT_DEBUG = 9201 JOB_INDEX_TYPE = "jobs/_doc" -LAB = (ENV.has_key?("LAB") ? ENV["LAB"] : "nolab") +LAB = (ENV.has_key?("lab") ? ENV["lab"] : "nolab") SCHED_HOST = (ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1") SCHED_PORT = (ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"] : 3000).to_i32 @@ -25,7 +25,7 @@ SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" -OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" -SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" +OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" +SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" DEMO_JOB = %({"suite":"pixz","testcase":"pixz","category":"benchmark","nr_threads":1,"pixz":null,"job_origin":"jobs/pixz.yaml","testbox":"wfg-e595","arch":"x86_64","tbox_group":"wfg-e595","id":"100","kmsg":null,"boot-time":null,"uptime":null,"iostat":null,"heartbeat":null,"vmstat":null,"numa-numastat":null,"numa-vmstat":null,"numa-meminfo":null,"proc-vmstat":null,"proc-stat":null,"meminfo":null,"slabinfo":null,"interrupts":null,"kconfig":"x86_64-rhel-7.6","compiler":"gcc-7"}) -- 2.23.0

1 0

[PATCH v7 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 13 Nov '20

13 Nov '20

[Why] hackbench-git is an old version tool required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..f2be0c9d --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,30 @@ +#!/bin/sh +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is both a benchmark and a stress test for the Linux kernel scheduler. +## Its main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes)which communicate via either sockets +## or pipes and time how long it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' +[ -n "$iterations" ] || iterations=10 +[ -n "$loops" ] || loops=100 +[ -n "$nr_threads" ] || nr_threads=1 + +run_hackbench() +{ + # args: groups number, mode [process/thread], loops, ipc [pipe/socket] + hackbench_path="$BENCHMARK_ROOT"/hackbench-git/hackbench/hackbench + [ ! -x "${hackbench_path}" ] && exit 1 + + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

1 1

[PATCH v3 lkp-tests] test/build-pkg: fix a softlink will be established if the package fails
by Sun Yukui 13 Nov '20

13 Nov '20

[why] After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the softlink should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/build-pkg | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index ad8d4843..093bbe4d 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -65,6 +65,19 @@ request_pkg() [ -s PKGBUILD ] || die "PKGBUILD is empty" } +create_softlink() +{ + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" +} build_source_pkg() { @@ -86,24 +99,11 @@ build_source_pkg() cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -} -create_softlink() -{ - local soft_path="$PKG_MNT/${pack_to}/${pkgname}" - local bm_name=$(basename $(realpath ${cgz_name})) - - if [ -n "$upstream_tag" ]; then - echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" - fi - - echo "create soft link: latest.cgz -> ${bm_name}" - ln -sf "${bm_name}" "${soft_path}/latest.cgz" + [ "$?" == 0 ] || create_softlink } check_vars mount_dest request_pkg build_source_pkg -create_softlink -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats: add hackbench-git result parsing script.
by Zhang Yale 13 Nov '20

13 Nov '20

[input] --------socket thread num=20-------- 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] socket_thread_num_800: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..187f1c19 --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(' ', '_') + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + seconds = line.split[1] + time << seconds.to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

2 2