- Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests] lkp-tests/*: update AUR Helper for archlinux
by Wang Yong 26 Nov '20

26 Nov '20

yaourt is unmaintained and yay also support native pacman operation, so only minor changes are needed Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 15 +++++++++++++++ distro/installer/archlinux | 15 +++++++++------ lib/install.sh | 2 +- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/distro/archlinux b/distro/archlinux index 408c5fa23..3f8d62ebc 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -109,3 +109,18 @@ pack_benchmark() ) done } + +add_aur_helper() +{ + echo "lkp ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers + + pacman -Q yay 2>/dev/null && return + + local user=lkp + local dir=/home/${user} + + mkdir -p ${dir} && chown ${user}:${user} ${dir} + pacman -S --needed --noconfirm git base-devel 2>/dev/null + su - ${user} -c "git clone https://aur.archlinux.org/yay.git" + su - ${user} -c "cd yay && makepkg -si --needed --noconfirm" +} diff --git a/distro/installer/archlinux b/distro/installer/archlinux index 27cdc63cf..3be9e7953 100755 --- a/distro/installer/archlinux +++ b/distro/installer/archlinux @@ -2,12 +2,15 @@ # Archlinux package installation -# yaourt is a wrapper for pacman that adds automated access -# to the AUR. -# See: https://wiki.archlinux.org/index.php/Yaourt +# Yet Another Yogurt - An AUR Helper Written in Go +# Yay is based on the design of yaourt, apacman and pacaur +# See: https://github.com/Jguer/yay -# yaourt not support root to build package +# Yay not support root to build package # Use regular user to do this -regular_user=`grep 1000 /etc/passwd | awk -F':' '{ print $1 }'` +. $LKP_SRC/distro/${DISTRO} +add_aur_helper -sudo -u $regular_user yaourt -S --noconfirm $* +regular_user=$(grep 1090 /etc/passwd | awk -F':' '{ print $1 }') + +sudo -u $regular_user yay -S --needed --noconfirm $* diff --git a/lib/install.sh b/lib/install.sh index 881ade9f4..bf7bf21d7 100755 --- a/lib/install.sh +++ b/lib/install.sh @@ -17,7 +17,7 @@ sync_distro_sources() yum update fi ;; redhat) yum update ;; - archlinux) yaourt -Sy ;; + archlinux) yay -Sy ;; opensuse) zypper update ;; oracle) yum update ;; -- 2.23.0

1 0

[PATCH lkp-tests 2/2] spec/submit/link_jobs: update monitors value
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml | 2 +- spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml | 2 +- spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml index 3f26d1ae..9a613c00 100644 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml @@ -26,4 +26,4 @@ pp: freq: 1000 test: add cpufreq_governor: -monitors: {} +monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml index 87945a00..8bf52d0f 100644 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml @@ -26,4 +26,4 @@ pp: freq: 10000 test: add cpufreq_governor: -monitors: {} +monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml index 247cdc37..8cb78287 100644 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml @@ -26,4 +26,4 @@ pp: freq: 20 test: cache cpufreq_governor: -monitors: {} +monitors: -- 2.23.0

1 0

[PATCH lkp-tests 1/2] lib/job.rb: set monitors to nil when no category input
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index c8893433..fe373e86 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -584,8 +584,9 @@ class Job def add_monitors @job[comment_to_symbol('auto generated by add_monitors()')] = nil - @job['monitors'] = {} + @job['monitors'] = nil for_each_in(@job, monitor_params) do |_pk, _h, m_n, _m_args| + @job['monitors'] ||= {} @job['monitors'][m_n] = nil end end -- 2.23.0

1 0

[PATCH compass-ci] doc/job/: add category.md
by Zhang Yu 26 Nov '20

26 Nov '20

Add a document for introducing the category field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/category.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/job/category.md diff --git a/doc/job/category.md b/doc/job/category.md new file mode 100644 index 0000000..9842f58 --- /dev/null +++ b/doc/job/category.md @@ -0,0 +1,21 @@ +# category + +Meaning: +- Every test has its category, the category field is a test type of test cases. +- When you write a job yaml, you can chose one of benchmark, functional, noise-benchmark + to assign the value of category. +- Here are the difference of difference category: + 1. If category: benchmark, default monitors: + kmsg: print the kernel startup information + boot-time: + uptime: start time diafnosis item + iostat: print the time when the data is collected + heartbeat: provide the heartbeat monitors service + ... + other monitors you can see them in $LKP_SRC/include/category/benchmark + 2. If category: functional, default monitors: + kmsg: print the kernel startup information + heartbeat: provide the heartbeat monitors service + meminfo: solid memory usage status or vm usage status + ... + you can see the monitors in $LKP_SRC/include/category/functional -- 2.23.0

2 2

[PATCH lkp-tests] stats/crystal: change the file permissions
by Zhang Yu 26 Nov '20

26 Nov '20

[why] the file permissions should be 664. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mariadb.cr | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 stats/crystal/sysbench-mariadb.cr diff --git a/stats/crystal/sysbench-mariadb.cr b/stats/crystal/sysbench-mariadb.cr old mode 100755 new mode 100644 -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/crystal: add sysbench-mysql.cr
by Zhang Yu 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mysql.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 stats/crystal/sysbench-mysql.cr diff --git a/stats/crystal/sysbench-mysql.cr b/stats/crystal/sysbench-mysql.cr new file mode 100644 index 00000000..1f6709c2 --- /dev/null +++ b/stats/crystal/sysbench-mysql.cr @@ -0,0 +1,17 @@ +#!/usr/bin/env ruby + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) + +while (line = STDIN.gets) + next unless line =~ /transactions:.*\(\s*(\S+)/ + puts "transactions: #{$1}" +end -- 2.23.0

3 3

[PATCH lkp-tests 2/3] iso2rootfs: delete the unnecessary fields
by Yu Chuan 26 Nov '20

26 Nov '20

[Why] 'INITRD_HTTP_HOST' and 'INITRD_HTTP_PORT' should be added by the user on demand at runtime, no need to merge into the git repository. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 2 -- tests/iso2rootfs | 5 ++--- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 4f3e9c83e6cd..a0ffd280676d 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -7,8 +7,6 @@ iso2rootfs: rootfs_protocol: rootfs_server: rootfs_path: - initrd_http_host: - initrd_http_port: # config rootfs related fields ## supported fields: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 761fbb8b70cf..4a19d42ddd94 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -49,8 +49,6 @@ check_yaml_vars() "rootfs_protocol" "rootfs_server" "rootfs_path" - "initrd_http_host" - "initrd_http_port" ) for yaml_t in "${yaml_vars[@]}" @@ -190,7 +188,8 @@ check_os() get_cgz() { - local cgz_url="http://${initrd_http_host}:${initrd_http_port}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/iso2rootfs.cgz" + + local cgz_url="http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/iso2rootfs.cgz" local cgz_name cgz_name=$(basename "${cgz_url}") local cgz_path_t=${ENV_DIR}/${cgz_name} -- 2.23.0

2 2

[PATCH v2 compass-ci] create define_files in result_root
by Lu Weitao 26 Nov '20

26 Nov '20

[Why] User need debug their result-parse-script which haven't commit in lkp-tests/stats. [How] We already support user write script as "define_file" => {"stats/$script_name" => "$script_content"} in job.yaml, the job.yaml will be save as job to ES DB, write the job['define_file']['stats/$script_name'](value) at $result_root/stats/$script, extract-stats service will parse result by $result_root/stats/$script(user's result-parse-script) firstly Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index bcf6e52..74afee1 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -22,6 +22,8 @@ class StatsWorker if job_id # will consume the job by post-processing job = @es.get_job_content(job_id) result_root = job["result_root"]? + define_files = job["define_files"]? + write_user_stats(result_root.to_s, define_files.as_h) if define_files.is_a?(JSON::Any) result_post_processing(job_id, result_root.to_s, queue_path) @tq.delete_task(queue_path + "/in_process", "#{job_id}") end @@ -139,4 +141,17 @@ class StatsWorker STDERR.puts e.message end end + + def write_user_stats(result_root : String, define_files : Hash(String, JSON::Any)) + return nil unless result_root && File.exists?(result_root) + + define_files.each do |k, v| + if k.match(/^stats/) + Dir.mkdir("#{result_root}/stats") unless File.exists?("#{result_root}/stats") + File.open("#{result_root}/#{k}", "w+", File::Permissions.new(0o775)) do |file| + file.puts v + end + end + end + end end -- 2.23.0

2 2

Re: [PATCH v2 lkp-tests] tats/crystal: add hackbench-git.cr
by Bai Jing 26 Nov '20

26 Nov '20

^^^ stats Thanks, Baijing Reply-To: In-Reply-To: <20201125115826.760733-1-ylzhangah(a)qq.com> On Wed, Nov 25, 2020 at 07:58:26PM +0800, Zhang Yale wrote: >Make ruby2crystal convert > >Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> >--- > stats/crystal/hackbench-git.cr | 22 ++++++++++++++++++++++ > 1 file changed, 22 insertions(+) > create mode 100644 stats/crystal/hackbench-git.cr >diff --git a/stats/crystal/hackbench-git.cr b/stats/crystal/hackbench-git.cr >new file mode 100644 >index 00000000..2c043fc9 >--- /dev/null >+++ b/stats/crystal/hackbench-git.cr >@@ -0,0 +1,22 @@ >+#!/usr/bin/env crystal >+ >+# --------socket thread num=20-------- >+# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 >+# Running with 20*40 (== 800) tasks. >+# Time: 0.495 >+# ... >+ >+time = [] of Float64 >+ >+while (line = gets) >+ case line >+ when /^-+(\w+ \w+ \w+)/ >+ args = $1.tr(" ", "_") >+ when /^Running with .* $== (\d+)$ tasks/ >+ tasks = $1.to_i >+ when /^Time:/ >+ time << line.split[1].to_f >+ end >+end >+ >+puts "#{args}_#{tasks}: #{time.sum / time.size}" >-- >2.23.0 >

2 1

[PATCH v3 lkp-tests 3/3] tests/iozone: refactor format optimization
by Lu Kaiyi 26 Nov '20

26 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/iozone b/tests/iozone index 5186766f..923d8331 100755 --- a/tests/iozone +++ b/tests/iozone @@ -47,8 +47,7 @@ if [ -n "$block_size" ]; then ;; "preadv") args+=" -i 12" ;; - *) echo "invalid test option, please check again!" - exit 1 + *) echo "invalid test option, please check again!" && exit 1 ;; esac done -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/3] stats/mysql-server: remove redundant whitespace
by Lu Kaiyi 26 Nov '20

26 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/mysql-server | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/stats/mysql-server b/stats/mysql-server index d3f3fd79..776f3f2e 100755 --- a/stats/mysql-server +++ b/stats/mysql-server @@ -16,9 +16,9 @@ while (line = STDIN.gets) # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 when /\[\s*\d+\%\]/ - results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') - results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') - results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') + results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') + results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') + results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tats/crystal: add hackbench-git.cr
by Zhang Yale 25 Nov '20

25 Nov '20

Make ruby2crystal convert Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/crystal/hackbench-git.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 stats/crystal/hackbench-git.cr diff --git a/stats/crystal/hackbench-git.cr b/stats/crystal/hackbench-git.cr new file mode 100644 index 00000000..2c043fc9 --- /dev/null +++ b/stats/crystal/hackbench-git.cr @@ -0,0 +1,22 @@ +#!/usr/bin/env crystal + +# --------socket thread num=20-------- +# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 +# Running with 20*40 (== 800) tasks. +# Time: 0.495 +# ... + +time = [] of Float64 + +while (line = gets) + case line + when /^-+(\w+ \w+ \w+)/ + args = $1.tr(" ", "_") + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + time << line.split[1].to_f + end +end + +puts "#{args}_#{tasks}: #{time.sum / time.size}" -- 2.23.0

1 0

[PATCH lkp-tests] stats/crystal: add sysbench-mysql.cr
by Zhang Yu 25 Nov '20

25 Nov '20

[why] Make ruby2crystal convert. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mysql.cr | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 stats/crystal/sysbench-mysql.cr diff --git a/stats/crystal/sysbench-mysql.cr b/stats/crystal/sysbench-mysql.cr new file mode 100644 index 00000000..1bab78c0 --- /dev/null +++ b/stats/crystal/sysbench-mysql.cr @@ -0,0 +1,19 @@ +#!/usr/bin/env crystal + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) + +while (line = STDIN.gets) + case line + when /transactions:.*\(\s*(\S+)/ + puts "transactions: #{$1}" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/job: add runtime.md
by Zhang Yale 25 Nov '20

25 Nov '20

Add a document for introducing the runtime field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/runtime.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 doc/job/runtime.md diff --git a/doc/job/runtime.md b/doc/job/runtime.md new file mode 100644 index 0000000..c604bfa --- /dev/null +++ b/doc/job/runtime.md @@ -0,0 +1,27 @@ +# runtime + +Meaning: +- Seconds to run before exiting. +- Limit the test's run time in $runtime. + +Usage example: + +```bash +#!/bin/sh +# - runtime +``` + +`runtime` is a parameter of test script. +For any test case needing parameter(s), +need such comment line and follow exactly +the same format `# - parameter`. + +```yaml +runtime: 300s +``` + +```yaml +runtime: 1d +sleep: +``` + -- 2.23.0

1 0

[PATCH lkp-tests 3/3] iso2rootfs: add comment to iso2rootfs.yaml
by Yu Chuan 25 Nov '20

25 Nov '20

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index a0ffd280676d..2c4d3a93ff62 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -1,9 +1,18 @@ suite: iso2rootfs category: benchmark iso2rootfs: + # iso related fields to be used to generate rootfs iso_os: iso_arch: iso_version: + + # place the result rootfs related fields + ## 1. result rootfs will be placed in the following localtion on the + ## remote file server: + ## - {remote_file_server}/{rootfs_path}/{iso_os}/{iso_arch}/ + ## 2. remote file server protocols current supported: + ## - nfs + ## - cifs rootfs_protocol: rootfs_server: rootfs_path: -- 2.23.0

1 0

[PATCH lkp-tests 1/3] iso2rootfs: delete the meaningless field
by Yu Chuan 25 Nov '20

25 Nov '20

[Why] The 'iso2rootfs_cgz' field will always be 'iso2rootfs.cgz', so delete this field, and use the 'iso2rootfs.cgz' directly. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 1 - tests/iso2rootfs | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 1a643b538ac1..4f3e9c83e6cd 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -1,7 +1,6 @@ suite: iso2rootfs category: benchmark iso2rootfs: - iso2rootfs_cgz: iso_os: iso_arch: iso_version: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index fcaadaa1ba46..761fbb8b70cf 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -43,7 +43,6 @@ check_yaml_vars() "os" "os_arch" "os_version" - "iso2rootfs_cgz" "iso_os" "iso_arch" "iso_version" @@ -191,7 +190,7 @@ check_os() get_cgz() { - local cgz_url="http://${initrd_http_host}:${initrd_http_port}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/${iso2rootfs_cgz}" + local cgz_url="http://${initrd_http_host}:${initrd_http_port}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/iso2rootfs.cgz" local cgz_name cgz_name=$(basename "${cgz_url}") local cgz_path_t=${ENV_DIR}/${cgz_name} -- 2.23.0

1 0

[PATCH v5 lkp-tests] job: fix each_jobs to fit multi-args
by Bai Jing 25 Nov '20

25 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash={"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03","os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20","os_arch"=>"aarch64"} h2 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} Because "@job2=hash" will merge the h2 to @job after "each_job". We need to merge the h2 to @job before "each_job". So do the merge advance and set @job2={} Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index c8893433..9272392e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -730,7 +730,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/job: add runtime.md
by Zhang Yale 25 Nov '20

25 Nov '20

Add a document for introducing the runtime field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/runtime.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/job/runtime.md diff --git a/doc/job/runtime.md b/doc/job/runtime.md new file mode 100644 index 0000000..296fc6e --- /dev/null +++ b/doc/job/runtime.md @@ -0,0 +1,20 @@ +# runtime + +Meaning: +Limit run time to runtime seconds. + +Usage example: + +```#/bin/bash +# - runtime +``` + +`runtime` is a parameter of test script. For any test case +needing paramter(s), need such comment line and follow exactly +the same format `# - parameter`. + +job.yaml: + +``` +runtime: 300s +``` -- 2.23.0

2 2

[PATCH lkp-tests] stats/atomic: fix rubocop offenses
by Zhang Yale 25 Nov '20

25 Nov '20

Inspecting 1 file C Offenses: atomic:6:1: C: [Corrected] Layout/TrailingWhitespace: Trailing whitespace detected. 1 file inspected, 1 offense detected, 1 offense corrected Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/atomic | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/atomic b/stats/atomic index 71fc4fe6..422d575b 100755 --- a/stats/atomic +++ b/stats/atomic @@ -3,7 +3,7 @@ while (line = STDIN.gets) line.chomp! next unless line =~ /atomic:\s+\d+\s+(\d+).+\s+(\d+)$/ - + puts "threads: #{$1}" puts "score: #{$2}" end -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/job/: add os-os_verison-os_arch.md
by Zhang Yu 25 Nov '20

25 Nov '20

Add a document for introducing the os, os_version, os_arch fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/os-os_verison-os_arch.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/os-os_verison-os_arch.md diff --git a/doc/job/os-os_verison-os_arch.md b/doc/job/os-os_verison-os_arch.md new file mode 100644 index 0000000..9939414 --- /dev/null +++ b/doc/job/os-os_verison-os_arch.md @@ -0,0 +1,16 @@ +# os, os_version, os_arch + +Meaning: +- Each test tells which os it needs, and the os_version, os_arch are also need to be specified, + then the machine will run an os according to these three fields. +- The os, os_version, os_arch are the key for users to specify the os related parameters. +- If os, os_version, os_arch are not given by users, it will use the default openeuler os related parameters. +- Here are some examples: + os os_version os_arch + openeuler 20.03, 1.0 aarch64, x86_64 + debian sid, 10 aarch64, x86_64 + centos 7.6, 7.8 aarch64, x86_64 + ... + +Usage example: +- submit iperf.yaml testbox=vm-2p8g os=openeuler os_arch=aarch64 os_version=20.03 -- 2.23.0

1 0

[PATCH compass-ci] doc/job: add runtime.md
by Zhang Yale 25 Nov '20

25 Nov '20

Add a document for introducing the runtime field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/runtime.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 doc/job/runtime.md diff --git a/doc/job/runtime.md b/doc/job/runtime.md new file mode 100644 index 0000000..844cdbb --- /dev/null +++ b/doc/job/runtime.md @@ -0,0 +1,20 @@ +# runtime + +Meaning: +Limit run time to runtime seconds. + +Usage example: + +```#/bin/bash +# - runtime +``` + +It explicitly tells that `runtime` are the parameters of test script. +For any test case needing paramter(s), need such comment lines and +follow exactly the same format `# - parameter`. + +job.yaml: + +``` +runtime: 300s +``` -- 2.23.0

1 1

[PATCH compass-ci 4/4] assign-account: build email message for updating configuration
by Luan Shengde 25 Nov '20

25 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../assign-account/build-update-email.rb | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 container/assign-account/build-update-email.rb diff --git a/container/assign-account/build-update-email.rb b/container/assign-account/build-update-email.rb new file mode 100755 index 0000000..40d6b3b --- /dev/null +++ b/container/assign-account/build-update-email.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_update_message(email, account_info, conf_info) + message = if conf_info['enable_login'] + + <<~EMAIL_MESSAGE + To: #{email} + Subject: [compass-ci] account update notice + + Dear user: + + Your account has been update successfully. + You can use the following command to login the server. + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + + regards + compass-ci + EMAIL_MESSAGE + else + <<~EMAIL_MESSAGE + To: #{email} + Subject: [compass-ci] account expiration notice + + Dear user: + + your account has expired. + consult the manager for help if you want to continue using the account. + + regards + compass-ci + EMAIL_MESSAGE + end + + return message +end -- 2.23.0

1 0

[PATCH compass-ci 3/4] assign-account: reattach new dir for the container
by Luan Shengde 25 Nov '20

25 Nov '20

[why] enable execute command 'usermod -s shell' to enable/disable account to login Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/start | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/container/assign-account/start b/container/assign-account/start index 2290982..793191a 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -. $CCI_SRC/container/defconfig.sh +. ../defconfig.sh docker_rm assign-account @@ -11,8 +11,7 @@ cmd=( --restart=always --name=assign-account -d - -v /etc/passwd:/etc/passwd:ro - -v /etc/group:/etc/group:ro + -v /etc:/etc:rw -v /srv/account-info:/opt/account_data/:rw -v /home:/home:rw -p 29999:29999 @@ -20,14 +19,3 @@ cmd=( ) "${cmd[@]}" - -cat << EOF -port: 29999 ---- -Client: -./assign-tool.rb email_file - -email subject: "apply ssh account" - -Refer to compass-ci/doc/apply-ssh-account.md for more information. -EOF -- 2.23.0

1 0

[PATCH compass-ci 2/4] assign-account: add new function and optimize
by Luan Shengde 25 Nov '20

25 Nov '20

[why] meet new options for answerback-email meet update configuration function for answerback-email [how] add new function for the new options for answerback-email optimize current function to enable update Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 117 +++++++++++++++---- 1 file changed, 92 insertions(+), 25 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 0e91057..95bec14 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -29,11 +29,14 @@ API: call graph: setup_jumper_account_info - read_account_info - build_account_name + read account + read_account_info + build_account_info + reread_account_info read_jumper_info config_default_yaml config_authorized_key + permit_login_config generate_ssh_key the returned data for setup_jumper_account_info like: @@ -41,12 +44,14 @@ the returned data for setup_jumper_account_info like: "my_login_name" => "login_name", "password" => "password", "jumper_host" => "0.0.0.0", - "jumper_port" => "10000" + "jumper_port" => "10000", + "my_jumper_pubkey" => my_jumper_pubkey } =end require 'fileutils' +require 'yaml' # get jumper and account info class AccountStorage @@ -95,14 +100,21 @@ class AccountStorage end def setup_jumper_account_info - login_name, password = read_account_info + if @data.key?('is_update_account') && @data['is_update_account'] + login_name = @data['my_login_name'] + password = reread_account_info + else + login_name, password = read_account_info + end + jumper_host, jumper_port = read_jumper_info - pub_key = @data['my_ssh_pubkey'][0] - ssh_dir = File.join('/home/', login_name, '.ssh') - config_authorized_key(login_name, pub_key, ssh_dir) unless pub_key.nil? + config_authorized_key(login_name) config_default_yaml(login_name) - my_jumper_pubkey = generate_ssh_key(login_name, ssh_dir) if @data['gen_sshkey'].eql? true + permit_login_config(login_name) + if @data.key?('gen_sshkey') && @data['gen_sshkey'] + my_jumper_pubkey = generate_ssh_key(login_name) + end jumper_account_info = { 'my_login_name' => login_name, @@ -115,34 +127,89 @@ class AccountStorage return jumper_account_info end - def generate_ssh_key(login_name, ssh_dir) + def permit_login_config(login_name) + if @data.key?('enable_login') && @data['enable_login'] + %x(usermod -s /usr/bin/zsh #{login_name}) + else + %x(usermod -s /sbin/nologin #{login_name}) + end + end + + def generate_ssh_key(login_name) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + pub_key_file = File.join(ssh_dir, 'id_rsa.pub') + + return if File.exist?(pub_key_file) && \ + @data['my_ssh_pubkey'].include?(File.read(pub_key_file).strip) + %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@account-vm") - %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) - File.read("/home/#{login_name}/.ssh/id_rsa.pub") + + FileUtils.chown_R(login_name, login_name, ssh_dir) + File.read("/home/#{login_name}/.ssh/id_rsa.pub").strip end - def config_default_yaml(login_name) + def touch_default_yaml(login_name) default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') - FileUtils.mkdir_p default_yaml_dir + FileUtils.mkdir_p default_yaml_dir unless File.exist? default_yaml_dir + + default_yaml = File.join(default_yaml_dir, 'account.yaml') + FileUtils.touch default_yaml unless File.exist? default_yaml + default_yaml + end + def config_default_yaml(login_name) + default_yaml = touch_default_yaml(login_name) + + account_yaml = YAML.load_file(default_yaml) || {} # my_email, my_name, my_uuid is required to config default yaml file # they are added along with 'my_ssh_pubkey' when sending assign account request - File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| - file.puts "my_email: #{@data['my_email']}" - file.puts "my_name: #{@data['my_name']}" - file.puts "my_uuid: #{@data['my_uuid']}" - end - %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + account_yaml['my_email'] = @data['my_email'] + account_yaml['my_name'] = @data['my_name'] + account_yaml['my_uuid'] = @data['my_uuid'] + + f = File.new(default_yaml, 'w') + f.puts account_yaml.to_yaml + f.close + FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") + end + + def config_authorized_key(login_name) + pub_key = @data['my_ssh_pubkey'][0] + + return if pub_key.nil? + return if pub_key.strip.end_with?('account-vm') + + ssh_dir = File.join('/home/', login_name, '.ssh') + Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + + authorized_file = File.join(ssh_dir, 'authorized_keys') + FileUtils.touch authorized_file unless File.exist? authorized_file + + store_pubkey(ssh_dir, login_name, authorized_file, pub_key) end - def config_authorized_key(login_name, pub_key, ssh_dir) - Dir.mkdir ssh_dir, 0o700 - Dir.chdir ssh_dir - f = File.new('authorized_keys', 'w') + def store_pubkey(ssh_dir, login_name, authorized_file, pub_key) + authorized_keys = File.read(authorized_file).split("\n") + + return if authorized_keys.include?(pub_key.strip) + + f = File.new(authorized_file, 'a') f.puts pub_key f.close - File.chmod 0o600, 'authorized_keys' - %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) + + FileUtils.chown_R(login_name, login_name, ssh_dir) + File.chmod 0o600, authorized_file + end + + def reread_account_info + my_login_name_file = File.join(@account_dir, 'assigned-users', @data['my_login_name']) + + message = "No such assigned account exists: #{my_login_name_file}." + raise message unless File.exist? my_login_name_file + + password = File.readlines(my_login_name_file)[0].chomp + + return password end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: submit separate_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will user 'rspec spec/submit_spec.rb' to separate job.yaml, it contains several yaml documents separated by "---" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/separate_yaml/ftq.yaml | 20 +++++++++++++ .../ftq/ftq-1000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-10000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-20-2-6000ss-cache.yaml | 28 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 5 files changed, 108 insertions(+) create mode 100644 spec/submit/separate_yaml/ftq.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/ftq.yaml new file mode 100644 index 00000000..e54adc8b --- /dev/null +++ b/spec/submit/separate_yaml/ftq.yaml @@ -0,0 +1,20 @@ +suite: ftq +category: noise-benchmark + +nr_task: 100% +samples: 100000ss + +ftq: + test: add + freq: 10000 + +--- +ftq: + test: add + freq: 1000 + +--- +samples: 6000ss +ftq: + test: cache + freq: 20 diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..855e7210 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..78725424 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..40edfcb2 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..ce6fed25 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'separate yaml spec' do + submit_job('separate_yaml') + end end -- 2.23.0

2 1