November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] stats/crystal: add libmicro.cr
by Bai Jing 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/crystal/libmicro.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100755 stats/crystal/libmicro.cr diff --git a/stats/crystal/libmicro.cr b/stats/crystal/libmicro.cr new file mode 100755 index 00000000..2a3f42f0 --- /dev/null +++ b/stats/crystal/libmicro.cr @@ -0,0 +1,10 @@ +#!/usr/bin/env crystal + +# prc thr usecs/call samples errors cnt/samp +# close_tmp 1 1 0.61880 201 0 640 + +while (line = STDIN.gets) + next unless line =~ /^(\w+)\s+\S+\s+\S+\s+(\S+)/ + + puts "#{$1}: #{$2}" +end -- 2.23.0

1 1

[PATCH v3 lkp-tests] lkp-tests/*: update AUR Helper for archlinux
by Wang Yong 26 Nov '20

26 Nov '20

yaourt is unmaintained and yay also support native pacman operation, so only minor changes are needed Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 15 +++++++++++++++ distro/installer/archlinux | 15 +++++++++------ lib/install.sh | 2 +- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/distro/archlinux b/distro/archlinux index 408c5fa23..3f8d62ebc 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -109,3 +109,18 @@ pack_benchmark() ) done } + +add_aur_helper() +{ + echo "lkp ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers + + pacman -Q yay 2>/dev/null && return + + local user=lkp + local dir=/home/${user} + + mkdir -p ${dir} && chown ${user}:${user} ${dir} + pacman -S --needed --noconfirm git base-devel 2>/dev/null + su - ${user} -c "git clone https://aur.archlinux.org/yay.git" + su - ${user} -c "cd yay && makepkg -si --needed --noconfirm" +} diff --git a/distro/installer/archlinux b/distro/installer/archlinux index 27cdc63cf..3be9e7953 100755 --- a/distro/installer/archlinux +++ b/distro/installer/archlinux @@ -2,12 +2,15 @@ # Archlinux package installation -# yaourt is a wrapper for pacman that adds automated access -# to the AUR. -# See: https://wiki.archlinux.org/index.php/Yaourt +# Yet Another Yogurt - An AUR Helper Written in Go +# Yay is based on the design of yaourt, apacman and pacaur +# See: https://github.com/Jguer/yay -# yaourt not support root to build package +# Yay not support root to build package # Use regular user to do this -regular_user=`grep 1000 /etc/passwd | awk -F':' '{ print $1 }'` +. $LKP_SRC/distro/${DISTRO} +add_aur_helper -sudo -u $regular_user yaourt -S --noconfirm $* +regular_user=$(grep 1090 /etc/passwd | awk -F':' '{ print $1 }') + +sudo -u $regular_user yay -S --needed --noconfirm $* diff --git a/lib/install.sh b/lib/install.sh index 881ade9f4..bf7bf21d7 100755 --- a/lib/install.sh +++ b/lib/install.sh @@ -17,7 +17,7 @@ sync_distro_sources() yum update fi ;; redhat) yum update ;; - archlinux) yaourt -Sy ;; + archlinux) yay -Sy ;; opensuse) zypper update ;; oracle) yum update ;; -- 2.23.0

1 0

[PATCH lkp-tests 2/2] spec/submit/link_jobs: update monitors value
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml | 2 +- spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml | 2 +- spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml index 3f26d1ae..9a613c00 100644 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml @@ -26,4 +26,4 @@ pp: freq: 1000 test: add cpufreq_governor: -monitors: {} +monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml index 87945a00..8bf52d0f 100644 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml @@ -26,4 +26,4 @@ pp: freq: 10000 test: add cpufreq_governor: -monitors: {} +monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml index 247cdc37..8cb78287 100644 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml @@ -26,4 +26,4 @@ pp: freq: 20 test: cache cpufreq_governor: -monitors: {} +monitors: -- 2.23.0

1 0

[PATCH lkp-tests 1/2] lib/job.rb: set monitors to nil when no category input
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index c8893433..fe373e86 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -584,8 +584,9 @@ class Job def add_monitors @job[comment_to_symbol('auto generated by add_monitors()')] = nil - @job['monitors'] = {} + @job['monitors'] = nil for_each_in(@job, monitor_params) do |_pk, _h, m_n, _m_args| + @job['monitors'] ||= {} @job['monitors'][m_n] = nil end end -- 2.23.0

1 0

[PATCH compass-ci] doc/job/: add category.md
by Zhang Yu 26 Nov '20

26 Nov '20

Add a document for introducing the category field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/category.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 doc/job/category.md diff --git a/doc/job/category.md b/doc/job/category.md new file mode 100644 index 0000000..9842f58 --- /dev/null +++ b/doc/job/category.md @@ -0,0 +1,21 @@ +# category + +Meaning: +- Every test has its category, the category field is a test type of test cases. +- When you write a job yaml, you can chose one of benchmark, functional, noise-benchmark + to assign the value of category. +- Here are the difference of difference category: + 1. If category: benchmark, default monitors: + kmsg: print the kernel startup information + boot-time: + uptime: start time diafnosis item + iostat: print the time when the data is collected + heartbeat: provide the heartbeat monitors service + ... + other monitors you can see them in $LKP_SRC/include/category/benchmark + 2. If category: functional, default monitors: + kmsg: print the kernel startup information + heartbeat: provide the heartbeat monitors service + meminfo: solid memory usage status or vm usage status + ... + you can see the monitors in $LKP_SRC/include/category/functional -- 2.23.0

2 2

[PATCH lkp-tests] stats/crystal: change the file permissions
by Zhang Yu 26 Nov '20

26 Nov '20

[why] the file permissions should be 664. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mariadb.cr | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100755 => 100644 stats/crystal/sysbench-mariadb.cr diff --git a/stats/crystal/sysbench-mariadb.cr b/stats/crystal/sysbench-mariadb.cr old mode 100755 new mode 100644 -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/crystal: add sysbench-mysql.cr
by Zhang Yu 26 Nov '20

26 Nov '20

Make ruby2crystal convert Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/crystal/sysbench-mysql.cr | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 stats/crystal/sysbench-mysql.cr diff --git a/stats/crystal/sysbench-mysql.cr b/stats/crystal/sysbench-mysql.cr new file mode 100644 index 00000000..1f6709c2 --- /dev/null +++ b/stats/crystal/sysbench-mysql.cr @@ -0,0 +1,17 @@ +#!/usr/bin/env ruby + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) + +while (line = STDIN.gets) + next unless line =~ /transactions:.*\(\s*(\S+)/ + puts "transactions: #{$1}" +end -- 2.23.0

3 3

[PATCH lkp-tests 2/3] iso2rootfs: delete the unnecessary fields
by Yu Chuan 26 Nov '20

26 Nov '20

[Why] 'INITRD_HTTP_HOST' and 'INITRD_HTTP_PORT' should be added by the user on demand at runtime, no need to merge into the git repository. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 2 -- tests/iso2rootfs | 5 ++--- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 4f3e9c83e6cd..a0ffd280676d 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -7,8 +7,6 @@ iso2rootfs: rootfs_protocol: rootfs_server: rootfs_path: - initrd_http_host: - initrd_http_port: # config rootfs related fields ## supported fields: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 761fbb8b70cf..4a19d42ddd94 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -49,8 +49,6 @@ check_yaml_vars() "rootfs_protocol" "rootfs_server" "rootfs_path" - "initrd_http_host" - "initrd_http_port" ) for yaml_t in "${yaml_vars[@]}" @@ -190,7 +188,8 @@ check_os() get_cgz() { - local cgz_url="http://${initrd_http_host}:${initrd_http_port}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/iso2rootfs.cgz" + + local cgz_url="http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/pkg/nfs/${os}/${os_arch}/${os_version}/iso2rootfs.cgz" local cgz_name cgz_name=$(basename "${cgz_url}") local cgz_path_t=${ENV_DIR}/${cgz_name} -- 2.23.0

2 2

[PATCH v2 compass-ci] create define_files in result_root
by Lu Weitao 26 Nov '20

26 Nov '20

[Why] User need debug their result-parse-script which haven't commit in lkp-tests/stats. [How] We already support user write script as "define_file" => {"stats/$script_name" => "$script_content"} in job.yaml, the job.yaml will be save as job to ES DB, write the job['define_file']['stats/$script_name'](value) at $result_root/stats/$script, extract-stats service will parse result by $result_root/stats/$script(user's result-parse-script) firstly Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index bcf6e52..74afee1 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -22,6 +22,8 @@ class StatsWorker if job_id # will consume the job by post-processing job = @es.get_job_content(job_id) result_root = job["result_root"]? + define_files = job["define_files"]? + write_user_stats(result_root.to_s, define_files.as_h) if define_files.is_a?(JSON::Any) result_post_processing(job_id, result_root.to_s, queue_path) @tq.delete_task(queue_path + "/in_process", "#{job_id}") end @@ -139,4 +141,17 @@ class StatsWorker STDERR.puts e.message end end + + def write_user_stats(result_root : String, define_files : Hash(String, JSON::Any)) + return nil unless result_root && File.exists?(result_root) + + define_files.each do |k, v| + if k.match(/^stats/) + Dir.mkdir("#{result_root}/stats") unless File.exists?("#{result_root}/stats") + File.open("#{result_root}/#{k}", "w+", File::Permissions.new(0o775)) do |file| + file.puts v + end + end + end + end end -- 2.23.0

2 2

Re: [PATCH v2 lkp-tests] tats/crystal: add hackbench-git.cr
by Bai Jing 26 Nov '20

26 Nov '20

^^^ stats Thanks, Baijing Reply-To: In-Reply-To: <20201125115826.760733-1-ylzhangah(a)qq.com> On Wed, Nov 25, 2020 at 07:58:26PM +0800, Zhang Yale wrote: >Make ruby2crystal convert > >Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> >--- > stats/crystal/hackbench-git.cr | 22 ++++++++++++++++++++++ > 1 file changed, 22 insertions(+) > create mode 100644 stats/crystal/hackbench-git.cr >diff --git a/stats/crystal/hackbench-git.cr b/stats/crystal/hackbench-git.cr >new file mode 100644 >index 00000000..2c043fc9 >--- /dev/null >+++ b/stats/crystal/hackbench-git.cr >@@ -0,0 +1,22 @@ >+#!/usr/bin/env crystal >+ >+# --------socket thread num=20-------- >+# 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 >+# Running with 20*40 (== 800) tasks. >+# Time: 0.495 >+# ... >+ >+time = [] of Float64 >+ >+while (line = gets) >+ case line >+ when /^-+(\w+ \w+ \w+)/ >+ args = $1.tr(" ", "_") >+ when /^Running with .* $== (\d+)$ tasks/ >+ tasks = $1.to_i >+ when /^Time:/ >+ time << line.split[1].to_f >+ end >+end >+ >+puts "#{args}_#{tasks}: #{time.sum / time.size}" >-- >2.23.0 >

2 1