- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] job.rb: fix add second program to pp
by Wei Jihui 04 Nov '20

04 Nov '20

[Why] Now we add new field in job yaml like this: on_fail: sleep: 300 and we want add sleep to pp, but at pp() function, it only traverse job's keys--->on_fail, this can not find sleep. it should find sleep at on_fail's keys. [How] when traverse job to find program, in this case, it is sleep, then find it at on_fail's keys, not job's keys. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 5c2c9517..3fdf8c79 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -576,10 +576,10 @@ class Job @job[comment_to_symbol('auto generated by add_pp()')] = nil @job["pp"] = Hash.new() init_program_options() - for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, p_args| - k = hash_key_re_string(@job, p_n, @job['pp'].keys) - if k && !(a)job[k].is_a?(Hash) - @job['pp'][k] = @job[k] + for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, _p_args| + k = hash_key_re_string(h, p_n, @job['pp'].keys) + if k && h[k] && !h[k].is_a?(Hash) + @job['pp'][k] = h[k] next end options_hash = @referenced_programs[p_n] -- 2.23.0

1 0

[PATCH lab-z9] hosts: fix taishan200-2280-2s48p-256g--a102 to taishan200-2280-2s48p-512g--a102
by Zhang Yu 04 Nov '20

04 Nov '20

[Why] The memory of taishan200-2280-2s48p-256g--a95 is 512g, so need to fix 256g to 512g. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename hosts/{taishan200-2280-2s48p-256g--a102 => taishan200-2280-2s48p-512g--a102} (100%) diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-512g--a102 similarity index 100% rename from hosts/taishan200-2280-2s48p-256g--a102 rename to hosts/taishan200-2280-2s48p-512g--a102 -- 2.23.0

1 1

[PATCH v4 lkp-tests] tests/build-pkg: modify save location for *.cgz file
by Liu Shaofei 04 Nov '20

04 Nov '20

[why] the path of *.cgz file generated by build-pkg script cannot be obtained during test. So the path of cgz file need to be changed. [how] below is example for openeuler: latest.cgz -> f71820d6055373c9b9381615e28a28f307bb2df4.cgz f71820d6055373c9b9381615e28a28f307bb2df4.cgz v1.8.1.cgz -> 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz v1.5.1.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz df0584e46db9e7b412626deab10b923504745b2c.cgz Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..f9bfa377 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,15 +23,19 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount $LKP_SERVER:$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } @@ -75,12 +81,22 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 } +create_softlink() +{ + [ -n "$upstream_tag" ] || return 0 + [ -e "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" ] && return 0 + + ln -sf "$(basename $(realpath ${cgz_name}))" "$PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz" || return 0 + echo "create soft link: $PKG_MNT/${pack_to}/${pkgname}/${upstream_tag}.cgz -> $(basename $(realpath ${cgz_name}))" +} + check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

3 4

[PATCH v2 compass-ci] monitoring/filter.cr: query value support regularity
by Wu Zhende 04 Nov '20

04 Nov '20

[Why] Enhanced monitoring function. Regular expressions can be used in query's value. query can be {"job_id": "z9.*"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index bc6f901..d520e6e 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,13 +56,21 @@ class Filter def match_query(query : Hash(String, JSON::Any), msg : Hash(String, JSON::Any)) query.each do |key, value| + return false unless msg.has_key?(key) + value = value.as_a - if value.includes?(nil) - return false unless msg.has_key?(key) - else - return false unless value.includes?(msg[key]?) - end + next if value.includes?(nil) + next if value.includes?(msg[key]?) + + return false unless regular_match(value, msg[key]?.to_s) end return true end + + private def regular_match(rules, string) + rules.each do |rule| + return true if string =~ /#{rule}/ + end + return false + end end -- 2.23.0

2 1

[PATCH v1 compass-ci] improve multi-qemu code to support queue parameter
by Xiao Shenwei 04 Nov '20

04 Nov '20

Usage: multi-qemu [-n] [-c] [-q] -n, --name HOSTNAME_PREFIX specify used hostname_prefix -c, --count count how much VM do you need -q, --queues queues queues from which tasks are obtained -h, --help show this message example: ./multi-qemu -n vm-2p8g.xsw -c 2 -q vm-2p8g~xsw,vm-2p8g.aarch64 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-qemu | 98 +++++++++++++++++++++++++++++++++----------- 1 file changed, 74 insertions(+), 24 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 23e0451..03564f7 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -4,41 +4,91 @@ # frozen_string_literal: true require 'fileutils' +require 'optparse' +require 'time' -PWD = Dir.pwd +opt = {} +options = OptionParser.new do |opts| + opts.banner = 'Usage: multi-qemu [-n] [-c] [-q]' + + opts.separator '' + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'specify used hostname_prefix') do |name| + opt['hostname_prefix'] = name + end + + opts.on('-c count', '--count count', 'how much VM do you need') do |num| + opt['nr_vm'] = num + end + + opts.on('-q queues', '--queues queues', 'requested queues, use "," to separate more than 2 values') do |queues| + opt['queues'] = queues + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.size.zero? + puts options + exit +end + +options.parse!(ARGV) # Run multiple QEMU in parallel -HOSTNAME = ARGV[0] || "vm-2p8g--#{ENV['USER']}" -NR_VM = ARGV[1] || 1 +PWD = Dir.pwd +HOSTNAME = opt['hostname_prefix'] || "vm-2p8g.#{ENV['USER']}" +NR_VM = opt['nr_vm'] || 1 +QUEUES = opt['queues'] || "vm-2p8g.#{RUBY_PLATFORM.split('-')[0]}" +LOG_DIR = '/srv/cci/serial/logs' -def run(seqno) - loop do - start_time = Time.new - hostname = "#{HOSTNAME}-#{seqno}" - log_file = "/srv/cci/serial/logs/#{hostname}" +def main(hostname) + start_time = Time.new.strftime('%Y-%m-%d %H:%M:%S') + log_file = "#{LOG_DIR}/#{hostname}" + record_runtime(log_file, start_time) + run_vm(hostname) + duration = ((Time.new - Time.parse(start_time)) / 60).round(2) + record_runtime(log_file, duration, is_start: false) +end +def record_runtime(log_file, message, is_start: true) + if is_start File.open(log_file, 'w') do |f| # fluentd refresh time is 1s # let fluentd to monitor this file first sleep(2) - f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting QEMU" + f.puts "\n#{message} starting QEMU" end + return + end + File.open(log_file, 'a') do |f| + f.puts "\nTotal QEMU duration: #{message} minutes" + end +end - pwd_hostname = File.join(PWD, hostname) - FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) - FileUtils.cd(pwd_hostname) - system( - { 'hostname' => hostname }, - ENV['CCI_SRC'] + '/providers/qemu.sh' - ) - - duration = ((Time.new - start_time) / 60).round(2) - File.open(log_file, 'a') do |f| - f.puts "\nTotal QEMU duration: #{duration} minutes" - end +def run_vm(hostname) + pwd_hostname = File.join(PWD, hostname) + FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) + FileUtils.cd(pwd_hostname) + system( + { 'hostname' => hostname, 'queues' => QUEUES }, + ENV['CCI_SRC'] + '/providers/qemu.sh' + ) +end - # sleep 5s is for fluentd to collect it's log - sleep(5) +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.backtrace + # if an exception happend, request the next time after 30 seconds + sleep 25 + ensure + sleep 5 + end end end @@ -55,7 +105,7 @@ def multiqemu pids = [] NR_VM.to_i.times do |i| pid = Process.fork do - run i + loop_main("#{HOSTNAME}-#{i}") end pids << pid end -- 2.23.0

2 3

[PATCH v2 lkp-tests 1/3] (job2sh.rb) add on_fail to job.sh
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh export_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..6e00b088 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors','on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -357,6 +359,30 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + if !job + job = (@jobx || @job).clone + end + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

3 2

[PATCH lkp-tests] add testcase for sysbench-threads-git
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] old version package needs to debug by new testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 tests/sysbench-threads-git diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git new file mode 100755 index 00000000..712b3359 --- /dev/null +++ b/tests/sysbench-threads-git @@ -0,0 +1,17 @@ +#!/bin/sh +# - nr_threads +# - thread_yields +# - thread_locks + +. "$LKP_SRC/lib/reproduce-log.sh" + +: "${nr_threads:=2}" +: "${thread_yields:=100}" +: "${thread_locks:=2}" + +args="\ + --threads=$nr_threads\ + --thread-yields=$thread_yields\ + --thread-locks=$thread_locks\ +" +log_cmd sysbench --test=threads $args run -- 2.23.0

2 1

[PATCH lkp-tests] add PKGBUILD for sysbench-threads-git
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] Adapting sysbench-threads needs to old version to user. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..2779c294 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,21 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=0.4.8 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="http://github.com/dallasmarlow/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() { + cd "$srcdir/$pkgname" + make DESTDIR=$pkgdir install +} -- 2.23.0

1 0

[PATCH lkp-tests] add PKGBUILD for sysbech-threads-git
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] Adapting sysbench-threads needs to old version to user. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- pkg/sysbench-threads-git/PKGBUILD | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 pkg/sysbench-threads-git/PKGBUILD diff --git a/pkg/sysbench-threads-git/PKGBUILD b/pkg/sysbench-threads-git/PKGBUILD new file mode 100644 index 00000000..bf47d5a1 --- /dev/null +++ b/pkg/sysbench-threads-git/PKGBUILD @@ -0,0 +1,21 @@ +pkgname=sysbench-threads-git +pkgver=git +pkgrel=0.4.8 +pkgdesc="Benchmark tool for evaluating OS parameters that are important for a system running a database under intensive load." +url="https://github.com/akopytov/sysbench.git" +arch=('x86_64' 'i386' 'aarch64') +license=('GPL') +source=("http://github.com/dallasmarlow/sysbench.git") +md5sums=('SKIP') + +build() { + cd "$srcdir/$pkgname" + ./autogen.sh + ./configure --prefix=/usr --without-gcc-arch --without-mysql + make +} + +package() { + cd "$srcdir/$pkgname" + make DESTDIR=$pkgdir install +} -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/3] submit on_fail to the scheduler
by Hu Xuejiao 04 Nov '20

04 Nov '20

[why] When job2sh.rb defined sh_on_fail, it needs to be invoked by the scheduler. Finally, the task is in the waiting state when the task is failed. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..0bb41f2d 100755 --- a/sbin/submit +++ b/sbin/submit @@ -115,12 +115,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_fail sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

1 0