- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: Support more fields in "jobs" index
by Zhang Yuhang 20 Nov '20

20 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..018f57e 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,10 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' -d ' + {"index.mapping.total_fields.limit": 10000} + ' fi fi -- 2.23.0

3 3

[PATCH compass-ci] sbin/es-jobs-mapping.sh: set "index.mapping.total_fields.limit: 10000" when create "jobs" mapping.
by Zhang Yuhang 20 Nov '20

20 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..018f57e 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,10 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' -d ' + {"index.mapping.total_fields.limit": 10000} + ' fi fi -- 2.23.0

3 4

[PATCH compass-ci] container/ssh-r: fix address already in use
by Liu Yinsi 20 Nov '20

20 Nov '20

[why] when execute ssh-r/start, error message: docker: Error response from daemon: driver failed programming external connectivity on endpoint ssh_r (55638ba16aa1264dff0fdc1badc93176eec07d2b848d4a8d418d07011d84313d): Error starting userland proxy: listen tcp 0.0.0.0:51998: bind: address already in use. [how] because ssh-r will bind ports 50000-51999, set ip_local_port_range to prevent occupation. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 523438c..c327707 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -12,6 +12,8 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 net.ipv6.bindv6only=1 vm.max_map_count=262144 +net.ipv4.ip_local_port_range = 1024 48999 + EOF sysctl -p -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/0-package: fix failed to start ssh-r
by Liu Yinsi 20 Nov '20

20 Nov '20

[why] when execute ssh-r/start, error message: docker: Error response from daemon: driver failed programming external connectivity on endpoint ssh_r (55638ba16aa1264dff0fdc1badc93176eec07d2b848d4a8d418d07011d84313d): Error starting userland proxy: listen tcp 0.0.0.0:51998: bind: address already in use. [how] because ssh-r will bind ports 50000-51999, set ip_local_port_range to prevent occupation. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 523438c..5eb9f76 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -12,6 +12,7 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 net.ipv6.bindv6only=1 vm.max_map_count=262144 +net.ipv4.ip_local_port_range = 1024 48999 EOF sysctl -p -- 2.23.0

1 0

[PATCH v5 compass-ci] lib/log.rb: add new class for json log
by Wu Zhende 20 Nov '20

20 Nov '20

[Why] The built-in log system prints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example1] use: log.info("test") output: {"level":"INFO","level_num":1,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"test"} [Example2] use: log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"", "job_id":"1","job_state":"submit","caller":["/usr/share/ruby/logger.rb:545:in `error'","./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/log.rb | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..aad00ca --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' +require 'json' + +# print logs in JSON format +class Log < Logger + LEVEL_INFO = { + 'DEBUG' => 0, + 'INFO' => 1, + 'WARN' => 2, + 'ERROR' => 3, + 'FATAL' => 4, + 'UNKNOWN' => 5 + }.freeze + + FORMATTER = proc { |severity, datetime, progname, msg| + level_num = LEVEL_INFO[severity] + begin + message = JSON.parse(msg) + rescue JSON::ParserError + message = { 'message' => msg } + end + h = { + 'level' => severity.to_s, + 'level_num' => level_num, + 'datetime' => datetime, + 'progname' => progname, + 'message' => '' + } + h.merge!(message) + h.merge!({ 'caller' => caller }) if level_num >= 2 + h.to_json + } + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

2 2

[PATCH lkp-tests 1/2] submit: support add job to personal queue
by Xiao Shenwei 20 Nov '20

20 Nov '20

usage: submit --my-queue testbox=vm-2p8g jobs/borrow-1h.yaml if use this parameter, your job will be add to the queue which is $tbox_group~$USER Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/submit | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/submit b/sbin/submit index 9bba1018..53ea5bf0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -16,6 +16,7 @@ opt_output_dir = nil opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} +personal_job = false actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -54,6 +55,11 @@ options = OptionParser.new do |opts| ARGV.shift end end + + opts.on('--my-queue', 'add to personal queue') do + personal_job = true + end + end options.parse!(ARGV) @@ -93,6 +99,11 @@ def find_jobfile(jobfile) exit 1 end +def add_personal_queue(job) + queue = "#{job['tbox_group']}~#{ENV['USER']}" + job['queue'] = queue +end + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -134,6 +145,7 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) + add_personal_queue(job_hash) if personal_job # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

1 0

[PATCH v3 lkp-tests] stats/lmbench3: add new parsing parameters for performance test
by Bai Jing 20 Nov '20

20 Nov '20

[input] Memory read bandwidth 0.000512 16353.73 0.001024 16577.89 0.002048 16752.28 [output] L1_$: 1.539 L2_$: 0.539 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/lmbench3 | 62 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 2 deletions(-) diff --git a/stats/lmbench3 b/stats/lmbench3 index 211c3abc..ba36dbb6 100755 --- a/stats/lmbench3 +++ b/stats/lmbench3 @@ -21,6 +21,27 @@ def largest_bandwidth bandwidth end +def designated_file_size + designated_hash = {} + result_str = '' + + $stdin.each_line do |line| + result_str += line + break if line =~ /(^\s|^\t)/ + end + + result_str.split("\n").each do |e| + designated_hash['L1_$'] = e.split[1] if e =~ /0.01758/ + designated_hash['Rand_mem'] = e.split[1] if e =~ /512.0000/ + designated_hash['Mmap_Latency'] = e.split[1] if e =~ /536.870912/ + designated_hash['L2_$'] = e.split[1] if e =~ /0.05078/ + end + + designated_hash +end + +null_io_array = [] + while (line = STDIN.gets) line = line.resolve_invalid_bytes case line @@ -32,14 +53,15 @@ while (line = STDIN.gets) # Simple fstat: 0.3517 microseconds # Simple open/close: 1.3696 microseconds when /^Simple (\S+): (\d+.\d+) microseconds$/ + null_io_array << $2.to_f if $1.to_s == 'read' || $1.to_s == 'write' puts "syscall.#{$1}.latency.us: #{$2}" # Extract select test result. # Select on 100 fd's: 1.2293 microseconds # Select on 100 tcp fd's: 5.0377 microseconds - when /^Select on 100 fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 fd's: (\d+.\d+) microseconds$/ puts "Select.100fd.latency.us: #{$1}" - when /^Select on 100 tcp fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 tcp fd's: (\d+.\d+) microseconds$/ puts "Select.100tcp.latency.us: #{$1}" # Extract proc test result. @@ -198,5 +220,41 @@ while (line = STDIN.gets) # ... when /^"Mmap read open2close bandwidth$/ puts "MMAP.read_open2close.bandwidth.MB/sec: #{largest_bandwidth}" + + # Random load latency + # 416.00000 92.824 + # 448.00000 92.772 + # 512.00000 92.640 + # ... + when /^Random load latency$/ + result_rand_mem = designated_file_size + puts "Rand_mem: #{result_rand_mem['Rand_mem']}" + + # Memory load latency + # 0.00049 1.539 + # 0.00098 1.539 + # 512.00000 74.421 + # ... + when /^Memory load latency$/ + result_rand_mem = designated_file_size + puts "L1_$: #{result_rand_mem['L1_$']}" + puts "L2_$: #{result_rand_mem['L2_$']}" + + # "mappings + # 67.108864 82 + # 134.217728 151 + # 268.435456 309 + # 536.870912 615 + # ... + when /^"mappings$/ + result_map = designated_file_size + puts "Mmap_Latency: #{result_map['Mmap_Latency']}" end end + +num = 0 +null_io_array.each do |v| + num += v +end +null_num = format('%.4f', num / 2).to_f +puts "null_io: #{null_num}" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] git_mirror: fix that too few repos in upstream-repos cause memory leak
by Li Yuanchao 20 Nov '20

20 Nov '20

[why] when there is only a few repo files in upstream-repos, that means the same repo will be called very frequently. As the main loop is too fast, it seems the new flow would be called before the last flow of the same repo end completely. That cause a problem. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 72ca328..5b6b889 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,6 +203,7 @@ class MirrorMain push_git_queue handle_feedback Signal.trap(:SIGCHLD, 'SIG_IGN') + sleep(0.1) end end end -- 2.23.0

2 2

[PATCH v2 compass-ci 1/3] git_mirror: fix that README.md in upstream-repos cause a load error
by Li Yuanchao 20 Nov '20

20 Nov '20

[why] As README.md or other docs in upstream-repos, and they are not yaml file. But git-mirror load all files in upstream-repos as yaml file, so there comes out a load error. Now files whose dirname is 'xxx/upstream-repos' will not be loaded. [errmsg] in `load_repo_file': undefined method `[]=' for x:Integer (NoMethodError) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9648063..72ca328 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -135,6 +135,8 @@ class MirrorMain traverse_repodir("#{repodir}/#{entry}") end else + return if File.dirname(repodir) == REPO_DIR + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") fork_name = File.basename(repodir) load_repo_file(repodir, project, fork_name) -- 2.23.0

2 2

[PATCH lkp-tests] daemon/multi-qemu: add a default value for the queues field
by Zhang Yu 20 Nov '20

20 Nov '20

[why] For multi-qemu.yaml, if the queues field is not set, a default value is required. [how] If the multi-qemu.yaml not set queues field, there are two queues by default: "$tbox_group.$HOSTNAME, $tbox_group.$(arch)". Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- daemon/multi-qemu | 1 + 1 file changed, 1 insertion(+) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index 8d9dd38d..7cc625f4 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -6,6 +6,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +: ${queues:="$tbox_group.$HOSTNAME, $tbox_group.$(arch)"} multi_qemu() { export CCI_SRC=/c/compass-ci -- 2.23.0

3 3