November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v9 compass-ci 2/3] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 21 Nov '20

21 Nov '20

[why] Avoid to execute some unnecessary operations and improve performance. [how] Move the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ff4ea31..2065039 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

1 0

[PATCH v6 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 21 Nov '20

21 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..afe5d664 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +$stdin.each_line do |line| + next unless line =~ /^\s+total time:\s+([0-9.]+)/ + + puts "total time: #{$1.to_f}s" +end -- 2.23.0

2 1

[PATCH v2 lkp-tests] stats/lmbench3: add new parsing parameters for performance test
by Bai Jing 21 Nov '20

21 Nov '20

[input] Memory read bandwidth 0.000512 16353.73 0.001024 16577.89 0.002048 16752.28 [output] L1_$: 1.539 L2_$: 0.539 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/lmbench3 | 62 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 2 deletions(-) diff --git a/stats/lmbench3 b/stats/lmbench3 index 211c3abc..750e8bfc 100755 --- a/stats/lmbench3 +++ b/stats/lmbench3 @@ -21,6 +21,27 @@ def largest_bandwidth bandwidth end +def designated_file_size + designated_hash = {} + result_str = '' + + $stdin.each_line do |line| + result_str += line + break if line =~ /(^\s|^\t)/ + end + + result_str.split("\n").each do |e| + designated_hash['L1_$'] = e.split[1] if e =~ /0.01758/ + designated_hash['Rand_mem'] = e.split[1] if e =~ /512.0000/ + designated_hash['Mmap_Latency'] = e.split[1] if e =~ /536.870912/ + designated_hash['L2_$'] = e.split[1] if e =~ /0.05078/ + end + + designated_hash +end + +null_io_array = [] + while (line = STDIN.gets) line = line.resolve_invalid_bytes case line @@ -32,14 +53,15 @@ while (line = STDIN.gets) # Simple fstat: 0.3517 microseconds # Simple open/close: 1.3696 microseconds when /^Simple (\S+): (\d+.\d+) microseconds$/ + null_io_array << $2.to_f if $1.to_s == 'read' || $1.to_s == 'write' puts "syscall.#{$1}.latency.us: #{$2}" # Extract select test result. # Select on 100 fd's: 1.2293 microseconds # Select on 100 tcp fd's: 5.0377 microseconds - when /^Select on 100 fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 fd's: (\d+.\d+) microseconds$/ puts "Select.100fd.latency.us: #{$1}" - when /^Select on 100 tcp fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 tcp fd's: (\d+.\d+) microseconds$/ puts "Select.100tcp.latency.us: #{$1}" # Extract proc test result. @@ -198,5 +220,41 @@ while (line = STDIN.gets) # ... when /^"Mmap read open2close bandwidth$/ puts "MMAP.read_open2close.bandwidth.MB/sec: #{largest_bandwidth}" + + # "Random Memory load latency" + # 416.00000 92.824 + # 448.00000 92.772 + # 512.00000 92.640 + # ... + when /^Random load latency$/ + result_rand_mem = designated_file_size + puts "Rand_mem: #{result_rand_mem['Rand_mem']}" + + # "Memory load latency" + # 0.00049 1.539 + # 0.00098 1.539 + # 512.00000 74.421 + # ... + when /^Memory load latency$/ + result_rand_mem = designated_file_size + puts "L1_$: #{result_rand_mem['L1_$']}" + puts "L2_$: #{result_rand_mem['L2_$']}" + + # "mappings" + # 67.108864 82 + # 134.217728 151 + # 268.435456 309 + # 536.870912 615 + # ... + when /^"mappings$/ + result_map = designated_file_size + puts "Mmap_Latency: #{result_map['Mmap_Latency']}" end end + +num = 0 +null_io_array.each do |v| + num += v +end +null_num = format('%.4f', num / 2).to_f +puts "null_io: #{null_num}" -- 2.23.0

3 6

git count
by Wu Fengguang 20 Nov '20

20 Nov '20

Shengde is our code star, thanks Shengde! wfg@crystal ~/todo% bin/git-count 21 Luan Shengde 13 Sun Yukui 10 Li Yuanchao 8 Hu Xuejiao 8 Xiao Shenwei 8 Zhang Yale 8 Yu Chuan 8 Lin Jiaxin 7 Bai Jing 7 Liu Yinsi 6 Liu Shaofei 6 Zhang Yu 5 Cao Xueliang 5 Cui Yili 5 Wang Yong 4 Xu Xijian 4 Wei Jihui 4 Lu Kaiyi 3 Wu Zhende 2 Lu Weitao 2 Wang Chenglong 2 Zhang Yuhang 1 Wu Fengguang 1 Ren Wen 1 Li Ping

1 0

review star
by Wu Fengguang 20 Nov '20

20 Nov '20

Xueliang is our review star, thanks Xueliang! wfg@crystal /c/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 51 44 107 34 Wu Fengguang 118 37 48 24 Cao Xueliang 99 35 50 20 Wei Jihui 29 18 22 14 Xu Xijian 33 6 18 9 Xiao Shenwei 75 23 54 9 Luan Shengde 47 19 21 7 Liu Shaofei 61 14 13 7 Lu Kaiyi 31 13 17 6 Zhang Yu 16 6 11 6 Zhang Yuhang 52 9 11 5 Bai Jing 18 5 10 4 Wang Yong 34 6 11 4 Wang Chenglong 22 14 17 4 Cui Yili 56 7 10 4 Lin Jiaxin 13 3 5 4 Lu Weitao 19 6 8 3 Wu Zhende 61 7 9 3 Hu Xuejiao 47 3 8 2 Yu Chuan 31 7 9 1 Liu Yinsi 8 0 0 0 Li Ping 1 1 1 0 Zhang Dewan 12 2 3 0 Ren Wen 28 9 9 -2 Zhang Yale 23 5 9 -3 Li Yuanchao 41 15 26 -3 Sun Yukui

1 0

help star
by Wu Fengguang 20 Nov '20

20 Nov '20

Yuchuan and Jihui is our help star, thanks Yuchuan and Jihui! wfg@crystal ~/todo% bin/help-star people/*/credit/2020-11-20.md 18.8 Wu Fengguang 17.2 Yu Chuan 17.2 Wei Jihui 16.3 Cao Xueliang 13.7 Xiao Shenwei 12.0 Wu Zhende 11.6 Xu Xijian 10.1 Wang Chenglong 9.8 Liu Shaofei 9.7 Lu Weitao 9.5 Hu Xuejiao 9.0 Luan Shengde 8.5 Cui Yili 8.4 Zhang Yuhang 7.9 Liu Yinsi 7.2 Wang Yong 7.0 Zhang Dewan 6.9 Zhang Yu 6.4 Lin Jiaxin 6.0 Ren Wen 5.8 Li Yuanchao 5.0 Bai Jing 4.3 Du Kaitian 4.2 Zhang Yale 3.3 Lu Kaiyi 2.0 Sun Yukui 1.4 Tong Qunfeng 0.0 Wang Shuai 0.0 Li Ping 0.0 Cui Xucui

1 0

[PATCH compass-ci] doc: add Linux article
by Wu Fengguang 20 Nov '20

20 Nov '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/learning-resources.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/learning-resources.md b/doc/learning-resources.md index 62716cf..e631f4a 100644 --- a/doc/learning-resources.md +++ b/doc/learning-resources.md @@ -55,6 +55,9 @@ https://github.com/DocSpring/ruby_crystal_codemod shell ===== +Linux 的概念与体系 +https://www.cnblogs.com/vamei/archive/2012/10/10/2718229.html + man bash https://devhints.io/bash https://github.com/denysdovhan/bash-handbook/blob/master/translations/zh-CN… -- 2.23.0

1 1

[PATCH v2 compass-ci] scheduler: fix: alter processes of request_state
by Ren Wen 20 Nov '20

20 Nov '20

There're 2 "finished" states before, one is *daemon* request, the other is *test* request. Now change *daemon*'s to "started", means daemon started. This state will save to redis key "sched/cluster_state". Why move "finished" to another "when condition"? When all nodes in cluster do *run_test*, no *start_daemon*, the node will restart immediately after its tests finished. Now need to wait other nodes, so move "finished" to another "when condition" to wait all nodes finished. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/request_cluster_state.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index ac6cb8e..5bbbad1 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -13,13 +13,14 @@ class Sched cluster_state = "" states = {"abort" => "abort", + "started" => "start", "finished" => "finish", "failed" => "abort", "wait_ready" => "ready", "wait_finish" => "finish"} case request_state - when "abort", "finished", "failed" + when "abort", "started", "failed" # update node state only update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" @@ -30,7 +31,7 @@ class Sched } return cluster_state - when "wait_finish" + when "wait_finish", "finished" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: alter processes of request_state
by Ren Wen 20 Nov '20

20 Nov '20

There're 2 "finished" states before, one is *daemon* request, the other is *test* request. Now change *daemon*'s to "started", means daemon started. This state will save to redis key "sched/cluster_state". Why move "finished" to another "when condition"? When all nodes in cluster do *run_test*, no *start_daemon*, the node will restart immediately after its tests finished. Now need to wait other nodes, so move "finished" to another "when condition" to wait all nodes finished. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/request_cluster_state.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index ac6cb8e..fffa4cf 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -13,13 +13,14 @@ class Sched cluster_state = "" states = {"abort" => "abort", + "started" => "started", "finished" => "finish", "failed" => "abort", "wait_ready" => "ready", "wait_finish" => "finish"} case request_state - when "abort", "finished", "failed" + when "abort", "started", "failed" # update node state only update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" @@ -30,7 +31,7 @@ class Sched } return cluster_state - when "wait_finish" + when "wait_finish", "finished" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) -- 2.23.0

2 3

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: Support more fields in "jobs" index
by Zhang Yuhang 20 Nov '20

20 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..018f57e 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,10 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' -d ' + {"index.mapping.total_fields.limit": 10000} + ' fi fi -- 2.23.0

3 3