- Compass-ci - mailweb.openeuler.org

review star
by Wu Fengguang 20 Nov '20

20 Nov '20

Xueliang is our review star, thanks Xueliang! wfg@crystal /c/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 51 44 107 34 Wu Fengguang 118 37 48 24 Cao Xueliang 99 35 50 20 Wei Jihui 29 18 22 14 Xu Xijian 33 6 18 9 Xiao Shenwei 75 23 54 9 Luan Shengde 47 19 21 7 Liu Shaofei 61 14 13 7 Lu Kaiyi 31 13 17 6 Zhang Yu 16 6 11 6 Zhang Yuhang 52 9 11 5 Bai Jing 18 5 10 4 Wang Yong 34 6 11 4 Wang Chenglong 22 14 17 4 Cui Yili 56 7 10 4 Lin Jiaxin 13 3 5 4 Lu Weitao 19 6 8 3 Wu Zhende 61 7 9 3 Hu Xuejiao 47 3 8 2 Yu Chuan 31 7 9 1 Liu Yinsi 8 0 0 0 Li Ping 1 1 1 0 Zhang Dewan 12 2 3 0 Ren Wen 28 9 9 -2 Zhang Yale 23 5 9 -3 Li Yuanchao 41 15 26 -3 Sun Yukui

1 0

help star
by Wu Fengguang 20 Nov '20

20 Nov '20

Yuchuan and Jihui is our help star, thanks Yuchuan and Jihui! wfg@crystal ~/todo% bin/help-star people/*/credit/2020-11-20.md 18.8 Wu Fengguang 17.2 Yu Chuan 17.2 Wei Jihui 16.3 Cao Xueliang 13.7 Xiao Shenwei 12.0 Wu Zhende 11.6 Xu Xijian 10.1 Wang Chenglong 9.8 Liu Shaofei 9.7 Lu Weitao 9.5 Hu Xuejiao 9.0 Luan Shengde 8.5 Cui Yili 8.4 Zhang Yuhang 7.9 Liu Yinsi 7.2 Wang Yong 7.0 Zhang Dewan 6.9 Zhang Yu 6.4 Lin Jiaxin 6.0 Ren Wen 5.8 Li Yuanchao 5.0 Bai Jing 4.3 Du Kaitian 4.2 Zhang Yale 3.3 Lu Kaiyi 2.0 Sun Yukui 1.4 Tong Qunfeng 0.0 Wang Shuai 0.0 Li Ping 0.0 Cui Xucui

1 0

[PATCH compass-ci] doc: add Linux article
by Wu Fengguang 20 Nov '20

20 Nov '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/learning-resources.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/learning-resources.md b/doc/learning-resources.md index 62716cf..e631f4a 100644 --- a/doc/learning-resources.md +++ b/doc/learning-resources.md @@ -55,6 +55,9 @@ https://github.com/DocSpring/ruby_crystal_codemod shell ===== +Linux 的概念与体系 +https://www.cnblogs.com/vamei/archive/2012/10/10/2718229.html + man bash https://devhints.io/bash https://github.com/denysdovhan/bash-handbook/blob/master/translations/zh-CN… -- 2.23.0

1 1

[PATCH v2 compass-ci] scheduler: fix: alter processes of request_state
by Ren Wen 20 Nov '20

20 Nov '20

There're 2 "finished" states before, one is *daemon* request, the other is *test* request. Now change *daemon*'s to "started", means daemon started. This state will save to redis key "sched/cluster_state". Why move "finished" to another "when condition"? When all nodes in cluster do *run_test*, no *start_daemon*, the node will restart immediately after its tests finished. Now need to wait other nodes, so move "finished" to another "when condition" to wait all nodes finished. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/request_cluster_state.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index ac6cb8e..5bbbad1 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -13,13 +13,14 @@ class Sched cluster_state = "" states = {"abort" => "abort", + "started" => "start", "finished" => "finish", "failed" => "abort", "wait_ready" => "ready", "wait_finish" => "finish"} case request_state - when "abort", "finished", "failed" + when "abort", "started", "failed" # update node state only update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" @@ -30,7 +31,7 @@ class Sched } return cluster_state - when "wait_finish" + when "wait_finish", "finished" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: alter processes of request_state
by Ren Wen 20 Nov '20

20 Nov '20

There're 2 "finished" states before, one is *daemon* request, the other is *test* request. Now change *daemon*'s to "started", means daemon started. This state will save to redis key "sched/cluster_state". Why move "finished" to another "when condition"? When all nodes in cluster do *run_test*, no *start_daemon*, the node will restart immediately after its tests finished. Now need to wait other nodes, so move "finished" to another "when condition" to wait all nodes finished. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/request_cluster_state.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index ac6cb8e..fffa4cf 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -13,13 +13,14 @@ class Sched cluster_state = "" states = {"abort" => "abort", + "started" => "started", "finished" => "finish", "failed" => "abort", "wait_ready" => "ready", "wait_finish" => "finish"} case request_state - when "abort", "finished", "failed" + when "abort", "started", "failed" # update node state only update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" @@ -30,7 +31,7 @@ class Sched } return cluster_state - when "wait_finish" + when "wait_finish", "finished" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) -- 2.23.0

2 3

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: Support more fields in "jobs" index
by Zhang Yuhang 20 Nov '20

20 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..018f57e 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,10 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' -d ' + {"index.mapping.total_fields.limit": 10000} + ' fi fi -- 2.23.0

3 3

[PATCH compass-ci] sbin/es-jobs-mapping.sh: set "index.mapping.total_fields.limit: 10000" when create "jobs" mapping.
by Zhang Yuhang 20 Nov '20

20 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..018f57e 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,10 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' -d ' + {"index.mapping.total_fields.limit": 10000} + ' fi fi -- 2.23.0

3 4

[PATCH compass-ci] container/ssh-r: fix address already in use
by Liu Yinsi 20 Nov '20

20 Nov '20

[why] when execute ssh-r/start, error message: docker: Error response from daemon: driver failed programming external connectivity on endpoint ssh_r (55638ba16aa1264dff0fdc1badc93176eec07d2b848d4a8d418d07011d84313d): Error starting userland proxy: listen tcp 0.0.0.0:51998: bind: address already in use. [how] because ssh-r will bind ports 50000-51999, set ip_local_port_range to prevent occupation. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 523438c..c327707 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -12,6 +12,8 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 net.ipv6.bindv6only=1 vm.max_map_count=262144 +net.ipv4.ip_local_port_range = 1024 48999 + EOF sysctl -p -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/0-package: fix failed to start ssh-r
by Liu Yinsi 20 Nov '20

20 Nov '20

[why] when execute ssh-r/start, error message: docker: Error response from daemon: driver failed programming external connectivity on endpoint ssh_r (55638ba16aa1264dff0fdc1badc93176eec07d2b848d4a8d418d07011d84313d): Error starting userland proxy: listen tcp 0.0.0.0:51998: bind: address already in use. [how] because ssh-r will bind ports 50000-51999, set ip_local_port_range to prevent occupation. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 523438c..5eb9f76 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -12,6 +12,7 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 net.ipv6.bindv6only=1 vm.max_map_count=262144 +net.ipv4.ip_local_port_range = 1024 48999 EOF sysctl -p -- 2.23.0

1 0

[PATCH v5 compass-ci] lib/log.rb: add new class for json log
by Wu Zhende 20 Nov '20

20 Nov '20

[Why] The built-in log system prints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example1] use: log.info("test") output: {"level":"INFO","level_num":1,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"test"} [Example2] use: log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"", "job_id":"1","job_state":"submit","caller":["/usr/share/ruby/logger.rb:545:in `error'","./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/log.rb | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..aad00ca --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' +require 'json' + +# print logs in JSON format +class Log < Logger + LEVEL_INFO = { + 'DEBUG' => 0, + 'INFO' => 1, + 'WARN' => 2, + 'ERROR' => 3, + 'FATAL' => 4, + 'UNKNOWN' => 5 + }.freeze + + FORMATTER = proc { |severity, datetime, progname, msg| + level_num = LEVEL_INFO[severity] + begin + message = JSON.parse(msg) + rescue JSON::ParserError + message = { 'message' => msg } + end + h = { + 'level' => severity.to_s, + 'level_num' => level_num, + 'datetime' => datetime, + 'progname' => progname, + 'message' => '' + } + h.merge!(message) + h.merge!({ 'caller' => caller }) if level_num >= 2 + h.to_json + } + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

2 2