- Compass-ci - mailweb.openeuler.org

help star
by Wu Fengguang 20 Nov '20

20 Nov '20

Yuchuan and Jihui is our help star, thanks Yuchuan and Jihui! wfg@crystal ~/todo% bin/help-star people/*/credit/2020-11-20.md 18.8 Wu Fengguang 17.2 Yu Chuan 17.2 Wei Jihui 16.3 Cao Xueliang 13.7 Xiao Shenwei 12.0 Wu Zhende 11.6 Xu Xijian 10.1 Wang Chenglong 9.8 Liu Shaofei 9.7 Lu Weitao 9.5 Hu Xuejiao 9.0 Luan Shengde 8.5 Cui Yili 8.4 Zhang Yuhang 7.9 Liu Yinsi 7.2 Wang Yong 7.0 Zhang Dewan 6.9 Zhang Yu 6.4 Lin Jiaxin 6.0 Ren Wen 5.8 Li Yuanchao 5.0 Bai Jing 4.3 Du Kaitian 4.2 Zhang Yale 3.3 Lu Kaiyi 2.0 Sun Yukui 1.4 Tong Qunfeng 0.0 Wang Shuai 0.0 Li Ping 0.0 Cui Xucui

1 0

[PATCH compass-ci] doc: add Linux article
by Wu Fengguang 20 Nov '20

20 Nov '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/learning-resources.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/learning-resources.md b/doc/learning-resources.md index 62716cf..e631f4a 100644 --- a/doc/learning-resources.md +++ b/doc/learning-resources.md @@ -55,6 +55,9 @@ https://github.com/DocSpring/ruby_crystal_codemod shell ===== +Linux 的概念与体系 +https://www.cnblogs.com/vamei/archive/2012/10/10/2718229.html + man bash https://devhints.io/bash https://github.com/denysdovhan/bash-handbook/blob/master/translations/zh-CN… -- 2.23.0

1 1

[PATCH v2 compass-ci] scheduler: fix: alter processes of request_state
by Ren Wen 20 Nov '20

20 Nov '20

There're 2 "finished" states before, one is *daemon* request, the other is *test* request. Now change *daemon*'s to "started", means daemon started. This state will save to redis key "sched/cluster_state". Why move "finished" to another "when condition"? When all nodes in cluster do *run_test*, no *start_daemon*, the node will restart immediately after its tests finished. Now need to wait other nodes, so move "finished" to another "when condition" to wait all nodes finished. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/request_cluster_state.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index ac6cb8e..5bbbad1 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -13,13 +13,14 @@ class Sched cluster_state = "" states = {"abort" => "abort", + "started" => "start", "finished" => "finish", "failed" => "abort", "wait_ready" => "ready", "wait_finish" => "finish"} case request_state - when "abort", "finished", "failed" + when "abort", "started", "failed" # update node state only update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" @@ -30,7 +31,7 @@ class Sched } return cluster_state - when "wait_finish" + when "wait_finish", "finished" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: alter processes of request_state
by Ren Wen 20 Nov '20

20 Nov '20

There're 2 "finished" states before, one is *daemon* request, the other is *test* request. Now change *daemon*'s to "started", means daemon started. This state will save to redis key "sched/cluster_state". Why move "finished" to another "when condition"? When all nodes in cluster do *run_test*, no *start_daemon*, the node will restart immediately after its tests finished. Now need to wait other nodes, so move "finished" to another "when condition" to wait all nodes finished. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/request_cluster_state.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index ac6cb8e..fffa4cf 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -13,13 +13,14 @@ class Sched cluster_state = "" states = {"abort" => "abort", + "started" => "started", "finished" => "finish", "failed" => "abort", "wait_ready" => "ready", "wait_finish" => "finish"} case request_state - when "abort", "finished", "failed" + when "abort", "started", "failed" # update node state only update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" @@ -30,7 +31,7 @@ class Sched } return cluster_state - when "wait_finish" + when "wait_finish", "finished" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) -- 2.23.0

2 3

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: Support more fields in "jobs" index
by Zhang Yuhang 20 Nov '20

20 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..018f57e 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,10 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' -d ' + {"index.mapping.total_fields.limit": 10000} + ' fi fi -- 2.23.0

3 3

[PATCH compass-ci] sbin/es-jobs-mapping.sh: set "index.mapping.total_fields.limit: 10000" when create "jobs" mapping.
by Zhang Yuhang 20 Nov '20

20 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..018f57e 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,10 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' -d ' + {"index.mapping.total_fields.limit": 10000} + ' fi fi -- 2.23.0

3 4

[PATCH compass-ci] container/ssh-r: fix address already in use
by Liu Yinsi 20 Nov '20

20 Nov '20

[why] when execute ssh-r/start, error message: docker: Error response from daemon: driver failed programming external connectivity on endpoint ssh_r (55638ba16aa1264dff0fdc1badc93176eec07d2b848d4a8d418d07011d84313d): Error starting userland proxy: listen tcp 0.0.0.0:51998: bind: address already in use. [how] because ssh-r will bind ports 50000-51999, set ip_local_port_range to prevent occupation. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 523438c..c327707 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -12,6 +12,8 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 net.ipv6.bindv6only=1 vm.max_map_count=262144 +net.ipv4.ip_local_port_range = 1024 48999 + EOF sysctl -p -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/0-package: fix failed to start ssh-r
by Liu Yinsi 20 Nov '20

20 Nov '20

[why] when execute ssh-r/start, error message: docker: Error response from daemon: driver failed programming external connectivity on endpoint ssh_r (55638ba16aa1264dff0fdc1badc93176eec07d2b848d4a8d418d07011d84313d): Error starting userland proxy: listen tcp 0.0.0.0:51998: bind: address already in use. [how] because ssh-r will bind ports 50000-51999, set ip_local_port_range to prevent occupation. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 523438c..5eb9f76 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -12,6 +12,7 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 net.ipv6.bindv6only=1 vm.max_map_count=262144 +net.ipv4.ip_local_port_range = 1024 48999 EOF sysctl -p -- 2.23.0

1 0

[PATCH v5 compass-ci] lib/log.rb: add new class for json log
by Wu Zhende 20 Nov '20

20 Nov '20

[Why] The built-in log system prints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example1] use: log.info("test") output: {"level":"INFO","level_num":1,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"test"} [Example2] use: log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"", "job_id":"1","job_state":"submit","caller":["/usr/share/ruby/logger.rb:545:in `error'","./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/log.rb | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..aad00ca --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' +require 'json' + +# print logs in JSON format +class Log < Logger + LEVEL_INFO = { + 'DEBUG' => 0, + 'INFO' => 1, + 'WARN' => 2, + 'ERROR' => 3, + 'FATAL' => 4, + 'UNKNOWN' => 5 + }.freeze + + FORMATTER = proc { |severity, datetime, progname, msg| + level_num = LEVEL_INFO[severity] + begin + message = JSON.parse(msg) + rescue JSON::ParserError + message = { 'message' => msg } + end + h = { + 'level' => severity.to_s, + 'level_num' => level_num, + 'datetime' => datetime, + 'progname' => progname, + 'message' => '' + } + h.merge!(message) + h.merge!({ 'caller' => caller }) if level_num >= 2 + h.to_json + } + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

2 2

[PATCH lkp-tests 1/2] submit: support add job to personal queue
by Xiao Shenwei 20 Nov '20

20 Nov '20

usage: submit --my-queue testbox=vm-2p8g jobs/borrow-1h.yaml if use this parameter, your job will be add to the queue which is $tbox_group~$USER Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/submit | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/submit b/sbin/submit index 9bba1018..53ea5bf0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -16,6 +16,7 @@ opt_output_dir = nil opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} +personal_job = false actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -54,6 +55,11 @@ options = OptionParser.new do |opts| ARGV.shift end end + + opts.on('--my-queue', 'add to personal queue') do + personal_job = true + end + end options.parse!(ARGV) @@ -93,6 +99,11 @@ def find_jobfile(jobfile) exit 1 end +def add_personal_queue(job) + queue = "#{job['tbox_group']}~#{ENV['USER']}" + job['queue'] = queue +end + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -134,6 +145,7 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) + add_personal_queue(job_hash) if personal_job # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

1 0