December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] answerback-email: add execute tool limit
by Luan Shengde 03 Dec '20

03 Dec '20

limit the scope of executing answerback-email to 'z9' [why] when call es_client to store the account data, the data will be stored in the es installed at the localhost. It will cause validation to fail when submit a job if the data is not at the server 'z9'. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 31 +++++++++++++++++--- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 825d815..4c34e8f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -187,11 +187,14 @@ def apply_account(my_info, conf_info) end def check_my_email(my_info) + return true if my_info['my_email'] + message = "No email address specified\n" message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' + puts message - raise message if my_info['my_email'].nil? + return false end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) @@ -219,13 +222,32 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] end +def check_server + return true if ENV['HOSTNAME'] == 'crystal.ci' + + message = 'please run the tool on z9 server' + + puts message + return false +end + +def check_my_name_exist(my_info) + return true if my_info['my_name'] + + message = 'No my_name found, please use -n to add one' + puts message + + return false +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) - check_my_email(my_info) + return unless check_server + return unless check_my_email(my_info) + my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) - message = 'No my_name found, please use -n to add one' - raise message if my_info['my_name'].nil? + return unless check_my_name_exist(my_info) account_info = apply_account(my_info, conf_info) build_my_info_from_account_info(my_info, account_info, conf_info) @@ -241,6 +263,7 @@ def send_mail(my_info, account_info, conf_info) else build_message(my_info['my_email'], account_info) end + %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] src/lib/web_backend.rb: add methods to search testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

Query all testboxes from es, and return a non repetitive list Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index e548d61..014600a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -435,3 +435,29 @@ def template_body(request_body) formatter = FormatEchartData.new(compare_results, request_body) formatter.format_for_echart.to_json end + +def search_testboxes + query = { size: 0, aggs: { testboxes: { terms: { size: 10000, field: 'testbox' } } } } + buckets = es_query(query)['aggregations']['testboxes']['buckets'] + testboxes = [] + buckets.each { |tbox_agg| testboxes << tbox_agg['key'] } + return testboxes, testboxes.length +end + +def testboxes_body + testboxes, total = search_testboxes + { + total: total, + testboxes: testboxes + }.to_json +end + +def query_testboxes + begin + body = testboxes_body + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/web_backend: add a interface /get_testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

It would return a testbox list to web request Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index d68e9f2..11ba604 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -103,3 +103,10 @@ post '/compare_template' do request.body.rewind # in case someone already read it compare_template(request.body.read) end + +# GET /get_testboxes +# Response: +# - { total: total, testboxes[ 'testbox1', 'testbox2', ... ] } +get '/get_testboxes' do + query_testboxes +end -- 2.23.0

1 0

[PATCH v4 compass-ci] scheduler: fix the values diplayed in job.yaml
by Bai Jing 02 Dec '20

02 Dec '20

[original values] initrds_uri: 'initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… [now values] - initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… - initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… Signed-off-by: Bai Jing <799286817(a)qq.com> --- src/lib/job.cr | 8 ++++---- src/scheduler/find_job_boot.cr | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2a811cb..a4345e0 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -413,7 +413,7 @@ class Job end private def get_initrds - temp_initrds = [] of String + temp_initrds = [] of String if "#{os_mount}" == "initramfs" temp_initrds.concat(initramfs_initrds()) @@ -437,13 +437,13 @@ class Job end private def set_initrds_uri - uris = "" + initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| - uris += "initrd #{initrd}\n" + initrds_uri_values << JSON::Any.new("initrd #{initrd}") end - self["initrds_uri"] = uris + @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) end private def set_user_lkp_src diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9805464..3b68b59 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -134,7 +134,7 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" - response += job.initrds_uri + response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" response += job.kernel_uri response += job.kernel_params response += "\nboot\n" -- 2.23.0

1 0

[PATCH v3 compass-ci] scheduler: fix the values diplayed in job.yaml
by Bai Jing 02 Dec '20

02 Dec '20

[original values] initrds_uri: 'initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… [now values] - initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20201022… - initrd http://172.168.131.113:8800/initrd/deps/nfs/debian/aarch64/sid/run-ipconfig… - initrd http://172.17.0.1:8000/os/openeuler/aarch64/20.03/boot/modules-4.19.90-2003… Signed-off-by: Bai Jing <799286817(a)qq.com> --- src/lib/job.cr | 6 ++++-- src/scheduler/find_job_boot.cr | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2a811cb..ae3ba6e 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -413,7 +413,7 @@ class Job end private def get_initrds - temp_initrds = [] of String + temp_initrds = [] of String if "#{os_mount}" == "initramfs" temp_initrds.concat(initramfs_initrds()) @@ -438,12 +438,14 @@ class Job private def set_initrds_uri uris = "" + initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| uris += "initrd #{initrd}\n" + initrds_uri_values << JSON::Any.new("initrd #{initrd}") end - self["initrds_uri"] = uris + @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) end private def set_user_lkp_src diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9805464..3b68b59 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -134,7 +134,7 @@ class Sched private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" - response += job.initrds_uri + response += Array(String).from_json(job.initrds_uri).join("\n") + "\n" response += job.kernel_uri response += job.kernel_params response += "\nboot\n" -- 2.23.0

2 1

[PATCH v3 lkp-tests] spec/submit: add job yaml for changing multi-args
by Bai Jing 02 Dec '20

02 Dec '20

[test cmd] rspec $LKP_SRC/spec/submit_spec.rb [output] /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml /lkp-tests/spec/submit/matrix/unixbench-spec.yaml => /lkp-tests/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrix/unixbench-spec.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/unixbench-spec.yaml index e17322b4..851f0473 100644 --- a/spec/submit/matrix/unixbench-spec.yaml +++ b/spec/submit/matrix/unixbench-spec.yaml @@ -2,8 +2,8 @@ suite: unixbench runtime: 300s -nr_task test: - - 1 whetstone-double - - 3 shell1 +nr_task|test: + - 1|whetstone-double + - 3|shell1 unixbench: -- 2.23.0

1 0

[PATCH v1 compass-ci] sched: optimize regular matching in format_string function
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/pp_params.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr index da8d8f3..03bf0b8 100644 --- a/src/scheduler/pp_params.cr +++ b/src/scheduler/pp_params.cr @@ -50,10 +50,8 @@ class Job private def format_string(original_str) temp = [] of String - original_str.each_char do |char| - if "#{char}" =~ /-|\.|\/|\w/ - temp << "#{char}".gsub("\/", "-") - end + original_str.gsub('/', '-').each_char do |char| + temp << "#{char}" if "#{char}" =~ /\w|-|\./ end return temp.join() -- 2.23.0

1 0

[PATCH compass-ci] sched: improve format string function
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/pp_params.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr index da8d8f3..03bf0b8 100644 --- a/src/scheduler/pp_params.cr +++ b/src/scheduler/pp_params.cr @@ -50,10 +50,8 @@ class Job private def format_string(original_str) temp = [] of String - original_str.each_char do |char| - if "#{char}" =~ /-|\.|\/|\w/ - temp << "#{char}".gsub("\/", "-") - end + original_str.gsub('/', '-').each_char do |char| + temp << "#{char}" if "#{char}" =~ /\w|-|\./ end return temp.join() -- 2.23.0

2 4

[PATCH compass-ci] How to add a machine to compass-ci
by Zhang Dewan 02 Dec '20

02 Dec '20

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- ...00\350\210\254\350\277\207\347\250\213.md" | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 "doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" diff --git "a/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" "b/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" new file mode 100644 index 0000000..3fa402b --- /dev/null +++ "b/doc/manual/\346\267\273\345\212\240\346\234\272\345\231\250\345\210\260compass-ci\347\232\204\344\270\200\350\210\254\350\277\207\347\250\213.md" @@ -0,0 +1,49 @@ + +这个过程可以使用“注册--发现”的模型来理解。 + +## 一台机器成为测试机: +- 需要完成注册 +- 需要能够执行调度器（scheduler）给它的任务 + +## 注册 +### 1. 注册到哪里 +测试机需要注册到z9上面的redis-db. + +x.x.x.x:xxxx> `hgetall "sched/mac2host"` # 登录redis, 这个命令可以查看已经注册的测试机信息 + +### 2. 怎么注册 +使用scheduler服务接口注册，模板： +`curl -X PUT "http://$server_ip:$port/set_host_mac?hostname=$hostname&mac=$mac` + +可以看到，这个调用需要两个参数：mac和hostname + +#### 2.1 接口参数说明 +##### 2.1.1 mac +当前我们的机器都是局域网内的资源，所以没有比使用mac注册更直接、方便的了。 + +###### 2.1.1.1 如何获取mac地址，这里用TaiShan服务器举例： +- 登录bmc，打开信息--系统信息--网络--网卡--第一张网卡--port1，即可查看。 +- 登录bios，进入Advanced--Network Device List，即可查看。 +- 使用一台和机器的bmc同网段的机器，执行以下命令即可查看： +`ipmitool -I lanplus -H $cur_ip -U $iBMC_user -P $iBMC_passwd raw 0x30 0x90 0x01 0x03 0x0${nic_idx} 0x00|awk '{print $5,$6,$7,$8,$9,$10}' |sed 's/\s/\-/g'` +- 给服务器安装os，登录上去，执行命令，即可查看： +`ip link | grep "^2:" -A 1| grep link |awk '{print $2}` + +##### 2.1.2 hostname +如同每一张网卡的mac是规则且唯一的，hostname作为资源的标识也是如此。 + +###### 2.1.2.1 如何定义hostname，这里还是以Taishan服务器为例： +` taishan200-2280-${cpuNum}s${coreNum}p-${memSize:-256}g--${aN}` +> taishan200-2280 服务器型号 +> cpuNum cpu个数 +> coreNum cpu核数 +> memSize 内存大小 +> aN 该类型机器编号 + +## 发现 +### 1. 怎么发现测试机并给它分配任务 +测试机器需要将启动方式设置pxe启动，然后在启动环节向scheduler发起请求，获取启动参数和任务信息。 + +### 2. 设置PXE启动 +- 华为服务器默认pxe启动，无需配置。 +- 其他设备，根据产品类型，按文档操作即可。 -- 2.23.0

2 2

[PATCH lab-z9] hosts: add host file of x86_64
by Zhang Yu 02 Dec '20

02 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- hosts/2288hv5-2s44p-384g | 4 ++++ hosts/2288hv5-2s44p-384g--b6 | 12 ++++++++++++ 2 files changed, 16 insertions(+) create mode 100644 hosts/2288hv5-2s44p-384g create mode 100644 hosts/2288hv5-2s44p-384g--b6 diff --git a/hosts/2288hv5-2s44p-384g b/hosts/2288hv5-2s44p-384g new file mode 100644 index 0000000..ffb5d45 --- /dev/null +++ b/hosts/2288hv5-2s44p-384g @@ -0,0 +1,4 @@ +arch: x86_64 +nr_cpu: 88 +nr_node: 2 +memory: 384G diff --git a/hosts/2288hv5-2s44p-384g--b6 b/hosts/2288hv5-2s44p-384g--b6 new file mode 100644 index 0000000..2262524 --- /dev/null +++ b/hosts/2288hv5-2s44p-384g--b6 @@ -0,0 +1,12 @@ +arch: x86_64 +nr_node: 2 +nr_cpu: 88 +memory: 384G +nr_ssd_partitions: 1 +ssd_partitions: + - /dev/disk/by-id/ata-MTFDDAK480TDC-1AT1ZABYY_20032653CE5E +mac_addr: + - 6c:eb:b6:50:c0:31 + - 6c:eb:b6:50:c0:32 +model_name: Intel(R) Xeon(R) Gold 6266C CPU @ 3.00GHz +ipmi_ip: 9.3.3.11 -- 2.23.0

1 0