December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] jobs: rm my_email field and set "pub_key" field as level-1 parameter in borrow-*.yaml
by Zhang Yale 03 Dec '20

03 Dec '20

[Why] my_email field is seted in .config/compass-ci/default/account.yaml, and now the "pub_key" field is not placed under the "sshd" field in yaml. Use "pub_key" to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-10d.yaml | 15 +++++++-------- jobs/borrow-1d.yaml | 16 +++++++--------- 2 files changed, 14 insertions(+), 17 deletions(-) diff --git a/jobs/borrow-10d.yaml b/jobs/borrow-10d.yaml index aeb55466..a2b50025 100644 --- a/jobs/borrow-10d.yaml +++ b/jobs/borrow-10d.yaml @@ -1,14 +1,13 @@ suite: borrow testcase: borrow +pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> sshd: - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - my_email: # sleep at the bottom sleep: 10d diff --git a/jobs/borrow-1d.yaml b/jobs/borrow-1d.yaml index 72949982..7767f2d7 100644 --- a/jobs/borrow-1d.yaml +++ b/jobs/borrow-1d.yaml @@ -3,15 +3,13 @@ testcase: borrow timeout: 1d +pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> sshd: - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - my_email: - # sleep at the bottom sleep: 1d -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/mail_job_result.rb: fix NoMethodError
by Lu Weitao 03 Dec '20

03 Dec '20

[Why] /home/lwt/compass-ci/lib/mail_job_result.rb:45:in `query_job': undefined method `[]' for nil:NilClass (NoMethodError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/mail_job_result.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/mail_job_result.rb b/lib/mail_job_result.rb index 29a9c1a..50e2f53 100644 --- a/lib/mail_job_result.rb +++ b/lib/mail_job_result.rb @@ -33,7 +33,7 @@ class MailJobResult def set_submitter_info job = query_job - exit unless job['email'] + exit unless job && job['email'] @submitter_email = job['email'] @result_root = job['result_root'] @@ -42,6 +42,11 @@ class MailJobResult def query_job es = ESQuery.new query_result = es.multi_field_query({ 'id' => @job_id }) + if query_result['hits']['hits'].empty? + warn "Non-existent job: #{@job_id}" + return nil + end + query_result['hits']['hits'][0]['_source'] end end -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: add methods to search testboxes
by Li Yuanchao 03 Dec '20

03 Dec '20

Query all testboxes from es, and return a non repetitive list Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index e548d61..fe9545f 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -435,3 +435,31 @@ def template_body(request_body) formatter = FormatEchartData.new(compare_results, request_body) formatter.format_for_echart.to_json end + +def search_testboxes + query = { size: 0, aggs: { testboxes: { terms: { size: 10000, field: 'testbox' } } } } + buckets = es_query(query)['aggregations']['testboxes']['buckets'] + testboxes = [] + buckets.each_index do |index| + testboxes[index] = buckets[index]['key'] + end + return testboxes, testboxes.length +end + +def testboxes_body + testboxes, total = search_testboxes + { + total: total, + testboxes: testboxes + }.to_json +end + +def query_testboxes + begin + body = testboxes_body + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

2 2

[PATCH compass-ci] doc: change file name nr_threads->nr_threads.md
by Xu Xijian 03 Dec '20

03 Dec '20

Specify it as a markdown file. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/{nr_threads => nr_threads.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename doc/job/{nr_threads => nr_threads.md} (100%) diff --git a/doc/job/nr_threads b/doc/job/nr_threads.md similarity index 100% rename from doc/job/nr_threads rename to doc/job/nr_threads.md -- 2.23.0

1 0

[PATCH v2 compass-ci] container: change vmlinuz path
by Xu Xijian 03 Dec '20

03 Dec '20

Since vmlinuz path changed from $rootfs_dir/vmlinuz to $rootfs_dir/boot/vmlinuz, here keep consistent. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/update-os-docker/lib/common.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/update-os-docker/lib/common.rb b/container/update-os-docker/lib/common.rb index 23c4fdc..5b1353d 100755 --- a/container/update-os-docker/lib/common.rb +++ b/container/update-os-docker/lib/common.rb @@ -27,7 +27,7 @@ end def check_argv(argv) usage(argv) rootfs_dir = Pathname.new(OS_PATH + argv[0]).realpath.to_s - raise 'Wrong vmlinuz path' unless File.exist?(rootfs_dir + '/vmlinuz') + raise 'Wrong vmlinuz path' unless File.exist?(rootfs_dir + '/boot/vmlinuz') return rootfs_dir end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: set ENV DOCKER_REGISTRY_HOST default value
by Liu Yinsi 03 Dec '20

03 Dec '20

$DOCKER_REGISTRY_HOST default value should keep consistent with sparrow/3-code/dev-env, use server ip. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2753b9d..74905f1 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -26,7 +26,9 @@ cat >> /etc/modules-load.d/cifs <<EOF cifs EOF -: ${DOCKER_REGISTRY_HOST:="127.0.0.1"} +server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') + +: ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} cat > /etc/docker/daemon.json <<EOF { -- 2.23.0

1 0

[PATCH v3 compass-ci] answerback-email: add execute tool limit
by Luan Shengde 03 Dec '20

03 Dec '20

limit the scope of executing answerback-email to 'z9' [why] when execute answerback-email, the account info will be stored to the es installed at the localhost. It will cause validation to fail when submit a job if the data is not at the server 'z9'. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 31 +++++++++++++++++--- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 825d815..53c321d 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -187,11 +187,14 @@ def apply_account(my_info, conf_info) end def check_my_email(my_info) + return true if my_info['my_email'] + message = "No email address specified\n" message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' + puts message - raise message if my_info['my_email'].nil? + return false end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) @@ -219,13 +222,32 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] end +def check_server + return true if ENV['HOSTNAME'] == 'z9' + + message = 'please run the tool on z9 server' + puts message + + return false +end + +def check_my_name_exist(my_info) + return true if my_info['my_name'] + + message = 'No my_name found, please use -n to add one' + puts message + + return false +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) - check_my_email(my_info) + return unless check_server + return unless check_my_email(my_info) + my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) - message = 'No my_name found, please use -n to add one' - raise message if my_info['my_name'].nil? + return unless check_my_name_exist(my_info) account_info = apply_account(my_info, conf_info) build_my_info_from_account_info(my_info, account_info, conf_info) @@ -241,6 +263,7 @@ def send_mail(my_info, account_info, conf_info) else build_message(my_info['my_email'], account_info) end + %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/es_query.rb: add traverse es data of function
by Cui Yili 03 Dec '20

03 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- lib/es_query.rb | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index c027aa9..3827ea8 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -11,6 +11,7 @@ class ESQuery PORT = (ENV.key?('ES_PORT') ? ENV['ES_PORT'] : ES_PORT).to_i def initialize(host = HOST, port = PORT, index: 'jobs') @index = index + @scroll_id = '' @client = Elasticsearch::Client.new url: "http://#{host}:#{port}" raise 'Connect Elasticsearch error!' unless @client.ping end @@ -33,6 +34,25 @@ class ESQuery @client.search index: 'jobs*', body: query end + def traverse_field(size) + if @scroll_id.empty? + query = { + query: { + bool: { + must: { + match_all: {} + } + } + }, size: size + } + result = @client.search index: @index, scroll: '10m', body: query + @scroll_id = result['_scroll_id'] + return result + else + @client.scroll scroll: '10m', scroll_id: @scroll_id + end + end + def query_by_id(id) @client.get_source({ index: @index, type: '_doc', id: id }) rescue Elasticsearch::Transport::Transport::Errors::NotFound -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/manual: 修改名字为如何登录测试机调测任务.md并增加和修改内容
by Zhang Yale 03 Dec '20

03 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- ...03\346\265\213\344\273\273\345\212\241.md" | 91 +++++++++++++------ 1 file changed, 65 insertions(+), 26 deletions(-) rename doc/manual/how-to-log-in-the-machine.md => "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" (56%) diff --git a/doc/manual/how-to-log-in-the-machine.md "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" similarity index 56% rename from doc/manual/how-to-log-in-the-machine.md rename to "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" index fefd718..22da3c5 100644 --- a/doc/manual/how-to-log-in-the-machine.md +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -1,17 +1,22 @@ -这篇文档将告诉你如何登陆测试环境 +这篇文档将告诉你如何登陆测试环境去调测任务 -# 1. 前提条件 -请先学习: +# 1. 前提条件： +## 请先学习: * [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 * [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 * [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 # 2. 操作方法 -## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： -```shell +## 2.1 如何修改yaml? +### 在测试机运行script之前进入测试机调测任务 + 可以在job.yaml里加上sshd和sleep字段，测试机在运行脚本之前sleep, + 并免密登录进去，手动输入命令或脚本进行调试，以host-info.yaml为例： + +```yaml suite: host-info category: functional + pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -19,6 +24,7 @@ nil end %> + sshd: # sleep at the bottom sleep: 1h @@ -26,31 +32,42 @@ host-info: ``` -## 2.2 以下有2种方式可以登录到测试机： -### 第一种：使用submit -m -c的方式： - 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 - 命令：submit -m -c host-info.yaml - 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： - 效果如下： +### 在测试任务运行失败时进入测试机调测任务 + 可以在job.yaml里加上pub_key和on_fail字段，并在on_fail下加上sshd和sleep字段, + 测试任务失败后，测试机保留并免密登录进去，以方便手动调试，以iperf.yaml为例： -```shell - hi6325@account-vm ~% submit -m -c atomic.yaml - submit atomic.yaml, got job_id=crystal.146528 - query=>{"job_id":["crystal.146528"]} - connect to ws://localhost:11310/filter - {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" - {"job_id": "crystal.146528", "job_state": "boot"} - {"job_id": "crystal.146528", "job_state": "download"} - "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} - {"job_state":"running","job_id":"crystal.146528"} - {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} +```yaml + suite: iperf + category: benchmark - root@vm-2p8g ~# + runtime: 300s + + cluster: cs-localhost + + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + + if role server: + iperf-server: + + if role client: + iperf: + protocol: + - tcp + - udp ``` -### 第二种：根据邮件信息使用ssh方式登录测试机： - 这种是使用submit方式提交的任务完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 - 命令： submit host-info.yaml +## 2.2 如何免密登录执行机？ +### 直接submit job.yaml或者submit加-m参数 + 可在收到的邮件里，按照邮件提示手动免密登录到执行机调测 + + 命令：submit iperf.yaml或者submit -m iperf.yaml + 任务执行完成后，系统发送邮件内容如下： ```shell @@ -84,3 +101,25 @@ root@vm-2p8g ~# ``` + +### submit job.yaml时加上-m -c + 该方法不用查看邮件，可自动免密登录到测试机： + + 命令：submit -m -c iperf.yaml + + 效果如下： + +```shell + hi6325@account-vm ~% submit -m -c iperf.yaml + submit iperf.yaml, got job_id=crystal.146528 + query=>{"job_id":["crystal.146528"]} + connect to ws://localhost:11310/filter + {"job_id": "crystal.146528", "result_root": "/srv/result/iperf/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "job_state": "boot"} + {"job_id": "crystal.146528", "job_state": "download"} + "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} + {"job_state":"running","job_id":"crystal.146528"} + {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} + + root@vm-2p8g ~# +``` -- 2.23.0

2 3

[PATCH compass-ci] container: change vmlinuz path
by Xu Xijian 03 Dec '20

03 Dec '20

Since vmlinuz path changed from $rootfs_dir/vmlinuz to rootfs_dir/boot/vmlinuz, here keep consistent. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/update-os-docker/lib/common.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/update-os-docker/lib/common.rb b/container/update-os-docker/lib/common.rb index 23c4fdc..5b1353d 100755 --- a/container/update-os-docker/lib/common.rb +++ b/container/update-os-docker/lib/common.rb @@ -27,7 +27,7 @@ end def check_argv(argv) usage(argv) rootfs_dir = Pathname.new(OS_PATH + argv[0]).realpath.to_s - raise 'Wrong vmlinuz path' unless File.exist?(rootfs_dir + '/vmlinuz') + raise 'Wrong vmlinuz path' unless File.exist?(rootfs_dir + '/boot/vmlinuz') return rootfs_dir end -- 2.23.0

2 2