- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] service/scheduler: add event time
by Wu Zhende 31 Mar '21

31 Mar '21

example: "submit_time": "2021-03-31T16:01:07+0800" "boot_time": "2021-03-31T16:01:33+0800" "close_time": "2021-03-31T16:02:47+0800" Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 5 +++++ src/scheduler/close_job.cr | 5 ++++- src/scheduler/find_job_boot.cr | 1 + 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 5767f61..2d15aae 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -170,6 +170,11 @@ class Job set_sshr_info() set_queue() set_subqueue() + set_time("submit_time") + end + + def set_time(key) + self[key] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") end private def set_docker_os diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 746f70a..a112a38 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -15,6 +15,9 @@ class Sched job["job_state"] = job_state if job_state job["job_state"] = "complete" if job["job_state"] == "boot" + job.set_time("close_time") + @env.set "close_time", job["close_time"] + response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -34,7 +37,7 @@ class Sched mq_msg = { "job_id" => @env.get?("job_id").to_s, "job_state" => "close", - "time" => get_time + "time" => @env.get?("close_time").to_s } spawn mq_publish_confirm(JOB_MQ, mq_msg.to_json) end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 2596f83..6e8d5d1 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -92,6 +92,7 @@ class Sched if job job.update({"testbox" => testbox}) job.set_result_root + job.set_time("boot_time") @log.info(%({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state" : "set result root"})) set_id2job(job) -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/git_mirror.rb: fix es update fail
by Li Yuanchao 31 Mar '21

31 Mar '21

As the es update to 7.11, some compatibility error occurs. [error msg] git-mirror: warning: 299 Elasticsearch-7.11.1-ff17057114c2199c9c1bbecc727003a907c0db7a "[types removal] Specifying types in doc ument update requests is deprecated, use the endpoint /{index}/_update/{id} instead." Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 7f36be3..1f4e5b5 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -308,11 +308,8 @@ class MirrorMain def es_repo_update(git_repo) repo_info = { 'git_repo' => git_repo, 'url' => @git_info[git_repo]['url'] } repo_info = repo_info.merge(@fork_stat[git_repo]) - body = { - "doc": repo_info, - "doc_as_upsert": true - } - @es_client.update(index: 'repo', type: '_doc', id: git_repo, body: body) + body = repo_info + @es_client.index(index: 'repo', type: '_doc', id: git_repo, body: body) end def update_stat_fetch(git_repo) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/git_mirror.rb: fix no positive? method error
by Li Yuanchao 31 Mar '21

31 Mar '21

The result of query es changed, the result['hits']['total'] changes from a number to a hash, so the positive? method can not be used. Now use the count method which contained by es_client. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index c967c8b..7f36be3 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -414,9 +414,9 @@ class MirrorMain new_refs_count: {} } query = { query: { match: { _id: git_repo } } } - result = @es_client.search(index: 'repo', body: query)['hits'] - return fork_stat unless result['total'].positive? + return fork_stat unless @es_client.count(index: 'repo', body: query)['count'].positive? + result = @es_client.search(index: 'repo', body: query)['hits'] fork_stat.each_key do |key| fork_stat[key] = result['hits'][0]['_source'][key.to_s] || fork_stat[key] end -- 2.23.0

1 0

[PATCH lkp-tests] add self-test.yaml
by Liu Yinsi 31 Mar '21

31 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- jobs/self-test.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 jobs/self-test.yaml diff --git a/jobs/self-test.yaml b/jobs/self-test.yaml new file mode 100644 index 000000000..ad2dcd2a2 --- /dev/null +++ b/jobs/self-test.yaml @@ -0,0 +1,6 @@ +suite: self-test +category: functional + +deploy-cci: +multi-qemu: +multi-docker: -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow: add 8-testbox/multi-docker
by Liu Yinsi 31 Mar '21

31 Mar '21

For user locally deploy compass-ci, execute multi-docker in the background, it is convenient for users to submit jobs. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/8-testbox/multi-docker | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/8-testbox/multi-docker diff --git a/sparrow/8-testbox/multi-docker b/sparrow/8-testbox/multi-docker new file mode 100755 index 0000000..25cb82d --- /dev/null +++ b/sparrow/8-testbox/multi-docker @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +nohup $CCI_SRC/providers/multi-docker -n dc-8g -c 10 -q dc-8g.$(arch) & diff --git a/sparrow/install-tiny b/sparrow/install-tiny index e48bd25..40358a4 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -23,3 +23,4 @@ cd $CCI_SRC/sparrow || exit 5-build/os-ready 5-build/register-account 7-systemd/systemd-setup +8-testbox/multi-docker -- 2.23.0

1 0

[PATCH compass-ci] sparrow: add 8-testbox/multi-docker
by Liu Yinsi 31 Mar '21

31 Mar '21

For user locally deploy compass-ci, prepare to run some docker testbox, it is convenient for users to submit jobs. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/8-testbox/multi-docker | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/8-testbox/multi-docker diff --git a/sparrow/8-testbox/multi-docker b/sparrow/8-testbox/multi-docker new file mode 100755 index 0000000..94f6bde --- /dev/null +++ b/sparrow/8-testbox/multi-docker @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +$CCI_SRC/providers/multi-docker -n dc-8g -c 10 -q dc-8g.$(arch) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index e48bd25..40358a4 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -23,3 +23,4 @@ cd $CCI_SRC/sparrow || exit 5-build/os-ready 5-build/register-account 7-systemd/systemd-setup +8-testbox/multi-docker -- 2.23.0

2 1

[PATCH v2 compass-ci] doc: update sparrow README.md
by Liu Yinsi 31 Mar '21

31 Mar '21

Locally deploy compass-ci just for sparrow, if users want to use other services, prompt users to apply for account and borrow machine. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 54 +++++++++++++++-------------------------------- 1 file changed, 17 insertions(+), 37 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index d7b8e81..a448bb8 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -41,7 +41,7 @@ 3. 创建工作目录并克隆 compass-ci 项目代码 ```bash mkdir /c/ && ln -s /c/compass-ci /c/cci - git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/compass-ci.git ``` 4. 编辑setup.yaml配置用户名和邮箱 @@ -57,9 +57,14 @@ cd compass-ci/sparrow && ./install-tiny ``` -#### 提交测试任务前的准备 +#### 提交测试任务 +本文以/c/lkp-tests/jobs/目录下已有的通用测试用例iperf.yaml为例 +- 使环境变量生效 + ```bash + . /etc/profile.d/compass-ci + ``` -1. 测试环境是否可以提交job测试 +- 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… ```bash submit iperf.yaml ``` @@ -68,42 +73,17 @@ ``` submit_id=bf5e7ad7-839d-48ec-a033-23281323c750 submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - ``` - compass-ci搭建完毕，下面就可以开始进行测试了。 - -#### 提交测试任务到本地compass-ci -本文以测试用例iperf.yaml为例 - -1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - -2. 根据测试需要[编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… - -3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… - ```bash - submit iperf.yaml - ``` - -#### 运行测试任务并查看任务结果 - -1. 运行测试任务 - ```bash - cd /c/compass-ci/providers/ && ./my-qemu.sh + submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.2 ``` -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/lab-… [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… +- 查看任务结果 +任务结果文件会生成在本地/srv/result目录下，根据上一步骤显示出的job id可以查看对应任务的结果文件output(可使用tab键自动补全多级目录) ```bash - cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 - cat output + cd /srv/result/iperf/2021-03-18/dc-8g/centos-7-aarch64/tcp-300/nolab.1 && cat output + cd /srv/result/iperf/2021-03-18/dc-8g/centos-7-aarch64/udp-300/nolab.2 && cat output ``` + + >**说明：** + >output 文件记录了用例的执行过程，文件最后部分一般会有 check_exit_code 这个状态码，非 0 代表测试用例错误。 - >**说明：** - >[登陆测试环境调测任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - -# FAQ - -* 选择下载os rootfs - - 启动qemu测试机需要[os rootfs文件](http://api.compass-ci.openeuler.org:11304/os/)， - 一键部署默认下载os rootfs为openeuler aarch64 20.03，当部署完毕后，可在该目录下查看: /srv/os/openeuler/aarch64/20.03 - 如需要其他os rootfs，可使用该脚本下载: /c/compass-ci/sbin/download-rootfs +体验更多服务例如[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-project.zh.md)、[调测环境登录](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/log-in-machine-debug.md)、[测试结果分析](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/compare-results.zh.md)等，请[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.zh.md)、[申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/borrow-machine.zh.md)。 -- 2.23.0

1 0

[PATCH compass-ci] doc: update sparrow README.md
by Liu Yinsi 31 Mar '21

31 Mar '21

Locally deploy compass-ci just for sparrow, if users want to use other services, prompt users to apply for account and borrow machine. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 50 ++++++++++++----------------------------------- 1 file changed, 13 insertions(+), 37 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index d7b8e81..b161877 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -41,7 +41,7 @@ 3. 创建工作目录并克隆 compass-ci 项目代码 ```bash mkdir /c/ && ln -s /c/compass-ci /c/cci - git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/compass-ci.git ``` 4. 编辑setup.yaml配置用户名和邮箱 @@ -57,9 +57,10 @@ cd compass-ci/sparrow && ./install-tiny ``` -#### 提交测试任务前的准备 +#### 提交测试任务 +本文以/c/lkp-tests/jobs/目录下已有的通用测试用例iperf.yaml为例 -1. 测试环境是否可以提交job测试 +1. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… ```bash submit iperf.yaml ``` @@ -68,42 +69,17 @@ ``` submit_id=bf5e7ad7-839d-48ec-a033-23281323c750 submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - ``` - compass-ci搭建完毕，下面就可以开始进行测试了。 - -#### 提交测试任务到本地compass-ci -本文以测试用例iperf.yaml为例 - -1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - -2. 根据测试需要[编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… - -3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… - ```bash - submit iperf.yaml + submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.2 ``` -#### 运行测试任务并查看任务结果 - -1. 运行测试任务 - ```bash - cd /c/compass-ci/providers/ && ./my-qemu.sh - ``` - -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/lab-… [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… +2. 查看任务结果 +任务结果文件会生成在本地/srv/result目录下，根据上一步骤显示出的job id可以查看对应任务的结果文件output(可使用tab键自动补全多级目录) ```bash - cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 - cat output + cd /srv/result/iperf/2021-03-18/dc-8g/centos-7-aarch64/tcp-300/nolab.1 && cat output + cd /srv/result/iperf/2021-03-18/dc-8g/centos-7-aarch64/udp-300/nolab.2 && cat output ``` + + >**说明：** + >output 文件记录了用例的执行过程，文件最后部分一般会有 check_exit_code 这个状态码，非 0 代表测试用例错误。 - >**说明：** - >[登陆测试环境调测任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - -# FAQ - -* 选择下载os rootfs - - 启动qemu测试机需要[os rootfs文件](http://api.compass-ci.openeuler.org:11304/os/)， - 一键部署默认下载os rootfs为openeuler aarch64 20.03，当部署完毕后，可在该目录下查看: /srv/os/openeuler/aarch64/20.03 - 如需要其他os rootfs，可使用该脚本下载: /c/compass-ci/sbin/download-rootfs +体验更多服务例如[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-project.zh.md)、[调测环境登录](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/log-in-machine-debug.md)、[测试结果分析](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/compare-results.zh.md)等，请[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.zh.md)、[申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/borrow-machine.zh.md)。 -- 2.23.0

1 1

[PATCH compass-ci 1/2] lib/stats_wrapper.rb: fix TypeError
by Lu Weitao 31 Mar '21

31 Mar '21

[Why] linjiaxin@crystal ~/crystal.116494% ~/compass-ci/sbin/result2stats ./ Traceback (most recent call last): 7: from /home/linjiaxin/compass-ci/sbin/result2stats:37:in `<main>' 6: from /home/linjiaxin/compass-ci/lib/stats.rb:47:in `extract_stats' 5: from /usr/share/ruby/set.rb:338:in `each' 4: from /usr/share/ruby/set.rb:338:in `each_key' 3: from /home/linjiaxin/compass-ci/lib/stats.rb:49:in `block in extract_stats' 2: from /home/linjiaxin/compass-ci/lib/stats_wrapper.rb:61:in `wrapper_func' 1: from /home/linjiaxin/compass-ci/lib/stats_wrapper.rb:287:in `delete_log_package' /home/linjiaxin/compass-ci/lib/stats_wrapper.rb:287:in `exist?': no implicit conversion of nil into String (TypeError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index faa81df..6a08797 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -284,7 +284,7 @@ module StatsWrapper # delete temporarily extracted dmesg above File.delete(@dmesg_log) if File.exist?("#{(a)dmesg_log}.xz") - File.delete(@tmpfile) if File.exist?(@tmpfile) + File.delete(@tmpfile) if @tmpfile && File.exist?(@tmpfile) end end -- 2.23.0

2 1

[PATCH compass-ci 2/2] lib/matrix2.rb: fix NoMethodError
by Lu Weitao 31 Mar '21

31 Mar '21

[Why] linjiaxin@crystal ~/crystal.116494% ~/compass-ci/sbin/result2stats ./ Traceback (most recent call last): 7: from /home/linjiaxin/compass-ci/sbin/result2stats:38:in `<main>' 6: from /home/linjiaxin/compass-ci/lib/matrix2.rb:52:in `create_stats' 5: from /home/linjiaxin/compass-ci/lib/matrix2.rb:52:in `each' 4: from /home/linjiaxin/compass-ci/lib/matrix2.rb:58:in `block in create_stats' 3: from /home/linjiaxin/compass-ci/lib/matrix2.rb:30:in `extract_pre_result' 2: from /home/linjiaxin/compass-ci/lib/matrix2.rb:30:in `each' 1: from /home/linjiaxin/compass-ci/lib/matrix2.rb:33:in `block in extract_pre_result' /home/linjiaxin/compass-ci/lib/matrix2.rb:22:in `set_pre_value': undefined method `/' for #<String:0x0000aaabcdd1a140> (NoMethodError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 5e0ad21..12245a8 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -10,7 +10,7 @@ require "#{LKP_SRC}/lib/matrix" require_relative './params_group.rb' def set_pre_value(item, value, sample_size) - return value if value.is_a?(String) + return value if item.end_with?('.message', '.log') if value.size == 1 value[0] -- 2.23.0

2 1