April 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] providers/my-docker: use dc-8g for universality
by Lin Jiaxin 01 Apr '21

01 Apr '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- providers/my-docker | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/my-docker b/providers/my-docker index c8420c7..0a9f8b0 100755 --- a/providers/my-docker +++ b/providers/my-docker @@ -5,7 +5,7 @@ require_relative './docker/docker' -tbox_group = 'dc-1g' +tbox_group = 'dc-8g' hostname = "#{tbox_group}.#{ENV['USER']}-#{Process.pid}" # specify which queues will be request, use "," to separate more than 2 values queues = "#{tbox_group}~#{ENV['USER']}" -- 2.23.0

1 0

[PATCH v3 compass-ci] doc: update sparrow README.md
by Liu Yinsi 01 Apr '21

01 Apr '21

Locally deploy compass-ci just for sparrow, if users want to use other services, prompt users to apply for account and borrow machine. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 60 ++++++++++++++++++----------------------------- 1 file changed, 23 insertions(+), 37 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index d7b8e81..532b7f9 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -41,7 +41,7 @@ 3. 创建工作目录并克隆 compass-ci 项目代码 ```bash mkdir /c/ && ln -s /c/compass-ci /c/cci - git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/compass-ci.git ``` 4. 编辑setup.yaml配置用户名和邮箱 @@ -57,53 +57,39 @@ cd compass-ci/sparrow && ./install-tiny ``` -#### 提交测试任务前的准备 - -1. 测试环境是否可以提交job测试 +#### 提交测试任务 +本文以/c/lkp-tests/jobs/目录下已有的通用测试用例host-info.yaml为例 +- 使环境变量生效 ```bash - submit iperf.yaml - ``` - - 执行上述命令正常情况下会提示信息如下: + source /etc/profile.d/compass-ci ``` - submit_id=bf5e7ad7-839d-48ec-a033-23281323c750 - submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - ``` - compass-ci搭建完毕，下面就可以开始进行测试了。 - -#### 提交测试任务到本地compass-ci -本文以测试用例iperf.yaml为例 -1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - -2. 根据测试需要[编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… - -3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… +- 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… ```bash - submit iperf.yaml + submit host-info.yaml ``` -#### 运行测试任务并查看任务结果 - -1. 运行测试任务 - ```bash - cd /c/compass-ci/providers/ && ./my-qemu.sh + 执行上述命令会打印提示信息如下: + ``` + submit_id=bf5e7ad7-839d-48ec-a033-23281323c750 + submit /c/lkp-tests/jobs/host-info.yaml, got job id=nolab.1 ``` -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/lab-… [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… +- 查看任务结果 +等待约1分钟，可根据上一步骤中打印的job id查看任务结果。 ```bash - cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 - cat output + cd $(es-find id=nolab.1 |grep result_root|awk -F '"' '{print "/srv/"$4}') && ls ``` - >**说明：** - >[登陆测试环境调测任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… + 结果文件介绍 + job.yaml文件 + job.yaml 文件中部分字段是用户提交上来的，其他字段是平台根据提交的 job 自动添加进来的。此文件包含了测试任务需要的所有参数。 -# FAQ + output文件 + output 文件记录了用例的执行过程，文件最后部分一般会有 check_exit_code 这个状态码，非 0 代表测试用例错误。 -* 选择下载os rootfs + stats.json + 测试用例执行完成会生成一个与测试用例同名的文件，记录它们的测试命令及标准化输出结果。compass-ci 会对这些文件进行解析，生成后缀名是 .json 的文件。 + stats.json 是所有的 json 文件的汇总，所有测试命令的关键结果都会统计到这个文件中，便于后续的比较和分析。 - 启动qemu测试机需要[os rootfs文件](http://api.compass-ci.openeuler.org:11304/os/)， - 一键部署默认下载os rootfs为openeuler aarch64 20.03，当部署完毕后，可在该目录下查看: /srv/os/openeuler/aarch64/20.03 - 如需要其他os rootfs，可使用该脚本下载: /c/compass-ci/sbin/download-rootfs +体验更多功能例如[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-project.zh.md)、[调测环境登录](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/log-in-machine-debug.md)、[测试结果分析](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/compare-results.zh.md)等，请[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.zh.md)、[申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/borrow-machine.zh.md)。 -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: add queues to testbox
by Wu Zhende 01 Apr '21

01 Apr '21

Save data like this "queues":["sched/dc-1g~wuzhende/ready"] to ES. So we can collect testbox information of a queue. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 6 ++++-- src/scheduler/find_job_boot.cr | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 6fd6657..c84b3f3 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -143,7 +143,7 @@ class Sched @log.warn(e) end - def set_lifecycle(job, testbox) + def set_lifecycle(job, testbox, queues) if job deadline = job.get_deadline job["deadline"] = deadline @@ -156,11 +156,13 @@ class Sched state = "requesting" end + queues = JSON.parse(queues.to_json) hash = { "job_id" => job_id, "state" => state, "time" => get_time, - "deadline" => deadline + "deadline" => deadline, + "queues" => queues } @redis.update_wtmp(testbox.to_s, hash) @es.update_tbox(testbox.to_s, hash) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 6e8d5d1..ff88d32 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -182,7 +182,7 @@ class Sched def get_job_boot(host, boot_type) queues = get_queues(host) job = get_job_from_queues(queues, host) - set_lifecycle(job, host) + set_lifecycle(job, host, queues) if job @es.set_job_content(job) -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: add parameters annotation
by Wang Yong 01 Apr '21

01 Apr '21

it's for choosing parameters clearly Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 7a101b6a0..a3868d54a 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,6 +1,7 @@ #!/bin/bash # - upstream_repo # - compat_os +# Git repo jobs only use the above parameters, SRPMs jobs use all # - repo_name # - repo_addr -- 2.23.0

1 0

[PATCH v3 lkp-tests] stat: optimize to obtain error for build DockerFile
by Liu Shaofei 01 Apr '21

01 Apr '21

[output] catch errors will be like this: { "openeuler_docker.wordpress_1.build.fail": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.php-mysql": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.php-mysql.message": [ "Error: Unable to find a match php-mysql" ], "openeuler_docker.tools_2.build.fail": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.netsniff-ng": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.netsniff-ng.message": [ "Error: Unable to find a match netsniff-ng" ], "openeuler_docker.yum.error.Unable-to-find-a-match.rpm-ostree": [ 1 ], ...... ...... } Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- stats/openeuler_docker.rb | 103 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100755 stats/openeuler_docker.rb diff --git a/stats/openeuler_docker.rb b/stats/openeuler_docker.rb new file mode 100755 index 000000000..5a9365938 --- /dev/null +++ b/stats/openeuler_docker.rb @@ -0,0 +1,103 @@ +#!/usr/bin/env ruby + +def pre_handel(result, file_path) + status = false + repo_set = Set[] + sys_set = Set[] + + File.readlines(file_path).each do |line| + case line.chomp! + + # Error: Unable to find a match: docker-registry mock xx + when /Error: Unable to find a match: (.+)/ + $1.split.each do |repo| + repo_set << repo + end + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + result['requested-URL-returned.error'] = [1] + result['requested-URL-returned.error.message'] = [line.to_s] + status = true + + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + result["unknown-repo.#{repo}"] = [1] + result["unknown-repo.#{repo}.message"] = [line.to_s] + status = true + + # Error: Module or Group 'convert' does not exist. + when /Error: Module or Group ('[^\s]+')/ + repo = $1.delete!("'") + result["error.not-exist-module-or-group.#{repo}"] = [1] + result["error.not-exist-module-or-group.#{repo}.message"] = [line.to_s] + status = true + + # /bin/sh: passwd: command not found + when /\/bin\/sh: (.+): command not found/ + result["sh.command-not-found.#{$1}"] = [1] + result["sh.command-not-found.#{$1}.message"] = [line.to_s] + status = true + end + + repo_set.each do |repo| + result["yum.error.Unable-to-find-a-match.#{repo}"] = [1] + result["yum.error.Unable-to-find-a-match.#{repo}.message"] = ["Error: Unable to find a match #{repo}"] + status = true + end + + sys_set.each do |sys| + result["yum.swap.error.unrecognized-arguments-install.#{sys}"] = [1] + result["yum.swap.error.unrecognized-arguments.#{sys}.message"] = + ["yum swap: error: unrecognized arguments install #{sys}"] + status = true + end + end + status +end + +def handle_unknown_error(_result, file_path) + line_num = %x(cat #{file_path} | grep -n 'Step ' | tail -1 | awk -F: '{print $1}') + + index = 1 + message = '' + File.readlines(file_path).each do |line| + if index == Integer(line_num) + message += line + else + index += 1 + end + end + + message = $1 if message =~ %r(\u001b\[91m(.+)) + message +end + +def openeuler_docker(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + + log_lines.each do |line| + next unless line =~ %r(([^\s]+).(build|run)\.fail) + + key, value = line.split(':') + key.chomp! + result[key] << value.to_i + + file_path = "#{RESULT_ROOT}/#{$1}" # $1 named by docker-image name + next unless File.exist?(file_path) + next if pre_handel(result, file_path) + + result["#{key}.message"] << handle_unknown_error(result, file_path) + end + + result +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] stat: optimize to obtain error for build DockerFile
by Liu Shaofei 01 Apr '21

01 Apr '21

[output] catch errors will be like this: { "openeuler_docker.wordpress_1.build.fail": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.php-mysql": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.php-mysql.message": [ "Error: Unable to find a match php-mysql" ], "openeuler_docker.tools_2.build.fail": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.netsniff-ng": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.netsniff-ng.message": [ "Error: Unable to find a match netsniff-ng" ], "openeuler_docker.yum.error.Unable-to-find-a-match.rpm-ostree": [ 1 ], } Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- stats/openeuler_docker.rb | 103 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100755 stats/openeuler_docker.rb diff --git a/stats/openeuler_docker.rb b/stats/openeuler_docker.rb new file mode 100755 index 000000000..5a9365938 --- /dev/null +++ b/stats/openeuler_docker.rb @@ -0,0 +1,103 @@ +#!/usr/bin/env ruby + +def pre_handel(result, file_path) + status = false + repo_set = Set[] + sys_set = Set[] + + File.readlines(file_path).each do |line| + case line.chomp! + + # Error: Unable to find a match: docker-registry mock xx + when /Error: Unable to find a match: (.+)/ + $1.split.each do |repo| + repo_set << repo + end + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + result['requested-URL-returned.error'] = [1] + result['requested-URL-returned.error.message'] = [line.to_s] + status = true + + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + result["unknown-repo.#{repo}"] = [1] + result["unknown-repo.#{repo}.message"] = [line.to_s] + status = true + + # Error: Module or Group 'convert' does not exist. + when /Error: Module or Group ('[^\s]+')/ + repo = $1.delete!("'") + result["error.not-exist-module-or-group.#{repo}"] = [1] + result["error.not-exist-module-or-group.#{repo}.message"] = [line.to_s] + status = true + + # /bin/sh: passwd: command not found + when /\/bin\/sh: (.+): command not found/ + result["sh.command-not-found.#{$1}"] = [1] + result["sh.command-not-found.#{$1}.message"] = [line.to_s] + status = true + end + + repo_set.each do |repo| + result["yum.error.Unable-to-find-a-match.#{repo}"] = [1] + result["yum.error.Unable-to-find-a-match.#{repo}.message"] = ["Error: Unable to find a match #{repo}"] + status = true + end + + sys_set.each do |sys| + result["yum.swap.error.unrecognized-arguments-install.#{sys}"] = [1] + result["yum.swap.error.unrecognized-arguments.#{sys}.message"] = + ["yum swap: error: unrecognized arguments install #{sys}"] + status = true + end + end + status +end + +def handle_unknown_error(_result, file_path) + line_num = %x(cat #{file_path} | grep -n 'Step ' | tail -1 | awk -F: '{print $1}') + + index = 1 + message = '' + File.readlines(file_path).each do |line| + if index == Integer(line_num) + message += line + else + index += 1 + end + end + + message = $1 if message =~ %r(\u001b\[91m(.+)) + message +end + +def openeuler_docker(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + + log_lines.each do |line| + next unless line =~ %r(([^\s]+).(build|run)\.fail) + + key, value = line.split(':') + key.chomp! + result[key] << value.to_i + + file_path = "#{RESULT_ROOT}/#{$1}" # $1 named by docker-image name + next unless File.exist?(file_path) + next if pre_handel(result, file_path) + + result["#{key}.message"] << handle_unknown_error(result, file_path) + end + + result +end -- 2.23.0

1 1

[PATCH compass-ci] doc: README: replace invalid url
by Liu Yinsi 01 Apr '21

01 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- README.en.md | 8 ++++---- README.zh.md | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/README.en.md b/README.en.md index fa88305..7449696 100644 --- a/README.en.md +++ b/README.en.md @@ -12,7 +12,7 @@ Compass-CI monitors git repos of a large amount of open source software. Once Co **Login to the Test Environment** -The SSH is used to [log in to the test environment for commissioning](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…. +The SSH is used to [log in to the test environment for commissioning](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…. **Test Result Analysis** @@ -38,15 +38,15 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Manually Submitting a Test Task** -1. [Install the Compass-CI client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E…. +1. [Install the Compass-CI client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/in…. 2. [Compile test cases and manually submit test tasks](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. 3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs **Logging in to the Test Environment** 1. Send an email to the compass-ci-robot(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. -2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. -3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. +2. Complete the environment configuration. +3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. ## Contributing to Compass-CI diff --git a/README.zh.md b/README.zh.md index 0e6cfb6..f897ae0 100644 --- a/README.zh.md +++ b/README.zh.md @@ -14,7 +14,7 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， **调测环境登录** -使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… **测试结果分析** @@ -41,15 +41,15 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， **手动提交测试任务** -1. [安装 Compass-CI 客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9… +1. [安装 Compass-CI 客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/insta… 2. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca…, [手动提交测试任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… 3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs **登录测试环境** 1. 向 compass-ci-robot(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… -2. 根据邮件反馈内容[完成环境配置](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… -3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… +2. 根据邮件反馈内容完成环境配置。 +3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… ## Contributing to Compass-CI -- 2.23.0

1 0