- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 2/2] git_mirror.rb: change git_repo of upstream-repos
by Li Yuanchao 30 Nov '20

30 Nov '20

Change git_repo of upstream-repos from 'upstream-repos/upstream-repos' to 'u/upstream-repos/upstream-repos'. It's to keep style the same with other repos. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5b6b889..edee331 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -174,7 +174,7 @@ class MirrorMain update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - return reload_fork_info if git_repo == 'upstream-repos/upstream-repos' + return reload_fork_info if git_repo == 'u/upstream-repos/upstream-repos' new_refs = check_new_refs(git_repo) return if new_refs[:heads].empty? @@ -254,7 +254,7 @@ class MirrorMain end def reload_fork_info - upstream_repos = 'upstream-repos/upstream-repos' + upstream_repos = 'u/upstream-repos/upstream-repos' if @git_info[upstream_repos][:cur_refs].empty? @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) else -- 2.23.0

1 0

[PATCH v2 compass-ci] container/netdata-slave: monitoring HW server
by Xiao Shenwei 30 Nov '20

30 Nov '20

[monitored object] file-server : master crystal.ci : slave z9 : slave ibmc-mgnt : slave backup-server: slave jumper-server: slave startup by multi-qemu or multi-docker job [function] slave node will collect host machine metrics, send to master slave node does not provider data storage, alarm, web services. [usage] config/stream.conf destination = master-HOST[:PORT] api key = refer-to: https://github.com/netdata/netdata/tree/master/streaming --- container/netdata-slave/Dockerfile | 10 ++++++ container/netdata-slave/build | 5 +++ container/netdata-slave/config/netdata.conf | 5 +++ container/netdata-slave/config/stream.conf | 10 ++++++ container/netdata-slave/start | 34 +++++++++++++++++++++ 5 files changed, 64 insertions(+) create mode 100755 container/netdata-slave/Dockerfile create mode 100755 container/netdata-slave/build create mode 100644 container/netdata-slave/config/netdata.conf create mode 100644 container/netdata-slave/config/stream.conf create mode 100755 container/netdata-slave/start diff --git a/container/netdata-slave/Dockerfile b/container/netdata-slave/Dockerfile new file mode 100755 index 0000000..5c2c02f --- /dev/null +++ b/container/netdata-slave/Dockerfile @@ -0,0 +1,10 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +FROM netdata/netdata + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN usermod -u 1090 netdata && \ + groupmod -g 1090 netdata diff --git a/container/netdata-slave/build b/container/netdata-slave/build new file mode 100755 index 0000000..1a84550 --- /dev/null +++ b/container/netdata-slave/build @@ -0,0 +1,5 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +docker build -t netdata/netdata-slave . diff --git a/container/netdata-slave/config/netdata.conf b/container/netdata-slave/config/netdata.conf new file mode 100644 index 0000000..456fe02 --- /dev/null +++ b/container/netdata-slave/config/netdata.conf @@ -0,0 +1,5 @@ +[global] + memory mode = none + +[health] + enabled = no diff --git a/container/netdata-slave/config/stream.conf b/container/netdata-slave/config/stream.conf new file mode 100644 index 0000000..ac91a7d --- /dev/null +++ b/container/netdata-slave/config/stream.conf @@ -0,0 +1,10 @@ +[stream] + enabled = yes + destination = + api key = + timeout seconds = 60 + default port = 19999 + send charts matching = * + buffer size bytes = 1048576 + reconnect delay seconds = 5 + initial clock resync iterations = 60 diff --git a/container/netdata-slave/start b/container/netdata-slave/start new file mode 100755 index 0000000..e790298 --- /dev/null +++ b/container/netdata-slave/start @@ -0,0 +1,34 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +. $CCI_SRC/container/defconfig.sh + +docker_rm netdata-slave + +DIR=$(dirname $(realpath $0)) + +cmd=( + docker run -d + --restart=unless-stopped + --hostname=$HOSTNAME + --name netdata-slave + -p 19999:19999 + -v $DIR/config/netdata.conf:/etc/netdata/netdata.conf + -v $DIR/config/stream.conf:/etc/netdata/stream.conf + -v /etc/passwd:/host/etc/passwd:ro + -v /etc/group:/host/etc/group:ro + -v /proc:/host/proc:ro + -v /sys:/host/sys:ro + -v /etc/localtime:/etc/localtime:ro + -v /etc/os-release:/host//etc/os-release:ro + -v /var/run/docker.sock:/var/run/docker.sock:ro + -v /srv/cache/netdata_cache:/var/cache/netdata + -v /srv/cache/netdata_lib:/var/lib/netdata + --log-driver json-file + --log-opt max-size=1g + --cap-add SYS_PTRACE + --security-opt apparmor=unconfined + netdata/netdata-slave +) +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: change git-repo of upstream-repos
by Li Yuanchao 30 Nov '20

30 Nov '20

Change git_repo of upstream-repos from 'upstream-repos/upstream-repos' to 'u/upstream-repos/upstream-repos'. It's to keep style the same with other repos. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5b6b889..edee331 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -174,7 +174,7 @@ class MirrorMain update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - return reload_fork_info if git_repo == 'upstream-repos/upstream-repos' + return reload_fork_info if git_repo == 'u/upstream-repos/upstream-repos' new_refs = check_new_refs(git_repo) return if new_refs[:heads].empty? @@ -254,7 +254,7 @@ class MirrorMain end def reload_fork_info - upstream_repos = 'upstream-repos/upstream-repos' + upstream_repos = 'u/upstream-repos/upstream-repos' if @git_info[upstream_repos][:cur_refs].empty? @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) else -- 2.23.0

1 0

[PATCH compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 30 Nov '20

30 Nov '20

Add a document for introducing the arch, nr_node, nr_cpu, memory.md fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..2a4336c --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,16 @@ +# arch, nr_node, nr_cpu, memory + +Meaning: +- Each test machine has some basic information, such as arch, memory, nr_cpu, nr_node + these fields mean: + arch: the architecture of the machine. + nr_node: number of NUMA nodes. + nr_cpu: number of logical CPUs. + memory: the storage space of the machine. +- These fields are from the file which in $LAB_SRC/hosts. +- Here is an example of these fields in the host file: + arch: aarch64 + nr_node: 4 + nr_cpu: 96 + memory: 256G + ... -- 2.23.0

1 0

[PATCH lkp-tests] job.rb: disable submit warn
by Wei Jihui 30 Nov '20

30 Nov '20

[why] it has a warn when submit job: WARN -- skip non-executable */lkp-tests/monitors/plain/wait [how] lkp-tests/monitors/plain/wait is link to bin/event/wakeup but bin/event/wakeup does not exist. so we will do not check this link file. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 8d1f7469..c485da58 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -115,6 +115,8 @@ def __create_programs_hash(glob, lkp_src) next if path =~ /\.[0-9]+$/ unless File.executable?(path) + next if File.symlink?(path) && File.readlink(path) == '../../bin/event/wakeup' + log_warn "skip non-executable #{path}" unless path =~ /\.cr$/ next end -- 2.23.0

1 0

[PATCH compass-ci] modify READNE.md
by Li Ping 30 Nov '20

30 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.md | 162 ++++++++---------------------------------------------- 1 file changed, 24 insertions(+), 138 deletions(-) diff --git a/README.md b/README.md index 644e9ba..80e0766 100644 --- a/README.md +++ b/README.md @@ -1,74 +1,32 @@ -# Compass-CI 用户指南 +## Compass-CI -## 简介 +### 关于Compass-CI -### 概念 +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务--。Compass-CI 基于开源软件PR进行自动化测试(包括构建测试，软件包自带用例测试等)，共同构建一个开放、完整的开源软件生态测试系统。 -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 - -### 功能描述 +### 功能介绍 - **测试服务** - 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 + 使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - **调测环境登录** - Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 + 使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - **测试结果分析** - Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 - -- **辅助定位** - - Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 - - -### 架构简介 -![Compass-CI架构](doc/pictures/compass-ci-architecture.png) -- **接入层** - - 成为衔接 Compass-CI 服务于开发者的纽带，同时提供内部设备管理的平台，提供 web 门户查阅测试结果、项目清单、ssh 环境登录。 - -- **服务层** - - 提供测试服务、环境登录、结果分析、辅助定界、用户项目注册能力，为本项目的主要功能。 - -- **支撑层** - - 作为项目主体功能实现，详细描述资源编排、部署、构建、测试等详细实现。 - -- **数据层** - - 为项目提供数据处理服务，包括测试结果格式化及归档。 - -- **资源层** - - 提供硬件设备，作为Compass-CI 服务部署的设备以及测试的物理设备、虚拟机、容器资源；提供测试过程中依赖软件仓库及依赖仓库、部署服务器。 - + 通过 [web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 -### 应用场景 - -- **聚合开发者测试用例** - - 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 +- **测试结果复现** -- **登录环境随时调测** - - 测试过程中，发现有 Bug 时，可随时提供调测资源服务，登录到环境进行复现、调试。 - -- **快照数据分析对比** - - 测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 - -- **辅助定界** + Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 - 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题的 Commit 信息。 - +- **辅助定位** + Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 ## 使用 Compass-CI > 您只需要先注册自己的仓库，当您的仓库有 commit 提交时，构建测试会自动执行，并且可在我们的网站中查看结果。 @@ -89,97 +47,25 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 web: https://compass-ci.openeuler.org/jobs +## Source +我们最新的，最完整的源码都存放在 [Gitee](https://gitee.com/wu_fengguang/compass-ci.git) 上，Fork 我们吧！ -## 高级功能 - -### 添加 OS 支持 - -制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 - -#### 操作步骤 - - 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） - - 通过 docker 获取 rootfs - - 1) 下载 openEuler 官方提供的 docker 镜像压缩包 - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) 加载 docker 镜像 - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) 启动 openEuler 容器 - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) 拷贝 docker 的 rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - 通过 qemu.img(qcow2格式)获取 rootfs (以 openEuler 为例) - - - 1) 下载 openEuler 官方网站提供的 qcow2 格式镜像 - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) 使用 {compass-ci}/container/qcow2rootfs 制作 rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. 定制 rootfs - 1. 使用 chroot 命令切换到 rootfs 中(此步骤需要 root 权限) - ```bash - chroot openEuler-rootfs - ``` - 2. 根据个人需要安装并配置服务 - - a. 修改 root 密码 - b. 配置 ssh 服务 - c. 检查系统时间 - d. 如果使用 docker 制作 osimage 还需要以下操作： - > 1. 安装所需版本内核 - > 2. 从 centos 官方网站下载内核 rpm 包 - > 3. 使用 yum 进行安装 - > 4. 删除 docker 环境变量文件 - - 3. 退出 rootfs，并打包 - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` -#### FAQ -1. 日志报错 “Unable to mount root fs on unknown-block” - - 问题现象 - ```bash - [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: - [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) - ... - [ 0.399404] Memory Limit: none - [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- - ``` - - 解决方法 - - 1）启动内存不足，增加内存可解决。 - 2）内核文件权限不足，给予 644 权限。 - -2. 系统运行缓慢 - - - 问题现象 +## Contributing to Compass-CI + +我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导 + +## Website - 打包镜像体积过大，会消耗很大内存。 - - 解决方法 +所有的测试结果，已加入的开源软件清单，历史测试查询比较都可以在我们的官网 [Website](https://compass-ci.openeuler.org) 上找到。 - 建议用户根据具体需要对 rootfs 进行裁剪。 +## 加入我们 +您可以通过以下的方式加入我们： + - 您可以加入我们的 [mailing list](https://mailweb.openeuler.org/postorius/lists/compass-ci.openeuler.or… + - 您可以加入我们每周四下午四点 16：00 的项目例会（会议链接将会通过邮件发送给您） -## 呼吁合作 +欢迎您跟我们一起： - 增强 git bisect 能力 - 增强数据分析能力 - 增强数据结果可视化能力 -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: fix: useless condition
by Ren Wen 30 Nov '20

30 Nov '20

'queue' will always have the value after initializing the Job. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..c8d0875 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -108,11 +108,6 @@ class Sched # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) queue = job.queue - return [{ - "job_id" => "0", - "message" => "get queue failed", - "job_state" => "submit", - }] unless queue # only single job will has "idle job" and "execute rate limiter" if job["idle_job"].empty? -- 2.23.0

1 0

[PATCH compass-ci] compare_matrixes.rb: put compare matrix with more members first
by Li Yuanchao 30 Nov '20

30 Nov '20

As hash table is disordered, the output of compare matrixes sometime would put one member matrix first, that is not a good comparation. We should put good comparation first. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare_matrixes.rb | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d8112e9..652d8f0 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -383,24 +383,33 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group +# the result with more comparison objects first def compare_group_matrices(group_matrices, suites_list, options) result_str = '' - group_matrices.each do |k, v| - matrices_list = [] - matrices_titles = [] - v.each do |dim, matrix| - matrices_titles << dim - matrices_list << matrix - end - if options[:no_print] - result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - else - print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - end + group_matrices_array = sort_by_matrice_size(group_matrices) + group_matrices_array.each do |matrice_kv| + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) end result_str end +def get_matrix_str(matrice_key, matrice_value, suites_list, options) + m_list = [] + m_titles = [] + matrice_value.each do |dim, matrix| + m_titles << dim + m_list << matrix + end + return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + + print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) +end + +#big size first +def sort_by_matrice_size(group_matrices) + group_matrices.sort { |a, b| b[1].size <=> a[1].size } +end + # input: groups_matrices # { # group_key_1 => { -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: field 'queue' always has the value
by Ren Wen 30 Nov '20

30 Nov '20

Field 'queue' will always have a value when initializing a Job. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..c8d0875 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -108,11 +108,6 @@ class Sched # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) queue = job.queue - return [{ - "job_id" => "0", - "message" => "get queue failed", - "job_state" => "submit", - }] unless queue # only single job will has "idle job" and "execute rate limiter" if job["idle_job"].empty? -- 2.23.0

1 1

[PATCH compass-ci] mail-robot: add backup email
by Luan Shengde 30 Nov '20

30 Nov '20

[why] backup email file for successfully applied account [how] backup email after build email message for successfully applied account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..2d5631d 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -98,12 +98,23 @@ class ApplyAccount es.put_source_by_id(@my_info['my_email'], @my_info) end + def store_email(email_message) + time_now = Time.new.strftime('%Y%m%d%H%M%S') + file_name = [@my_info['my_email'], time_now].join '_' + file_full_name = File.join(MAILDIR, 'sent', file_name) + f = File.new(file_full_name, 'w') + f.puts email_message + f.close + FileUtils.chown_R(1144, 1110, file_full_name) + end + def send_mail(error_message) - email_message = if error_message.empty? - build_apply_account_email(@my_info) - else - build_apply_account_fail_email(@my_info, error_message) - end + if error_message.empty? + email_message = build_apply_account_email(@my_info) + store_email(email_message) + else + email_message = build_apply_account_fail_email(@my_info, error_message) + end %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") end -- 2.23.0

2 2