- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] modify READNE.md
by Li Ping 30 Nov '20

30 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.md | 162 ++++++++---------------------------------------------- 1 file changed, 24 insertions(+), 138 deletions(-) diff --git a/README.md b/README.md index 644e9ba..80e0766 100644 --- a/README.md +++ b/README.md @@ -1,74 +1,32 @@ -# Compass-CI 用户指南 +## Compass-CI -## 简介 +### 关于Compass-CI -### 概念 +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务--。Compass-CI 基于开源软件PR进行自动化测试(包括构建测试，软件包自带用例测试等)，共同构建一个开放、完整的开源软件生态测试系统。 -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 - -### 功能描述 +### 功能介绍 - **测试服务** - 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 + 使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - **调测环境登录** - Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 + 使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - **测试结果分析** - Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 - -- **辅助定位** - - Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 - - -### 架构简介 -![Compass-CI架构](doc/pictures/compass-ci-architecture.png) -- **接入层** - - 成为衔接 Compass-CI 服务于开发者的纽带，同时提供内部设备管理的平台，提供 web 门户查阅测试结果、项目清单、ssh 环境登录。 - -- **服务层** - - 提供测试服务、环境登录、结果分析、辅助定界、用户项目注册能力，为本项目的主要功能。 - -- **支撑层** - - 作为项目主体功能实现，详细描述资源编排、部署、构建、测试等详细实现。 - -- **数据层** - - 为项目提供数据处理服务，包括测试结果格式化及归档。 - -- **资源层** - - 提供硬件设备，作为Compass-CI 服务部署的设备以及测试的物理设备、虚拟机、容器资源；提供测试过程中依赖软件仓库及依赖仓库、部署服务器。 - + 通过 [web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 -### 应用场景 - -- **聚合开发者测试用例** - - 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 +- **测试结果复现** -- **登录环境随时调测** - - 测试过程中，发现有 Bug 时，可随时提供调测资源服务，登录到环境进行复现、调试。 - -- **快照数据分析对比** - - 测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 - -- **辅助定界** + Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 - 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题的 Commit 信息。 - +- **辅助定位** + Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 ## 使用 Compass-CI > 您只需要先注册自己的仓库，当您的仓库有 commit 提交时，构建测试会自动执行，并且可在我们的网站中查看结果。 @@ -89,97 +47,25 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 web: https://compass-ci.openeuler.org/jobs +## Source +我们最新的，最完整的源码都存放在 [Gitee](https://gitee.com/wu_fengguang/compass-ci.git) 上，Fork 我们吧！ -## 高级功能 - -### 添加 OS 支持 - -制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 - -#### 操作步骤 - - 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） - - 通过 docker 获取 rootfs - - 1) 下载 openEuler 官方提供的 docker 镜像压缩包 - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) 加载 docker 镜像 - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) 启动 openEuler 容器 - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) 拷贝 docker 的 rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - 通过 qemu.img(qcow2格式)获取 rootfs (以 openEuler 为例) - - - 1) 下载 openEuler 官方网站提供的 qcow2 格式镜像 - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) 使用 {compass-ci}/container/qcow2rootfs 制作 rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. 定制 rootfs - 1. 使用 chroot 命令切换到 rootfs 中(此步骤需要 root 权限) - ```bash - chroot openEuler-rootfs - ``` - 2. 根据个人需要安装并配置服务 - - a. 修改 root 密码 - b. 配置 ssh 服务 - c. 检查系统时间 - d. 如果使用 docker 制作 osimage 还需要以下操作： - > 1. 安装所需版本内核 - > 2. 从 centos 官方网站下载内核 rpm 包 - > 3. 使用 yum 进行安装 - > 4. 删除 docker 环境变量文件 - - 3. 退出 rootfs，并打包 - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` -#### FAQ -1. 日志报错 “Unable to mount root fs on unknown-block” - - 问题现象 - ```bash - [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: - [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) - ... - [ 0.399404] Memory Limit: none - [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- - ``` - - 解决方法 - - 1）启动内存不足，增加内存可解决。 - 2）内核文件权限不足，给予 644 权限。 - -2. 系统运行缓慢 - - - 问题现象 +## Contributing to Compass-CI + +我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导 + +## Website - 打包镜像体积过大，会消耗很大内存。 - - 解决方法 +所有的测试结果，已加入的开源软件清单，历史测试查询比较都可以在我们的官网 [Website](https://compass-ci.openeuler.org) 上找到。 - 建议用户根据具体需要对 rootfs 进行裁剪。 +## 加入我们 +您可以通过以下的方式加入我们： + - 您可以加入我们的 [mailing list](https://mailweb.openeuler.org/postorius/lists/compass-ci.openeuler.or… + - 您可以加入我们每周四下午四点 16：00 的项目例会（会议链接将会通过邮件发送给您） -## 呼吁合作 +欢迎您跟我们一起： - 增强 git bisect 能力 - 增强数据分析能力 - 增强数据结果可视化能力 -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: fix: useless condition
by Ren Wen 30 Nov '20

30 Nov '20

'queue' will always have the value after initializing the Job. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..c8d0875 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -108,11 +108,6 @@ class Sched # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) queue = job.queue - return [{ - "job_id" => "0", - "message" => "get queue failed", - "job_state" => "submit", - }] unless queue # only single job will has "idle job" and "execute rate limiter" if job["idle_job"].empty? -- 2.23.0

1 0

[PATCH compass-ci] compare_matrixes.rb: put compare matrix with more members first
by Li Yuanchao 30 Nov '20

30 Nov '20

As hash table is disordered, the output of compare matrixes sometime would put one member matrix first, that is not a good comparation. We should put good comparation first. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare_matrixes.rb | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d8112e9..652d8f0 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -383,24 +383,33 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group +# the result with more comparison objects first def compare_group_matrices(group_matrices, suites_list, options) result_str = '' - group_matrices.each do |k, v| - matrices_list = [] - matrices_titles = [] - v.each do |dim, matrix| - matrices_titles << dim - matrices_list << matrix - end - if options[:no_print] - result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - else - print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - end + group_matrices_array = sort_by_matrice_size(group_matrices) + group_matrices_array.each do |matrice_kv| + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) end result_str end +def get_matrix_str(matrice_key, matrice_value, suites_list, options) + m_list = [] + m_titles = [] + matrice_value.each do |dim, matrix| + m_titles << dim + m_list << matrix + end + return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + + print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) +end + +#big size first +def sort_by_matrice_size(group_matrices) + group_matrices.sort { |a, b| b[1].size <=> a[1].size } +end + # input: groups_matrices # { # group_key_1 => { -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: field 'queue' always has the value
by Ren Wen 30 Nov '20

30 Nov '20

Field 'queue' will always have a value when initializing a Job. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..c8d0875 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -108,11 +108,6 @@ class Sched # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) queue = job.queue - return [{ - "job_id" => "0", - "message" => "get queue failed", - "job_state" => "submit", - }] unless queue # only single job will has "idle job" and "execute rate limiter" if job["idle_job"].empty? -- 2.23.0

1 1

[PATCH compass-ci] mail-robot: add backup email
by Luan Shengde 30 Nov '20

30 Nov '20

[why] backup email file for successfully applied account [how] backup email after build email message for successfully applied account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..2d5631d 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -98,12 +98,23 @@ class ApplyAccount es.put_source_by_id(@my_info['my_email'], @my_info) end + def store_email(email_message) + time_now = Time.new.strftime('%Y%m%d%H%M%S') + file_name = [@my_info['my_email'], time_now].join '_' + file_full_name = File.join(MAILDIR, 'sent', file_name) + f = File.new(file_full_name, 'w') + f.puts email_message + f.close + FileUtils.chown_R(1144, 1110, file_full_name) + end + def send_mail(error_message) - email_message = if error_message.empty? - build_apply_account_email(@my_info) - else - build_apply_account_fail_email(@my_info, error_message) - end + if error_message.empty? + email_message = build_apply_account_email(@my_info) + store_email(email_message) + else + email_message = build_apply_account_fail_email(@my_info, error_message) + end %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") end -- 2.23.0

2 2

[PATCH v2 compass-ci] modify README
by Li Ping 30 Nov '20

30 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.md | 162 ++++++++---------------------------------------------- 1 file changed, 24 insertions(+), 138 deletions(-) diff --git a/README.md b/README.md index 644e9ba..80e0766 100644 --- a/README.md +++ b/README.md @@ -1,74 +1,32 @@ -# Compass-CI 用户指南 +## Compass-CI -## 简介 +### 关于Compass-CI -### 概念 +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务--。Compass-CI 基于开源软件PR进行自动化测试(包括构建测试，软件包自带用例测试等)，共同构建一个开放、完整的开源软件生态测试系统。 -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 - -### 功能描述 +### 功能介绍 - **测试服务** - 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 + 使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - **调测环境登录** - Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 + 使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - **测试结果分析** - Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 - -- **辅助定位** - - Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 - - -### 架构简介 -![Compass-CI架构](doc/pictures/compass-ci-architecture.png) -- **接入层** - - 成为衔接 Compass-CI 服务于开发者的纽带，同时提供内部设备管理的平台，提供 web 门户查阅测试结果、项目清单、ssh 环境登录。 - -- **服务层** - - 提供测试服务、环境登录、结果分析、辅助定界、用户项目注册能力，为本项目的主要功能。 - -- **支撑层** - - 作为项目主体功能实现，详细描述资源编排、部署、构建、测试等详细实现。 - -- **数据层** - - 为项目提供数据处理服务，包括测试结果格式化及归档。 - -- **资源层** - - 提供硬件设备，作为Compass-CI 服务部署的设备以及测试的物理设备、虚拟机、容器资源；提供测试过程中依赖软件仓库及依赖仓库、部署服务器。 - + 通过 [web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 -### 应用场景 - -- **聚合开发者测试用例** - - 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 +- **测试结果复现** -- **登录环境随时调测** - - 测试过程中，发现有 Bug 时，可随时提供调测资源服务，登录到环境进行复现、调试。 - -- **快照数据分析对比** - - 测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 - -- **辅助定界** + Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 - 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题的 Commit 信息。 - +- **辅助定位** + Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 ## 使用 Compass-CI > 您只需要先注册自己的仓库，当您的仓库有 commit 提交时，构建测试会自动执行，并且可在我们的网站中查看结果。 @@ -89,97 +47,25 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 web: https://compass-ci.openeuler.org/jobs +## Source +我们最新的，最完整的源码都存放在 [Gitee](https://gitee.com/wu_fengguang/compass-ci.git) 上，Fork 我们吧！ -## 高级功能 - -### 添加 OS 支持 - -制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 - -#### 操作步骤 - - 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） - - 通过 docker 获取 rootfs - - 1) 下载 openEuler 官方提供的 docker 镜像压缩包 - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) 加载 docker 镜像 - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) 启动 openEuler 容器 - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) 拷贝 docker 的 rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - 通过 qemu.img(qcow2格式)获取 rootfs (以 openEuler 为例) - - - 1) 下载 openEuler 官方网站提供的 qcow2 格式镜像 - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) 使用 {compass-ci}/container/qcow2rootfs 制作 rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. 定制 rootfs - 1. 使用 chroot 命令切换到 rootfs 中(此步骤需要 root 权限) - ```bash - chroot openEuler-rootfs - ``` - 2. 根据个人需要安装并配置服务 - - a. 修改 root 密码 - b. 配置 ssh 服务 - c. 检查系统时间 - d. 如果使用 docker 制作 osimage 还需要以下操作： - > 1. 安装所需版本内核 - > 2. 从 centos 官方网站下载内核 rpm 包 - > 3. 使用 yum 进行安装 - > 4. 删除 docker 环境变量文件 - - 3. 退出 rootfs，并打包 - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` -#### FAQ -1. 日志报错 “Unable to mount root fs on unknown-block” - - 问题现象 - ```bash - [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: - [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) - ... - [ 0.399404] Memory Limit: none - [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- - ``` - - 解决方法 - - 1）启动内存不足，增加内存可解决。 - 2）内核文件权限不足，给予 644 权限。 - -2. 系统运行缓慢 - - - 问题现象 +## Contributing to Compass-CI + +我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导 + +## Website - 打包镜像体积过大，会消耗很大内存。 - - 解决方法 +所有的测试结果，已加入的开源软件清单，历史测试查询比较都可以在我们的官网 [Website](https://compass-ci.openeuler.org) 上找到。 - 建议用户根据具体需要对 rootfs 进行裁剪。 +## 加入我们 +您可以通过以下的方式加入我们： + - 您可以加入我们的 [mailing list](https://mailweb.openeuler.org/postorius/lists/compass-ci.openeuler.or… + - 您可以加入我们每周四下午四点 16：00 的项目例会（会议链接将会通过邮件发送给您） -## 呼吁合作 +欢迎您跟我们一起： - 增强 git bisect 能力 - 增强数据分析能力 - 增强数据结果可视化能力 -- 2.23.0

1 0

[PATCH compass-ci] doc: how to apply account
by Wu Fengguang 30 Nov '20

30 Nov '20

- fix title - fix format - improve logic - fix broken link Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/manual/apply-account.md | 40 +++++++++++++++++++++++++++++++++ doc/manual/apply-ssh-account.md | 38 ------------------------------- 2 files changed, 40 insertions(+), 38 deletions(-) create mode 100644 doc/manual/apply-account.md delete mode 100644 doc/manual/apply-ssh-account.md diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md new file mode 100644 index 0000000..804b6ca --- /dev/null +++ b/doc/manual/apply-account.md @@ -0,0 +1,40 @@ +# apply account + +## step 1: send email to apply account + +Email template: + +--- + From: {{ David Rientjes <rientjes(a)google.com> }} + To: compass-ci(a)qq.com + Subject: apply account + + my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} + + {{ attach your ssh pubkey file(s), e.g. ~/.ssh/id_rsa.pub }} + +--- + +- please replace the 3 parts in {{ }} with your information. +- my oss commit: a git commit URL that has your contribution. + +## step 2: receive an email + +Which contains account information for you: + + - my_email + - my_name + - my_uuid + - SCHED_HOST + - SCHED_PORT + +## step 3: one-time setup + +Follow the email to setup your local environment: + + - clone git repos + - write ~/.config/compass-ci/default/account.yaml + +## step 4: try it out + +Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… diff --git a/doc/manual/apply-ssh-account.md b/doc/manual/apply-ssh-account.md deleted file mode 100644 index 160657e..0000000 --- a/doc/manual/apply-ssh-account.md +++ /dev/null @@ -1,38 +0,0 @@ -# apply account - -## steps -1. send apply account email - - send apply account email to: compass-ci(a)qq.com - attention: - email subject: - apply account - email body: - example: - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… - attachment: - ssh pubkey file - - - email example: - - To: compass-ci(a)qq.com - Subject: apply account - - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… - - attach your ssh pub key to the email as attachment - -2. receive email from compass-ci(a)qq.com - the email contains following information: - - my_email - - my_name - - my_uuid - - SCHED_HOST - - SCHED_PORT - -3. local environment configuration - follow steps in the email to finish the following configuration - - setup default yaml - ~/.config/compass-ci/default/account.yaml - -for more information: how to use submit - https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/compass-ci… -- 2.23.0

2 3

[PATCH v3 compass-ci] scheduler: fix the paths to find cluster config file
by Ren Wen 30 Nov '20

30 Nov '20

To find cluster config file at *lab-z9/cluster/* first, *lkp-tests/cluster/* otherwise. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 3 +++ src/scheduler/submit_job.cr | 14 +++++++++----- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1b37333..1afdda0 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -24,6 +24,9 @@ OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i3 SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" +CCI_REPOS = "/c" +LAB_REPO = "lab-z9" + INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c8d0875..7e1ac27 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -15,7 +15,7 @@ class Sched else cluster_config = get_cluster_config(cluster_file, job.lkp_initrd_user, - job.os_arch) + job.os_arch).not_nil! return submit_cluster_job(job, cluster_config) end rescue ex @@ -147,12 +147,16 @@ class Sched @es.set_job_content(job) end - # get cluster config using own lkp_src cluster file, - # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) - cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - return YAML.parse(File.read(cluster_file_path)) + + cluster_file_paths = [ + Path.new(CCI_REPOS, LAB_REPO, "cluster", cluster_file), + Path.new(lkp_src, "cluster", cluster_file) + ] + cluster_file_paths.each do |f| + return YAML.parse(File.read(f)) if File.file?(f) + end end def get_commit_date(job) -- 2.23.0

1 0

[PATCH lkp-tests] job.rb: disable submit warn
by Wei Jihui 30 Nov '20

30 Nov '20

[why] it has a warn when submit job: WARN -- skip non-executable */lkp-tests/monitors/plain/wait [how] lkp-tests/monitors/plain/wait is link to bin/event/wakeup but bin/event/wakeup does not exist. so we will do not check link file. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 8d1f7469..78604d64 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -115,6 +115,8 @@ def __create_programs_hash(glob, lkp_src) next if path =~ /\.[0-9]+$/ unless File.executable?(path) + next if File.symlink?(path) + log_warn "skip non-executable #{path}" unless path =~ /\.cr$/ next end -- 2.23.0

2 1

[PATCH v7 compass-ci] answerback-email: fix bugs and optimize
by Luan Shengde 30 Nov '20

30 Nov '20

1. set default option '-h' [why] the tool will show help message if no option specified the tool will get no email address without both '-e' and '-f' [how] add '-h' as default option if no option specified options will be reset to '-h' if both '-e' and '-f' are not specified 2. fix undefined method 'empty' [why] the original value is defined 'nil' for my_email when execute answerback-email with no '-e' or '-f', it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) [how] use '.nil?' instead of '.empty?' 3. option parse error [why] if add no required value for option, the option will use the field that follows as its value. example: ./answerback-email.rb -e -n name Not a standard format email: -n. [how] add value check for options that need a value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 49 ++++++++++++++++++-- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..825d815 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -80,7 +80,7 @@ end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|n] [--update]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [-l|--login y|n] [-u|--update]\n" opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' opts.banner += "[-g|--gen-sshkey] [--login y|n] [--update]\n" opts.banner += " -e|-f is required when applying account or updating account\n" @@ -94,19 +94,45 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + unless email_address =~ /[^@]+@[\d\w]+\.[\w\d]+/ + message = "Not a standard format email: #{email_address}.\n\n" + puts message + + return false + end + my_info['my_email'] = email_address end opts.on('-n name', '--name name', 'appoint name') do |name| + unless name =~ /^\w[\w\d ]+/ + message = "Name should only contains letters, digits and spaces\n\n" + puts message + + return false + end + stdin_info['my_name'] = name end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| + unless File.exist? pub_key_file + message = "File not found: #{pub_key_file}.\n\n" + puts message + + return false + end stdin_info['new_ssh_pubkey'] = File.read(pub_key_file).strip end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + unless File.exist? email_file + message = "Email file not found: #{email_file}.\n\n" + puts message + + return false + end mail_content = Mail.read(email_file) init_info(mail_content, email_info, my_info) end @@ -121,12 +147,16 @@ options = OptionParser.new do |opts| end opts.on('-l value', '--login value', 'enable/disable login, value: y|n') do |value| - if value.downcase == 'y' + case value + when 'y', 'Y' conf_info['enable_login'] = true - elsif value.downcase == 'n' + when 'n', 'N' conf_info['enable_login'] = false else - raise 'invalid parameter, please use y|n' + message = "-l: bad value #{value}, please use y|n\n\n" + puts message + + return false end end @@ -136,6 +166,15 @@ options = OptionParser.new do |opts| end end +# if no option specified, set default option '-h' +# if both '-e' and '-f' are not specified, will clear the ARGV, +# and use '-h' instead +if ARGV.empty? + ARGV << '-h' +elsif (['-e', '-f'] - ARGV).eql? ['-e', '-f'] + ARGV.clear + ARGV << '-h' +end options.parse!(ARGV) def apply_account(my_info, conf_info) @@ -152,7 +191,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 0

[PATCH v2 lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 30 Nov '20

30 Nov '20

[why] Originally submit_job is too complicated, it need to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit_spec.rb | 32 +++++++------------------------- 1 file changed, 7 insertions(+), 25 deletions(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..2ea689ec 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -20,10 +20,12 @@ def traverse_file(output_dir) end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*/*.yaml").each do |yaml_file| + next unless File.exist?(yaml_file) + + path = File.dirname(yaml_file) + output_dir = File.join(path, File.basename(yaml_file, '.yaml')) submit_cmd = [ "#{LKP_SRC}/sbin/submit", '-o', output_dir, @@ -36,24 +38,4 @@ def submit_job(submit_item) end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0

2 1

[PATCH v6 compass-ci] answerback-email: fix bugs and optimizi
by Luan Shengde 30 Nov '20

30 Nov '20

1. set default option -h for tool [why] the tool will show help message if no option specified the tool will get no email address without both '-e' and '-f' [how] add '-h' as default option if no option specified options will be reset to '-h' if both '-e' and '-f' are not specified 2. fix undefined method 'empty' [why] the original value is defined nil for my_email when execute answerback-email with no -e or -f, it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) [how] use '.nil?' instead of '.empty?' 3. option parse error [why] if add no required value for option, it will use the field that follow as the value, example: ./answerback-email.rb -e -n name Not a standard format email: -n. [how] add value check for options that need a value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 49 ++++++++++++++++++-- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..825d815 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -80,7 +80,7 @@ end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|n] [--update]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [-l|--login y|n] [-u|--update]\n" opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' opts.banner += "[-g|--gen-sshkey] [--login y|n] [--update]\n" opts.banner += " -e|-f is required when applying account or updating account\n" @@ -94,19 +94,45 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + unless email_address =~ /[^@]+@[\d\w]+\.[\w\d]+/ + message = "Not a standard format email: #{email_address}.\n\n" + puts message + + return false + end + my_info['my_email'] = email_address end opts.on('-n name', '--name name', 'appoint name') do |name| + unless name =~ /^\w[\w\d ]+/ + message = "Name should only contains letters, digits and spaces\n\n" + puts message + + return false + end + stdin_info['my_name'] = name end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| + unless File.exist? pub_key_file + message = "File not found: #{pub_key_file}.\n\n" + puts message + + return false + end stdin_info['new_ssh_pubkey'] = File.read(pub_key_file).strip end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + unless File.exist? email_file + message = "Email file not found: #{email_file}.\n\n" + puts message + + return false + end mail_content = Mail.read(email_file) init_info(mail_content, email_info, my_info) end @@ -121,12 +147,16 @@ options = OptionParser.new do |opts| end opts.on('-l value', '--login value', 'enable/disable login, value: y|n') do |value| - if value.downcase == 'y' + case value + when 'y', 'Y' conf_info['enable_login'] = true - elsif value.downcase == 'n' + when 'n', 'N' conf_info['enable_login'] = false else - raise 'invalid parameter, please use y|n' + message = "-l: bad value #{value}, please use y|n\n\n" + puts message + + return false end end @@ -136,6 +166,15 @@ options = OptionParser.new do |opts| end end +# if no option specified, set default option '-h' +# if both '-e' and '-f' are not specified, will clear the ARGV, +# and use '-h' instead +if ARGV.empty? + ARGV << '-h' +elsif (['-e', '-f'] - ARGV).eql? ['-e', '-f'] + ARGV.clear + ARGV << '-h' +end options.parse!(ARGV) def apply_account(my_info, conf_info) @@ -152,7 +191,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 0

[PATCH v3 compass-ci 4/4] user-client/maintain: send report mail
by Liu Yinsi 30 Nov '20

30 Nov '20

call $CCI_SRC/lib/mail_client.rb to mail. 'LOG' is a file => walk-test.report, job result will write in it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 42a8b97..2ec95ce 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,17 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def mail_report + message = File.read(LOG).chomp + data = " + subject: os rootfs test report + to: #{EMAIL_ADDRESS} + body: #{message}" + + email = MailClient.new + email.send_mail(data) +end + def write_report(report_all) File.open(LOG, 'a') do |f| log = Logger.new(f, 'weekly') -- 2.23.0

1 0

[PATCH v3 compass-ci 3/4] user-client/maintain: write job result into report
by Liu Yinsi 30 Nov '20

30 Nov '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 4f3bc88..42a8b97 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,16 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def write_report(report_all) + File.open(LOG, 'a') do |f| + log = Logger.new(f, 'weekly') + log.formatter = proc { |_severity, datetime, _progname, msg| + "[#{datetime}]: #{msg}\n" + } + log.info(report_all) + end +end + def monitor(query, actions, timeout) monitor = Monitor.new monitor.overrides = query -- 2.23.0

1 0

[PATCH v3 compass-ci 2/4] user-client/maintain: get job result by monitor and es query
by Liu Yinsi 30 Nov '20

30 Nov '20

1. monitor job, start time => job_state is 'boot', end time => job_state is 'extract_finished'; 2. query es by field 'job_state' and 'result_root'. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 091b01b..4f3bc88 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,46 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def monitor(query, actions, timeout) + monitor = Monitor.new + monitor.overrides = query + monitor.action = actions + monitor.run(timeout) + monitor +end + +def monitor_state(job_id) + time_list = [] + + %w[boot extract_finished].each do |state| + result_array = monitor({ 'job_id' => job_id, 'job_state' => state }, { 'stop' => true }, '1800').result + time_list.push(Time.parse(result_dict[0]['time'])) unless result_array.empty? + end + + time_list +end + +def es_search(job_id) + es_list = [] + + es = ESQuery.new + result_dict = es.query_by_id(job_id) + %w[job_state result_root].each do |field| + es_list.push(result_dict[field]) unless result_dict.nil? + end + + es_list +end + +def find_by_id(job_id) + start_time, end_time = monitor_state(job_id) + cost_time = end_time - start_time unless end_time.nil? + + job_state, result_root = es_search(job_id) + + return start_time, end_time, cost_time.to_i.floor, job_state, result_root +end + def run_qemu Process.fork do %x(#{ENV['CCI_SRC']}/providers/my-qemu.sh >/dev/null 2>&1) -- 2.23.0

1 0

[PATCH v2 compass-ci] service/scheduler: initialize a Sched for each API
by Wu Zhende 30 Nov '20

30 Nov '20

So can use Sched's instance variable in Sched's any interface Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 7 +++--- src/scheduler/find_job_boot.cr | 6 ++--- src/scheduler/find_next_job_boot.cr | 6 ++--- src/scheduler/request_cluster_state.cr | 14 ++++++------ src/scheduler/scheduler.cr | 31 +++++++++++++++----------- src/scheduler/submit_job.cr | 4 ++-- src/scheduler/update_job_parameter.cr | 6 ++--- 7 files changed, 40 insertions(+), 34 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 2be7d41..26ddd92 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -25,12 +25,13 @@ class Sched property redis property block_helper - def initialize + def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new @redis = Redis::Client.new @task_queue = TaskQueueAPI.new @block_helper = BlockHelper.new @rgc = RemoteGitClient.new + @env = env end def normalize_mac(mac : String) @@ -67,7 +68,7 @@ class Sched extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end - def update_tbox_wtmp(env : HTTP::Server::Context) + def update_tbox_wtmp testbox = "" hash = Hash(String, String).new @@ -75,7 +76,7 @@ class Sched hash["time"] = time %w(mac ip job_id tbox_name tbox_state).each do |parameter| - if (value = env.params.query[parameter]?) + if (value = @env.params.query[parameter]?) case parameter when "tbox_name" testbox = value diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b5a23c5..9805464 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -2,9 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def find_job_boot(env : HTTP::Server::Context) - value = env.params.url["value"] - boot_type = env.params.url["boot_type"] + def find_job_boot + value = @env.params.url["value"] + boot_type = @env.params.url["boot_type"] case boot_type when "ipxe" diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 807a9ae..09abbd5 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -2,9 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def find_next_job_boot(env) - hostname = env.params.query["hostname"]? - mac = env.params.query["mac"]? + def find_next_job_boot + hostname = @env.params.query["hostname"]? + mac = @env.params.query["mac"]? if !hostname && mac hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) end diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 72e3810..79a13cc 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -6,9 +6,9 @@ class Sched # all request states: # wait_ready | abort | failed | finished | wait_finish | # write_state | roles_ip - def request_cluster_state(env) - request_state = env.params.query["state"] - job_id = env.params.query["job_id"] + def request_cluster_state + request_state = @env.params.query["state"] + job_id = @env.params.query["job_id"] cluster_id = @redis.hash_get("sched/id2cluster", job_id).not_nil! cluster_state = "" @@ -41,10 +41,10 @@ class Sched return cluster_state when "write_state" - node_roles = env.params.query["node_roles"] - node_ip = env.params.query["ip"] - direct_ips = env.params.query["direct_ips"] - direct_macs = env.params.query["direct_macs"] + node_roles = @env.params.query["node_roles"] + node_ip = @env.params.query["ip"] + direct_ips = @env.params.query["direct_ips"] + direct_macs = @env.params.query["direct_macs"] job_info = {"roles" => node_roles, "ip" => node_ip, diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 5f97567..98b8e2e 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -30,12 +30,17 @@ require "../lib/sched" module Scheduler VERSION = "0.2.0" - sched = Sched.new + add_context_storage_type(Sched) before_all do |env| + env.set "sched", Sched.new(env) env.response.headers["Connection"] = "close" end + def self.get_sched_from_env(env) + env.get("sched").as(Sched) + end + # for debug (maybe kemal debug|logger does better) def self.debug_message(env, response) puts %({"from": "#{env.request.remote_address}", "response": #{response.to_json}}) @@ -53,7 +58,7 @@ module Scheduler # /boot.xxx/host/${hostname} # /boot.yyy/mac/${mac} get "/boot.:boot_type/:parameter/:value" do |env| - response = sched.find_job_boot(env) + response = get_sched_from_env(env).find_job_boot job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? puts %({"job_id": "#{job_id}", "job_state": "boot"}) if job_id @@ -63,7 +68,7 @@ module Scheduler # /~lkp/cgi-bin/gpxelinux.cgi?hostname=:hostname&mac=:mac&last_kernel=:last_kernel get "/~lkp/cgi-bin/gpxelinux.cgi" do |env| - response = sched.find_next_job_boot(env) + response = get_sched_from_env(env).find_next_job_boot job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? puts %({"job_id": "#{job_id}", "job_state": "boot"}) if job_id @@ -75,7 +80,7 @@ module Scheduler # - echo job_id to caller # -- job_id = "0" ? means failed post "/submit_job" do |env| - job_messages = sched.submit_job(env) + job_messages = get_sched_from_env(env).submit_job job_messages.each do |job_message| puts job_message.to_json @@ -103,7 +108,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/set_host_mac?hostname=wfg&mac=00-01-02-03-04-05" put "/set_host_mac" do |env| if (client_hostname = env.params.query["hostname"]?) && (client_mac = env.params.query["mac"]?) - sched.set_host_mac(client_mac, client_hostname) + get_sched_from_env(env).set_host_mac(client_mac, client_hostname) "Done" else @@ -114,7 +119,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" put "/set_host2queues" do |env| if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) - sched.set_host2queues(client_host, client_queues) + get_sched_from_env(env).set_host2queues(client_host, client_queues) "Done" else @@ -125,7 +130,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| if client_mac = env.params.query["mac"]? - sched.del_host_mac(client_mac) + get_sched_from_env(env).del_host_mac(client_mac) "Done" else @@ -136,7 +141,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" put "/del_host2queues" do |env| if client_host = env.params.query["host"]? - sched.del_host2queues(client_host) + get_sched_from_env(env).del_host2queues(client_host) "Done" else @@ -150,7 +155,7 @@ module Scheduler # ?job_file=/lkp/scheduled/job.yaml&job_state=post_run&job_id=10 # ?job_file=/lkp/scheduled/job.yaml&loadavg=0.28 0.82 0.49 1/105 3389&start_time=1587725398&end_time=1587725698&job_id=10 get "/~lkp/cgi-bin/lkp-jobfile-append-var" do |env| - puts sched.update_job_parameter(env) + puts get_sched_from_env(env).update_job_parameter "Done" end @@ -173,7 +178,7 @@ module Scheduler # response: get "server ip" from cluster state, # return "server=<server ip>". get "/~lkp/cgi-bin/lkp-cluster-sync" do |env| - response = sched.request_cluster_state(env) + response = get_sched_from_env(env).request_cluster_state debug_message(env, response) @@ -187,14 +192,14 @@ module Scheduler # get job_id from request job_id = env.params.query["job_id"]? if job_id - puts sched.close_job(job_id) + puts get_sched_from_env(env).close_job(job_id) end "Done" end get "/~lkp/cgi-bin/lkp-wtmp" do |env| - puts sched.update_tbox_wtmp(env) + puts get_sched_from_env(env).update_tbox_wtmp "Done" end @@ -205,7 +210,7 @@ module Scheduler job_id = env.params.query["job_id"].to_s if testbox && ssh_port - sched.report_ssh_port(testbox, ssh_port) + get_sched_from_env(env).report_ssh_port(testbox, ssh_port) end puts %({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"}) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..4e78102 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -2,8 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def submit_job(env : HTTP::Server::Context) - body = env.request.body.not_nil!.gets_to_end + def submit_job + body = @env.request.body.not_nil!.gets_to_end job_content = JSON.parse(body) job = Job.new(job_content, job_content["id"]?) diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index 0692025..cee8b9b 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -2,8 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def update_job_parameter(env : HTTP::Server::Context) - job_id = env.params.query["job_id"]? + def update_job_parameter + job_id = @env.params.query["job_id"]? if !job_id return false end @@ -13,7 +13,7 @@ class Sched job_content["id"] = job_id (%w(start_time end_time loadavg job_state)).each do |parameter| - value = env.params.query[parameter]? + value = @env.params.query[parameter]? if !value || value == "" next end -- 2.23.0

2 1

[PATCH v5 compass-ci] answerback-email: fix bugs and optimizi
by Luan Shengde 30 Nov '20

30 Nov '20

1. set default option -h for tool [why] the tool will show help message if no option specified the tool will get no email address without both '-e' and '-f' [how] add '-h' as default option if no option specified options will be reset to '-h' if both '-e' and '-f' are not specified 2. fix undefined method 'empty' [why] the original value is defined nil for my_email when execute answerback-email with no -e or -f, it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) [how] use '.nil?' instead of '.empty?' 3. option parse error [why] if add no required value for option, it will use the field that follow as the value, example: ./answerback-email.rb -e -n name Not a standard format email: -n. [how] add value check for options that need a value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 51 +++++++++++++++++--- 1 file changed, 45 insertions(+), 6 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..ec1f5b7 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -80,7 +80,7 @@ end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|n] [--update]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [-l|--login y|n] [-u|--update]\n" opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' opts.banner += "[-g|--gen-sshkey] [--login y|n] [--update]\n" opts.banner += " -e|-f is required when applying account or updating account\n" @@ -94,19 +94,45 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + unless email_address =~ /[^@]+@[\d\w]+\.[\w\d]+/ + message = "Not a standard format email: #{email_address}.\n\n" + puts message + + return false + end + my_info['my_email'] = email_address end opts.on('-n name', '--name name', 'appoint name') do |name| + unless name =~ /^\w[\w\d ]+/ + message = "Name should only contains letters, digits and spaces\n\n" + puts message + + return false + end + stdin_info['my_name'] = name end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| + unless File.exist? pub_key_file + message = "File not found: #{pub_key_file}.\n\n" + puts message + + return false + end stdin_info['new_ssh_pubkey'] = File.read(pub_key_file).strip end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + unless File.exist? email_file + message = "Email file not found: #{email_file}.\n\n" + puts message + + return false + end mail_content = Mail.read(email_file) init_info(mail_content, email_info, my_info) end @@ -121,12 +147,16 @@ options = OptionParser.new do |opts| end opts.on('-l value', '--login value', 'enable/disable login, value: y|n') do |value| - if value.downcase == 'y' + case value + when 'y', 'Y' conf_info['enable_login'] = true - elsif value.downcase == 'n' + when 'n', 'N' conf_info['enable_login'] = false else - raise 'invalid parameter, please use y|n' + message = "-l: bad value #{value}, please use y|n\n\n" + puts message + + return false end end @@ -136,6 +166,15 @@ options = OptionParser.new do |opts| end end +# if no option specified, set default option '-h' +# if both '-e' and '-f' are not specified, will clear the ARGV, +# and use '-h' instead +if ARGV.empty? + ARGV << '-h' +elsif (['-e', '-f'] - ARGV).eql? ['-e', '-f'] + ARGV.clear + ARGV << '-h' +end options.parse!(ARGV) def apply_account(my_info, conf_info) @@ -152,7 +191,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) @@ -210,4 +249,4 @@ def store_account_info(my_info) es.put_source_by_id(my_info['my_email'], my_info) end -send_account(my_info, conf_info, email_info, my_info_es, stdin_info) +# send_account(my_info, conf_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 1

[PATCH compass-ci 4/5] user-client/maintain: write job result into report
by Liu Yinsi 30 Nov '20

30 Nov '20

close the file every time after open and write it, avoid duplicate printing of logs. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index d9d6bd6..7ed07c5 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -34,6 +34,16 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def write_report(report_all) + file = File.open(LOG, 'a') + log = Logger.new(file, 'weekly') + log.formatter = proc { |_severity, datetime, _progname, msg| + "[#{datetime}]: #{msg}\n" + } + log.info(report_all) + file.close +end + def monitor(query, actions, timeout) monitor = Monitor.new monitor.overrides = query -- 2.23.0

3 6

Re: [PATCH lkp-tests] job.rb: disable submit warn
by Ren Wen 30 Nov '20

30 Nov '20

> unless File.executable?(path) >+ next if File.symlink?(path) >+ Skip all symlinks here? How about skipping 'wait' only? Thanks, RenWen > log_warn "skip non-executable #{path}" unless path =~ /\.cr$/ > next > end >-- >2.23.0 >

1 0

[PATCH compass-ci] doc/job: add memory.md
by Zhang Yu 30 Nov '20

30 Nov '20

Add a document for introducing the memory field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/memory.md | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 doc/job/memory.md diff --git a/doc/job/memory.md b/doc/job/memory.md new file mode 100644 index 0000000..d45edb5 --- /dev/null +++ b/doc/job/memory.md @@ -0,0 +1,10 @@ +# memory + +Meaning: +- Memory exists on the machine, it is the memory function of the machine. +- The memory field comes from the file which in $LAB_SRC/hosts. +- When we run a job in compass we need to know the machine's memory first, + because the memory required for running the job must be smaller than the + actual memory of the machine, otherwise there will be an error. +- Here are some examples of the memory: + 8G, 16G, 128G ... -- 2.23.0

1 0

[PATCH v2 lkp-tests] distro/common: fix cgz file not generated
by Wang Yong 30 Nov '20

30 Nov '20

if packages have already installed in OS, cgz file should not be generated, so cgz file won't be uploaded Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/common | 2 ++ 1 file changed, 2 insertions(+) diff --git a/distro/common b/distro/common index cdb531403..166ff8550 100755 --- a/distro/common +++ b/distro/common @@ -89,5 +89,7 @@ upload_to_target_dir() local dir="$1" local file="$(ls -F ${dir} | grep -v "@")" + [ -z "${file}" ] && return 0 + upload_one_curl "${dir}/${file}" "${dir}" >/dev/null } -- 2.23.0

1 0

[PATCH lkp-tests] distro/common: fix cgz file not generated
by Wang Yong 30 Nov '20

30 Nov '20

if packages have already installed in OS, cci-depends cgz file should not be generated, so cgz file won't be uploaded Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/common | 2 ++ 1 file changed, 2 insertions(+) diff --git a/distro/common b/distro/common index cdb531403..166ff8550 100755 --- a/distro/common +++ b/distro/common @@ -89,5 +89,7 @@ upload_to_target_dir() local dir="$1" local file="$(ls -F ${dir} | grep -v "@")" + [ -z "${file}" ] && return 0 + upload_one_curl "${dir}/${file}" "${dir}" >/dev/null } -- 2.23.0

1 1

[PATCH compass-ci] service/scheduler: initialize a Sched for each API
by Wu Zhende 30 Nov '20

30 Nov '20

So can use Sched's class variable in Sched's any interfacce Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 7 +++--- src/scheduler/find_job_boot.cr | 6 ++--- src/scheduler/find_next_job_boot.cr | 6 ++--- src/scheduler/request_cluster_state.cr | 14 ++++++------ src/scheduler/scheduler.cr | 31 +++++++++++++++----------- src/scheduler/submit_job.cr | 4 ++-- src/scheduler/update_job_parameter.cr | 6 ++--- 7 files changed, 40 insertions(+), 34 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 2be7d41..26ddd92 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -25,12 +25,13 @@ class Sched property redis property block_helper - def initialize + def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new @redis = Redis::Client.new @task_queue = TaskQueueAPI.new @block_helper = BlockHelper.new @rgc = RemoteGitClient.new + @env = env end def normalize_mac(mac : String) @@ -67,7 +68,7 @@ class Sched extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end - def update_tbox_wtmp(env : HTTP::Server::Context) + def update_tbox_wtmp testbox = "" hash = Hash(String, String).new @@ -75,7 +76,7 @@ class Sched hash["time"] = time %w(mac ip job_id tbox_name tbox_state).each do |parameter| - if (value = env.params.query[parameter]?) + if (value = @env.params.query[parameter]?) case parameter when "tbox_name" testbox = value diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b5a23c5..9805464 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -2,9 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def find_job_boot(env : HTTP::Server::Context) - value = env.params.url["value"] - boot_type = env.params.url["boot_type"] + def find_job_boot + value = @env.params.url["value"] + boot_type = @env.params.url["boot_type"] case boot_type when "ipxe" diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 807a9ae..09abbd5 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -2,9 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def find_next_job_boot(env) - hostname = env.params.query["hostname"]? - mac = env.params.query["mac"]? + def find_next_job_boot + hostname = @env.params.query["hostname"]? + mac = @env.params.query["mac"]? if !hostname && mac hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) end diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 72e3810..79a13cc 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -6,9 +6,9 @@ class Sched # all request states: # wait_ready | abort | failed | finished | wait_finish | # write_state | roles_ip - def request_cluster_state(env) - request_state = env.params.query["state"] - job_id = env.params.query["job_id"] + def request_cluster_state + request_state = @env.params.query["state"] + job_id = @env.params.query["job_id"] cluster_id = @redis.hash_get("sched/id2cluster", job_id).not_nil! cluster_state = "" @@ -41,10 +41,10 @@ class Sched return cluster_state when "write_state" - node_roles = env.params.query["node_roles"] - node_ip = env.params.query["ip"] - direct_ips = env.params.query["direct_ips"] - direct_macs = env.params.query["direct_macs"] + node_roles = @env.params.query["node_roles"] + node_ip = @env.params.query["ip"] + direct_ips = @env.params.query["direct_ips"] + direct_macs = @env.params.query["direct_macs"] job_info = {"roles" => node_roles, "ip" => node_ip, diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 5f97567..98b8e2e 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -30,12 +30,17 @@ require "../lib/sched" module Scheduler VERSION = "0.2.0" - sched = Sched.new + add_context_storage_type(Sched) before_all do |env| + env.set "sched", Sched.new(env) env.response.headers["Connection"] = "close" end + def self.get_sched_from_env(env) + env.get("sched").as(Sched) + end + # for debug (maybe kemal debug|logger does better) def self.debug_message(env, response) puts %({"from": "#{env.request.remote_address}", "response": #{response.to_json}}) @@ -53,7 +58,7 @@ module Scheduler # /boot.xxx/host/${hostname} # /boot.yyy/mac/${mac} get "/boot.:boot_type/:parameter/:value" do |env| - response = sched.find_job_boot(env) + response = get_sched_from_env(env).find_job_boot job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? puts %({"job_id": "#{job_id}", "job_state": "boot"}) if job_id @@ -63,7 +68,7 @@ module Scheduler # /~lkp/cgi-bin/gpxelinux.cgi?hostname=:hostname&mac=:mac&last_kernel=:last_kernel get "/~lkp/cgi-bin/gpxelinux.cgi" do |env| - response = sched.find_next_job_boot(env) + response = get_sched_from_env(env).find_next_job_boot job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? puts %({"job_id": "#{job_id}", "job_state": "boot"}) if job_id @@ -75,7 +80,7 @@ module Scheduler # - echo job_id to caller # -- job_id = "0" ? means failed post "/submit_job" do |env| - job_messages = sched.submit_job(env) + job_messages = get_sched_from_env(env).submit_job job_messages.each do |job_message| puts job_message.to_json @@ -103,7 +108,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/set_host_mac?hostname=wfg&mac=00-01-02-03-04-05" put "/set_host_mac" do |env| if (client_hostname = env.params.query["hostname"]?) && (client_mac = env.params.query["mac"]?) - sched.set_host_mac(client_mac, client_hostname) + get_sched_from_env(env).set_host_mac(client_mac, client_hostname) "Done" else @@ -114,7 +119,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" put "/set_host2queues" do |env| if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) - sched.set_host2queues(client_host, client_queues) + get_sched_from_env(env).set_host2queues(client_host, client_queues) "Done" else @@ -125,7 +130,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| if client_mac = env.params.query["mac"]? - sched.del_host_mac(client_mac) + get_sched_from_env(env).del_host_mac(client_mac) "Done" else @@ -136,7 +141,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" put "/del_host2queues" do |env| if client_host = env.params.query["host"]? - sched.del_host2queues(client_host) + get_sched_from_env(env).del_host2queues(client_host) "Done" else @@ -150,7 +155,7 @@ module Scheduler # ?job_file=/lkp/scheduled/job.yaml&job_state=post_run&job_id=10 # ?job_file=/lkp/scheduled/job.yaml&loadavg=0.28 0.82 0.49 1/105 3389&start_time=1587725398&end_time=1587725698&job_id=10 get "/~lkp/cgi-bin/lkp-jobfile-append-var" do |env| - puts sched.update_job_parameter(env) + puts get_sched_from_env(env).update_job_parameter "Done" end @@ -173,7 +178,7 @@ module Scheduler # response: get "server ip" from cluster state, # return "server=<server ip>". get "/~lkp/cgi-bin/lkp-cluster-sync" do |env| - response = sched.request_cluster_state(env) + response = get_sched_from_env(env).request_cluster_state debug_message(env, response) @@ -187,14 +192,14 @@ module Scheduler # get job_id from request job_id = env.params.query["job_id"]? if job_id - puts sched.close_job(job_id) + puts get_sched_from_env(env).close_job(job_id) end "Done" end get "/~lkp/cgi-bin/lkp-wtmp" do |env| - puts sched.update_tbox_wtmp(env) + puts get_sched_from_env(env).update_tbox_wtmp "Done" end @@ -205,7 +210,7 @@ module Scheduler job_id = env.params.query["job_id"].to_s if testbox && ssh_port - sched.report_ssh_port(testbox, ssh_port) + get_sched_from_env(env).report_ssh_port(testbox, ssh_port) end puts %({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"}) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..4e78102 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -2,8 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def submit_job(env : HTTP::Server::Context) - body = env.request.body.not_nil!.gets_to_end + def submit_job + body = @env.request.body.not_nil!.gets_to_end job_content = JSON.parse(body) job = Job.new(job_content, job_content["id"]?) diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index 0692025..cee8b9b 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -2,8 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def update_job_parameter(env : HTTP::Server::Context) - job_id = env.params.query["job_id"]? + def update_job_parameter + job_id = @env.params.query["job_id"]? if !job_id return false end @@ -13,7 +13,7 @@ class Sched job_content["id"] = job_id (%w(start_time end_time loadavg job_state)).each do |parameter| - value = env.params.query[parameter]? + value = @env.params.query[parameter]? if !value || value == "" next end -- 2.23.0

3 2

[PATCH v6 compass-ci 3/3] openeuler: configure files about build image with one command
by Wang Chenglong 30 Nov '20

30 Nov '20

[Contains] openeuler-${os_version}.repo: yum configuration files files-to-exclude: list of files that need to be excluded packages-to-install: list of packages that need to install README.md: descriptive documentation Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/README.md | 15 +++++++ container/osimage/openeuler/files-to-exclude | 11 +++++ .../openeuler/openeuler-20.03-lts.repo | 43 +++++++++++++++++++ .../osimage/openeuler/openeuler-20.09.repo | 43 +++++++++++++++++++ .../osimage/openeuler/packages-to-install | 10 +++++ 5 files changed, 122 insertions(+) create mode 100644 container/osimage/openeuler/README.md create mode 100644 container/osimage/openeuler/files-to-exclude create mode 100644 container/osimage/openeuler/openeuler-20.03-lts.repo create mode 100644 container/osimage/openeuler/openeuler-20.09.repo create mode 100644 container/osimage/openeuler/packages-to-install diff --git a/container/osimage/openeuler/README.md b/container/osimage/openeuler/README.md new file mode 100644 index 0000000..cea68f9 --- /dev/null +++ b/container/osimage/openeuler/README.md @@ -0,0 +1,15 @@ +# Use the tool to create a new openeuler-${os_version} initramfs image. + +Usage: + cd ${CCI_SRC}/rootfs/initramfs/openeuler/aarch64/${os_version}/ + ./build + +Some configuration items: +./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + +./files-to-exclude + If you want remove some unnecessary files, you can write the names in ./files-to-exclude + +$HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. diff --git a/container/osimage/openeuler/files-to-exclude b/container/osimage/openeuler/files-to-exclude new file mode 100644 index 0000000..3854947 --- /dev/null +++ b/container/osimage/openeuler/files-to-exclude @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/files-to-exclude +/tmp/packages-to-install diff --git a/container/osimage/openeuler/openeuler-20.03-lts.repo b/container/osimage/openeuler/openeuler-20.03-lts.repo new file mode 100644 index 0000000..f80b6b6 --- /dev/null +++ b/container/osimage/openeuler/openeuler-20.03-lts.repo @@ -0,0 +1,43 @@ +# generic-repos is licensed under the Mulan PSL v2. +# You can use this software according to the terms and conditions of the Mulan PSL v2. +# You may obtain a copy of Mulan PSL v2 at: +# http://license.coscl.org.cn/MulanPSL2 +# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR +# PURPOSE. +# See the Mulan PSL v2 for more details. + +[OS] +name=OS +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/OS/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/OS/$basearch/RPM-GPG-KEY-openEuler + +[everything] +name=everything +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/everything/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/everything/$basearch/RPM-GPG-KEY-openEuler + +[EPOL] +name=EPOL +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/EPOL/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/EPOL/$basearch/RPM-GPG-KEY-openEuler + +[debuginfo] +name=debuginfo +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/debuginfo/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/debuginfo/$basearch/RPM-GPG-KEY-openEuler + +[source] +name=source +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/source/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/source/RPM-GPG-KEY-openEuler diff --git a/container/osimage/openeuler/openeuler-20.09.repo b/container/osimage/openeuler/openeuler-20.09.repo new file mode 100644 index 0000000..6685b08 --- /dev/null +++ b/container/osimage/openeuler/openeuler-20.09.repo @@ -0,0 +1,43 @@ +# generic-repos is licensed under the Mulan PSL v2. +# You can use this software according to the terms and conditions of the Mulan PSL v2. +# You may obtain a copy of Mulan PSL v2 at: +# http://license.coscl.org.cn/MulanPSL2 +# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR +# PURPOSE. +# See the Mulan PSL v2 for more details. + +[OS] +name=OS +baseurl=http://repo.openeuler.org/openEuler-20.09/OS/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/OS/$basearch/RPM-GPG-KEY-openEuler + +[everything] +name=everything +baseurl=http://repo.openeuler.org/openEuler-20.09/everything/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/everything/$basearch/RPM-GPG-KEY-openEuler + +[EPOL] +name=EPOL +baseurl=http://repo.openeuler.org/openEuler-20.09/EPOL/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/EPOL/$basearch/RPM-GPG-KEY-openEuler + +[debuginfo] +name=debuginfo +baseurl=http://repo.openeuler.org/openEuler-20.09/debuginfo/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/debuginfo/$basearch/RPM-GPG-KEY-openEuler + +[source] +name=source +baseurl=http://repo.openeuler.org/openEuler-20.09/source/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/source/RPM-GPG-KEY-openEuler diff --git a/container/osimage/openeuler/packages-to-install b/container/osimage/openeuler/packages-to-install new file mode 100644 index 0000000..f17eee1 --- /dev/null +++ b/container/osimage/openeuler/packages-to-install @@ -0,0 +1,10 @@ +cpio +systemd-udev.aarch64 +cifs-utils +nfs-utils +hostname +curl +openssh-server +openssl +iputils +wget -- 2.23.0

1 0

[PATCH v6 compass-ci 2/3] openeuler: build openeuler initramfs image with one command
by Wang Chenglong 30 Nov '20

30 Nov '20

[Why] Use the tool to create a new openeuler-$version initramfs image with one command, and we also can customize the pre-installed software by configuring ./packages-to-install. [How] We create a link button for the user to use. $CCI_SRC/rootfs/initramfs/openeuler/aarch64/${os_version}/build --- container/osimage/openeuler/run | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 container/osimage/openeuler/run diff --git a/container/osimage/openeuler/run b/container/osimage/openeuler/run new file mode 100755 index 0000000..5d7da72 --- /dev/null +++ b/container/osimage/openeuler/run @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/openeuler/lib +. ${CCI_SRC}/container/defconfig.sh + +export_root_passwd +DIR=$(dirname $(realpath $0)) + +cmd=( + docker run + --name initramfs_${image} + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/files-to-exclude:/tmp/files-to-exclude + -v $DIR/${image}.repo:/etc/yum.repos.d/openEuler.repo + -e ROOT_PASSWD=$ROOT_PASSWD + -e IMAGE_NAME=$IMAGE_NAME + $image + /root/bin/create-image +) + +load_docker_image +"${cmd[@]}" +cp_image_to_host +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

1 0