- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 2/2] modify readme.md
by Li Ping 01 Dec '20

01 Dec '20

--- README.md | 42 ++++++++++++++++++++++++------------------ 1 file changed, 24 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 128eba0..314a8c6 100644 --- a/README.md +++ b/README.md @@ -1,52 +1,58 @@ -## Compass-CI +## Compass-CI -### 关于Compass-CI +### 关于 Compass-CI -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务--。Compass-CI 基于开源软件PR进行自动化测试(包括构建测试，软件包自带用例测试等)，共同构建一个开放、完整的开源软件生态测试系统。 +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。Compass-CI 基于开源软件 PR 进行自动化测试(包括构建测试，软件包自带用例测试等)，共同构建一个开放、完整的开源软件生态测试系统。 ### 功能介绍 -- **测试服务** +**测试服务** -使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… +使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… job](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submi… -- **调测环境登录** +**调测环境登录** 使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- **测试结果分析** +**测试结果分析** 通过 [Web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 -- **测试结果复现** +**测试结果复现** -Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 +通过历史测试生成 job.yaml 文件，重新提交 job.yaml 还原测试环境，复现测试结果。 -- **辅助定位** +**辅助定位** -Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 +Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的 commit。 ## Getting started -- **自动化测试** +**自动化测试** 1. 添加待测试仓库 URL 到 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos.git) 仓库，[编写测试用例并添加到仓库](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/…, 详细流程请查看[这篇文档](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/man… 2. git push 更新仓库，自动触发测试。 -3. 在网页中搜索并查看结果 web: https://compass-ci.openeuler.org/jobs +3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%… web: https://compass-ci.openeuler.org/jobs。 + +**手动提交测试任务** -- **手动提交测试任务** +1. [安装 Compass-CI 客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9… +2. [编写测试用例](https/blob/master/doc/add-testcase.md), [手动提交测试任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… +3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%… web: https://compass-ci.openeuler.org/jobs。 -1. [安装Compass-CI客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… -2. [编写测试用例](https/blob/master/doc/add-testcase.md), [手动提交测试任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… -3. 在网页中搜索并查看结果 web: https://compass-ci.openeuler.org/jobs +**登录测试环境** + +1. 向 Compass-CI(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… +2. 根据邮件反馈内容[完成环境配置](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… +3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… ## Contributing to Compass-CI -我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导，Compass-CI 主要是使用 Ruby 和 Crystal 开发的一个项目，我们使用 rubocop和 ameba 来规范我d的代码风格，具体说明可以查看 [rococop源码仓](https://github.com/rubocop-hq/rubocop)和[ameba源码仓](https://githu… +我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导，Compass-CI 主要是使用 Ruby 开发的一个项目，我们遵循 [Ruby 社区代码风格](https://ruby-china.org/wiki/coding-style)。如果您想参与社区并为 Compass-CI 项目做出贡献，[这个页面](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/lea… Compass-CI 所使用的所有语言和工具等。 ## Website -- 2.23.0

1 0

[PATCH compass-ci] delete source and add getting started
by Li Ping 01 Dec '20

01 Dec '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.md | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 80e0766..128eba0 100644 --- a/README.md +++ b/README.md @@ -10,50 +10,43 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 - **测试服务** - 使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… +使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - **调测环境登录** - 使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - **测试结果分析** - 通过 [web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 +通过 [Web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 - **测试结果复现** - Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 +Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 - **辅助定位** - Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 +Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 -## 使用 Compass-CI -> 您只需要先注册自己的仓库，当您的仓库有 commit 提交时，构建测试会自动执行，并且可在我们的网站中查看结果。 +## Getting started -- 注册自己的仓库 +- **自动化测试** - 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 - ```bash - git clone https://gitee.com/wu_fengguang/upstream-repos.git - less upstream-repos/README.md - ``` +1. 添加待测试仓库 URL 到 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos.git) 仓库，[编写测试用例并添加到仓库](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/…, 详细流程请查看[这篇文档](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/man… -- `git push` - - 更新仓库，自动触发测试。 +2. git push 更新仓库，自动触发测试。 -- 在网页中搜索并查看结果 - - web: https://compass-ci.openeuler.org/jobs +3. 在网页中搜索并查看结果 web: https://compass-ci.openeuler.org/jobs -## Source +- **手动提交测试任务** -我们最新的，最完整的源码都存放在 [Gitee](https://gitee.com/wu_fengguang/compass-ci.git) 上，Fork 我们吧！ +1. [安装Compass-CI客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… +2. [编写测试用例](https/blob/master/doc/add-testcase.md), [手动提交测试任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… +3. 在网页中搜索并查看结果 web: https://compass-ci.openeuler.org/jobs ## Contributing to Compass-CI -我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导 +我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导，Compass-CI 主要是使用 Ruby 和 Crystal 开发的一个项目，我们使用 rubocop和 ameba 来规范我d的代码风格，具体说明可以查看 [rococop源码仓](https://github.com/rubocop-hq/rubocop)和[ameba源码仓](https://githu… ## Website -- 2.23.0

1 1

[PATCH lkp-tests] sbin/submit: fix option help info
by Wu Zhende 01 Dec '20

01 Dec '20

The user does not know what jobs is. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index cc2d940c..6766e704 100755 --- a/sbin/submit +++ b/sbin/submit @@ -20,7 +20,7 @@ opt_my_queue = false actions = ['output', 'stop'] options = OptionParser.new do |opts| - opts.banner = 'Usage: submit [options] jobs...' + opts.banner = 'Usage: submit [options] job1.yaml job2.yaml ...' opts.separator ' submit test jobs to the scheduler' opts.separator '' -- 2.23.0

1 0

[PATCH v3 compass-ci] lib/compare_matrixes.rb: refactor format optimization
by Lu Kaiyi 01 Dec '20

01 Dec '20

reduce unnecessary indent to simplify code Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d8112e9..7a9870a 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -177,7 +177,7 @@ def get_matrixes_values(matrixes_list, options) end def remove_unchanged_field(matrixes_values) - # remove unchanged field from matrixes valus and remove :changed key + # remove unchanged field from matrixes values and remove :changed key # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| @@ -316,21 +316,20 @@ def get_decimal_length(number, length) end def get_suitable_number_str(number, length, format_pattern) - # if number string length can't < target length, + # if number string length is no less than target length, # transform number string to scientific notation string format_str = format(format_pattern, number) return format_str if format_str.length <= length decimal_length = get_decimal_length(number, length) - unless decimal_length.negative? - scientific_str = format("%.#{decimal_length}e", number).sub('e+0', 'e+').sub('e-0', 'e-') - lack_length = length - scientific_str.length - unless lack_length.negative? - return scientific_str + ' ' * lack_length - end - end - format_str + return format_str if decimal_length.negative? + + scientific_str = format("%.#{decimal_length}e", number).sub('e+0', 'e+').sub('e-0', 'e-') + lack_length = length - scientific_str.length + return format_str if lack_length.negative? + + return scientific_str + ' ' * lack_length end # Colorize -- 2.23.0

1 0

[PATCH lkp-tests 2/2] depends/*: delete redundant packages
by Wang Yong 01 Dec '20

01 Dec '20

these packages will install in cci-makepkg/makepkg at distro_install_depends lkp-dev Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/cci-makepkg | 4 ---- distro/depends/makepkg | 4 ---- 2 files changed, 8 deletions(-) diff --git a/distro/depends/cci-makepkg b/distro/depends/cci-makepkg index a7ee67537..4ed4f19e9 100644 --- a/distro/depends/cci-makepkg +++ b/distro/depends/cci-makepkg @@ -1,10 +1,6 @@ curl -libarchive-tools -bzip2 -fakeroot gnupg gettext -openssl ncurses-bin binutils cifs-utils diff --git a/distro/depends/makepkg b/distro/depends/makepkg index aecd50e0f..781c93526 100644 --- a/distro/depends/makepkg +++ b/distro/depends/makepkg @@ -1,9 +1,5 @@ curl -libarchive-tools -bzip2 -fakeroot gnupg gettext -openssl ncurses-bin binutils -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: add 本地安装compass-ci客户端.md
by Li Ping 30 Nov '20

30 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- ...ci\345\256\242\346\210\267\347\253\257.md" | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 "doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" new file mode 100644 index 0000000..43a6bf5 --- /dev/null +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -0,0 +1,28 @@ +# 本地安装compass-ci客户端 + +Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客户端，通过本地安装 lkp-tests 可以手动提交测试任务 + +:exclamation: 前提条件：按照 [apply-ssh-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master… 完成如下动作： +- send apply account email +- receive email from compass-ci(a)qq.com + +## Getting started + +1. setup default config + run the following command to add the below setup to default config file + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF + my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx + my_email: xxx + my_name: xx + EOF + +2. download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ~/.bashrc && source ~/.bash_profile + +3. submit job + Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: add 本地暗转compass-ci客户端.md
by Li Ping 30 Nov '20

30 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- ...ci\345\256\242\346\210\267\347\253\257.md" | 28 +++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 "doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" diff --git "a/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" new file mode 100644 index 0000000..43a6bf5 --- /dev/null +++ "b/doc/manual/\346\234\254\345\234\260\345\256\211\350\243\205compass-ci\345\256\242\346\210\267\347\253\257.md" @@ -0,0 +1,28 @@ +# 本地安装compass-ci客户端 + +Compass-CI 将 [lkp-tests](https://gitee.com/wu_fengguang/lkp-tests) 作为客户端，通过本地安装 lkp-tests 可以手动提交测试任务 + +:exclamation: 前提条件：按照 [apply-ssh-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master… 完成如下动作： +- send apply account email +- receive email from compass-ci(a)qq.com + +## Getting started + +1. setup default config + run the following command to add the below setup to default config file + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/${USER}.yaml <<-EOF + my_uuid: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx + my_email: xxx + my_name: xx + EOF + +2. download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ~/.bashrc && source ~/.bash_profile + +3. submit job + Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 1

[PATCH v2 lkp-tests] job.rb: disable submit warn
by Wei Jihui 30 Nov '20

30 Nov '20

[why] it has a warn when submit job: WARN -- skip non-executable */lkp-tests/monitors/plain/wait [how] lkp-tests/monitors/plain/wait is link to bin/event/wakeup but bin/event/wakeup does not exist. so we will do not check link file when it real file is not exist. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 8d1f7469..f9e244a9 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -115,6 +115,11 @@ def __create_programs_hash(glob, lkp_src) next if path =~ /\.[0-9]+$/ unless File.executable?(path) + if File.symlink?(path) + real_path = File.readlink(path) + real_path = File.join(File.dirname(path), real_path) if real_path =~ /^\.\.|^\w/ + next unless File.exist?(real_path) + end log_warn "skip non-executable #{path}" unless path =~ /\.cr$/ next end -- 2.23.0

1 0

[PATCH lkp-tests 2/2] sbin: load PKGBUILD.src content
by Liu Shaofei 30 Nov '20

30 Nov '20

The contents of PKGBUILD.src is the following example: source=( "http://172.168.131.113:8800/initrd/build-tar/ansible/ansible-2.9.4.tar.gz" ) Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 6f8b95af..498b4872 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -895,6 +895,7 @@ update_pkgver() { sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" source_safe "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_safe "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_safe "PKGBUILD.src" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -3699,6 +3700,7 @@ else fi source_buildfile "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_buildfile "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_buildfile PKGBUILD.src fi # set defaults if they weren't specified in buildfile -- 2.23.0

1 0

[PATCH lab-z9] hosts/vm-2p16g: add vm-2p16g file
by Xiao Shenwei 30 Nov '20

30 Nov '20

some job need big memory like build-pkg Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/vm-2p16g | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 hosts/vm-2p16g diff --git a/hosts/vm-2p16g b/hosts/vm-2p16g new file mode 100644 index 0000000..10847a7 --- /dev/null +++ b/hosts/vm-2p16g @@ -0,0 +1,6 @@ +arch: aarch64 +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 16G -- 2.23.0

1 0

[PATCH lkp-tests] hosts/vm-2p16g: add vm-2p16g file
by Xiao Shenwei 30 Nov '20

30 Nov '20

some job need big memory like build-pkg Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/vm-2p16g | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 hosts/vm-2p16g diff --git a/hosts/vm-2p16g b/hosts/vm-2p16g new file mode 100644 index 00000000..42845072 --- /dev/null +++ b/hosts/vm-2p16g @@ -0,0 +1,5 @@ +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 16G -- 2.23.0

1 0

[PATCH v4 compass-ci 4/4] user-client/maintain: send report mail
by Liu Yinsi 30 Nov '20

30 Nov '20

call $CCI_SRC/lib/mail_client.rb to mail. 'LOG' is a file => walk-test.report, job result will write in it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 17edb60..c4ffbd4 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,17 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def mail_report + message = File.read(LOG).chomp + data = " + subject: os rootfs test report + to: #{EMAIL_ADDRESS} + body: #{message}" + + email = MailClient.new + email.send_mail(data) +end + def write_report(report_all) File.open(LOG, 'a') do |f| log = Logger.new(f, 'weekly') -- 2.23.0

1 0

[PATCH v4 compass-ci 3/4] user-client/maintain: write job result into report
by Liu Yinsi 30 Nov '20

30 Nov '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 202908e..17edb60 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,16 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def write_report(report_all) + File.open(LOG, 'a') do |f| + log = Logger.new(f, 'weekly') + log.formatter = proc { |_severity, datetime, _progname, msg| + "[#{datetime}]: #{msg}\n" + } + log.info(report_all) + end +end + def monitor(query, actions, timeout) monitor = Monitor.new monitor.overrides = query -- 2.23.0

1 0

[PATCH v4 compass-ci 2/4] user-client/maintain: get job result by monitor and es query
by Liu Yinsi 30 Nov '20

30 Nov '20

1. monitor job, start time => job_state is 'boot', end time => job_state is 'extract_finished'; 2. query es by field 'job_state' and 'result_root'. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index ba142d0..202908e 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,44 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def monitor(query, actions, timeout) + monitor = Monitor.new + monitor.overrides = query + monitor.action = actions + monitor.run(timeout) + monitor +end + +def monitor_state(job_id) + time_list = [] + + %w[boot extract_finished].each do |state| + result_array = monitor({ 'job_id' => job_id, 'job_state' => state }, { 'stop' => true }, '1800').result + time_list.push(Time.parse(result_dict[0]['time'])) unless result_array.empty? + end + time_list +end + +def es_search(job_id) + es_list = [] + + es = ESQuery.new + result_dict = es.query_by_id(job_id) + %w[job_state result_root].each do |field| + es_list.push(result_dict[field]) unless result_dict.nil? + end + es_list +end + +def find_by_id(job_id) + start_time, end_time = monitor_state(job_id) + cost_time = end_time - start_time unless end_time.nil? + + job_state, result_root = es_search(job_id) + + return start_time, end_time, cost_time.to_i.floor, job_state, result_root +end + def run_qemu Process.fork do %x(#{ENV['CCI_SRC']}/providers/my-qemu.sh >/dev/null 2>&1) -- 2.23.0

1 0

[PATCH v4 compass-ci 1/4] refactor: user-client/maintain: walk-os-iperf-test
by Liu Yinsi 30 Nov '20

30 Nov '20

1. submit job; 2. get job result by monitor and es query; 3. write job result into report; 4. send report mail; 5. cron task, test once an hours, mail once a week. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 148 +++++++++++------- 1 file changed, 88 insertions(+), 60 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 21e1428..ba142d0 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -1,62 +1,90 @@ -#!/bin/bash -e +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -# os test rounds -test_times=1 - -test_logs=walk-test.report -test_yaml=iperf-walk-os.yaml -test_os=( - 'openeuler aarch64 20' - 'openeuler aarch64 20.03' - 'centos aarch64 7.6' - 'centos aarch64 7.8' - 'centos aarch64 8.1.1911' - 'debian aarch64 sid' -) - -trap 'echo User Cancelled! && exit' SIGINT - -sec2date() { - date -d@$1 +"%Y%m%d_%H:%M:%S" -} - -logging() { - if [ 'new' == "$1" ] ; then - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" ${@:2} > $test_logs - else - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" $@ >> $test_logs - fi -} - -create_test_yaml() { - local hwarch os os_arch os_ver cur_os - hwarch=$(arch) - cur_os="$@" - os=$(echo $cur_os |cut -d' ' -f1) - os_arch=$(echo $cur_os |cut -d' ' -f2) - os_ver=$(echo $cur_os |cut -d' ' -f3) - sed "s/USER/$USER/" $test_yaml > test.yaml - sed -i "s/OS_ARCH/$os_arch/" test.yaml - sed -i "s/OS_VER/$os_ver/" test.yaml - sed -i "s/OS/$os/" test.yaml - echo "$hwarch $os/$os_arch/$os_ver" -} - -job_test() { - local job_id bef aft - job_id=$(submit test.yaml |awk '{print $NF}') - bef=$(date +%s) - ./my-qemu.sh - aft=$(date +%s) - logging "$1" "$2" "$3" "$(sec2date $bef)" "$(sec2date $aft)" "$((aft - bef))" "$job_id" -} - -logging new "Index" "hwArch" "OS" "Begin" "End" "Cost/sec" "Job_ID" -for round in $(seq $test_times) -do - for idx in "${!test_os[@]}" - do - job_test $round $(create_test_yaml "${test_os[$idx]}") - done -done +# rootfs test via job, including cifs, initramfs files +# boot way kvm. +# input: job yaml, os, os_version, os_arch, os_mount, testbox. +# output: a test report, including log print time, job ID, os, runtime, job state and url to check logs, and send mail. +# Eg: test report format: +# [2020-11-26 19:22:22 +0800]: ["crystal.143338", "centos", "aarch64", "7.6.1810", "cifs", 2020-11-26 19:21:12 +0800, +# 2020-11-26 19:22:21 +0800, 69, "finished", "/result/iperf-walk-os-test/vm-2p8g/2020-11-26/crystal.143338"] + +require "#{ENV['CCI_SRC']}/lib/es_query" +require "#{ENV['LKP_SRC']}/lib/monitor" +require "#{ENV['CCI_SRC']}/lib/mail_client" + +require 'time' +require 'logger' +require 'rufus-scheduler' + +TEST_OS = ['openeuler aarch64 20.03', + 'centos aarch64 7.6.1810', + 'centos aarch64 7.8.2003', + 'centos aarch64 8.1.1911', + 'debian aarch64 sid', + 'archlinux aarch64 2020-11-12'].freeze + +TEST_YAML = 'iperf-walk-os.yaml' + +TESTBOX = 'vm-2p8g' + +LOG = 'walk-test.report' + +OS_MOUNT = %w[cifs initramfs].freeze + +EMAIL_ADDRESS = (ENV['USER']).to_s.freeze + +def run_qemu + Process.fork do + %x(#{ENV['CCI_SRC']}/providers/my-qemu.sh >/dev/null 2>&1) + end +end + +def submit_job(os_str, os_mount) + `submit --my-queue 'os os_arch os_version=#{os_str}' 'testbox=#{TESTBOX}' 'os_mount=#{os_mount}' #{TEST_YAML}` +end + +def traversal_os_mount + OS_MOUNT.each do |os_mount| + traversal_test_os(os_mount) + end +end + +def traversal_test_os(os_mount) + TEST_OS.each do |os_str| + test_rootfs(os_str, os_mount) + end +end + +def test_rootfs(os_str, os_mount) + message = submit_job(os_str, os_mount).chomp + job_id = message.split('=')[1] + run_qemu + + job_result = if message.include? 'error' + [] + else + find_by_id(job_id) + end + + report_all = job_id.split(', ') + os_str.split + os_mount.split + job_result + write_report(report_all) +end + +def cron_job + test = Rufus::Scheduler.new + test.cron '0 8-22 * * *' do + traversal_os_mount + end + + mail = Rufus::Scheduler.new + mail.cron '0 8 * * 1' do + mail_report + end + + [test, mail].each(&:join) +end + +cron_job -- 2.23.0

1 0

[PATCH compass-ci] doc: standalone partitions and increase memory
by Liu Yinsi 30 Nov '20

30 Nov '20

8g memory is too small, and should set standalone partitions before deploy CCI, because run container will takes up /var/lib/docker a lot of space. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- doc/tutorial.md | 671 ++++++++++++++++++++++++------------------------ 1 file changed, 338 insertions(+), 333 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index 03f1342..ae8b67f 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -1,333 +1,338 @@ -# Compass-CI 用户指南 - -## 简介 - - -### 概念 - -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 - - -### 功能描述 - -- 测试服务 - - 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 - -- 调测环境登录 - - Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 - -- 测试结果分析 - - Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 - -- 辅助定位 - - Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 - -### 应用场景 - -- 聚合开发者测试用例 - - 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 - -- 登录环境随时调测 - - 测试过程中，发现有 Bug 时，可随时提供调测资源服务，登录到环境进行复现、调试。 - -- 快照数据分析对比 - - 测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 - -- 辅助定界 - - 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题 Commit 信息。 - - - -## 使用 Compass-CI -> 当您注册之后，便可以编写 yaml 文件并通过我们的工具上传任务以进行自定义测试，测试功能将尽快上线。 - - -- 注册自己的仓库 - - 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 - ```bash - git clone https://gitee.com/wu_fengguang/upstream-repos.git - less upstream-repos/README.md - ``` - -### 编写 job yaml 文件 - -#### job yaml 简介 -job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAML_for_ruby.html)的格式编写，所有 job 文件位于```$LKP_SRC/jobs```路径下。[$LKP_SRC](https://gitee.com/wu_fengguang/lkp-te… - -#### job yaml 的结构 - -- 标识头部 Header（必选） - - 每一个 job 文件的开始部分，都有一些基础的描述信息，称之为 yaml 的 Header。头部主要有 suite 和 category，suite 是测试套的名称。category 是测试类型，包括 benchmarch（性能测试）/functional（功能测试）/noise-benchmark（不常用）。 - - ```yaml - suite: netperf \\测试套名称 - category: benchmark \\测试类型：benchmarch|functional|noise-benchmark - ``` - -- 测试脚本和参数（必选） - - - 测试脚本位于```$LKP_SRC/tests```，```$LKP_SRC/daemon```路径下。job yaml 中写入与上述路径中脚本相匹配的文件，将其视为可执行脚本。 - - ``` - $LKP_SRC/setup - $LKP_SRC/monitors - $LKP_SRC/daemon - $LKP_SRC/tests - ``` - - 参数值作为环境变量传递，每个测试脚本将在其文件头的注释中记录可接受的参数。 - - ```yaml - netserver: - netperf: - runtime: 10s - test: - - TCP_STREAM - - UDP_STREAM - send_size: - - 1024 - - 2048 - ``` -- 测试资源相关变量（必选） - - SUT/schduler/部署等资源相关变量。 - ```yaml - testbox: vm-2p8g - os: openeuler - os_version: 20.03 - os_arch: aarch64 - ``` -- 系统设置脚本（可选） - - 位于```$LKP_SRC/setup```路径下。设置脚本将在测试脚本之前运行，主要是用于启停一些依赖性服务或工具，或者配置测试所需的参数等。 - - ```yaml - cgroup2: #$LKP_SRC/setup/cgroup2 executable script - memory.high: 90% - memory.low: 50% - memory.max: max - memory.swap.max: - io.max: - io.weight: - rdma.max: - ``` -- 监控脚本（可选） - - 位于```$LKP_SRC/monitors```路径下。监视器在运行基准测试时可以捕获性能统计数据，对性能分析和回归根源有价值。 - - ```yaml - proc-vmstats: - nterval: 5 - ``` - -**了解更多 job yaml 相关知识** [Job definition to execution](https://gitee.com/wu_fengguang/lkp-tests/blob/master/jobs/README… - - -### 提交测试任务 - -### 查看测试结果 - -### 比较测试结果 - -### 提交 borrow 任务 - -### 提交 bisect 任务 - -## 高级功能 - -### 添加 OS 支持 - -制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 - -#### 操作步骤 - - 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） - - 通过 docker 获取 rootfs - - 1) 下载 openEuler 官方提供的 docker 镜像压缩包 - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) 加载 docker 镜像 - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) 启动 openEuler 容器 - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) 拷贝 docker 的 rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - 通过 qemu.img(qcow2格式)获取 rootfs (openEuler为例) - - - 1) 下载 openEule r官方网站提供的 qcow2 格式镜像 - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) 使用{compass-ci}/container/qcow2rootfs 制作rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. 定制rootfs - 1. 使用chroot命令切换到 rootfs (此步骤需要 root 权限) - ```bash - chroot openEuler-rootfs - ``` - 2. 根据个人需要安装并配置服务 - - a. 修改 root 密码 - b. 配置 ssh 服务 - c. 检查系统时间 - d. 如果使用 docker 制作 osimage 还需要以下操作： - > 1. 安装所需版本内核 - > 2. 从 centos 官方网站下载内核rpm包 - > 3. 使用 yum 进行安装 - > 4. 删除 docker 环境变量文件 - - 3. 退出 rootfs，并打包 - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` -#### FAQ -1. 日志报错 “Unable to mount root fs on unknown-block” - - 问题现象 - ```bash - [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: - [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) - ... - [ 0.399404] Memory Limit: none - [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- - ``` - - 解决方法 - - 1）启动内存不足，增加内存可解决。 - 2）内核文件权限不足，给予 644 权限。 - -2. 系统运行缓慢 - - - 问题现象 - - 打包镜像体积过大，会消耗很大内存 - - 解决方法 - - 建议用户根据具体需要对 rootfs 进行裁剪。 - - -### 添加测试用例 - 参考文档详见：[添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/ad… - - -### PKGBUILD 构建 - -1. 概述： - - 使用 PKGBUILD 完成上游项目源代码的编译构建，同一项目可以使用不同 PKGBUILD 完成项目编译构建过程。 - PKGBUILD 相关参考文档：https://wiki.archlinux.org/index.php/PKGBUILD。 - -2. 步骤： - - 1) 注册仓库 - 如果您想在 git push 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 - ```bash - git clone https://gitee.com/wu_fengguang/upstream-repos.git - less upstream-repos/README.md - ``` - - 2) 执行构建测试 - 项目代码更新后自动触发构建任务，无需其他操作。 - - 3) 查看结果 - web: https://compass-ci.openeuler.org/jobs。 - - 4) 备注 - 目前提供 archlinux 下已有 PKGBUILD 项目构建测试，暂不支持自主导入 PKGBUILD 文件。 - PKGBUILD 存放路径： /srv/git/archlinux/*/*/PKGBUILD。 - - -### depends - -- 依赖包构建 - 流程: 以下默认使用 debian:sid 版本进行,如果使用其他OS跑测试用例，则需要在 lkp-tests/distro/adaptation/ 下将包名以键值对的形式写入对应 OS 的文件中 - > 例： `echo 'apache2: httpd' >> lkp-tests/distro/adaptation/openeuler` - - 1) 在lkp-tests/distro/depends/下新建一个与测试用例同名的文件 - `vim lkp-tests/distro/depends/test` - 2) 将测试用例所需debian下的包名填入创建的文件中 - `echo "apache2" >> lkp-tests/distro/depends/test` - 3) 生成lkp-aarch64.cgz - `./crystal-ci/container/lkp-initrd/run` - 4) 提交测试用例 - `submit test.yaml` - 5) 查看结果 - https://compass-ci.openeuler.org/jobs - - - -### 本地搭建 compass-ci 服务器节点 - -在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 - -#### 准备工作 -- 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 8GB - CPU：64 nuclear (建议) - 硬盘：>= 500G - -- 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 - - >**说明：** - >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… - - -#### 操作指导 -1. 登录 openEuler 系统 - -2. 创建工作目录并设置文件权限 - ```bash - mkdir demo && cd demo && umask 002 - ``` -3. 克隆 compass-ci 项目代码到 demo 目录 - ```bash - git clone https://gitee.com/wu_fengguang/compass-ci.git - ``` -4. 执行一键部署脚本 install-tiny - ```bash - cd compass-ci/sparrow && ./install-tiny - ``` - - -## 呼吁合作 - - 增强 git bisect 能力 - - 增强数据分析能力 - - 增强数据结果可视化能力 - -## 下一步计划 - -1. 优化 git bisect 精准率及效率。 -2. 优化数据分析联动能力，让数据分析更加聚焦。 -3. 数据可视化优化，更加友好展示数据比较结果。 -4. 增强登录认证机制，如：GPG 认证。 -5. 优化部署效率。 +# Compass-CI 用户指南 + +## 简介 + + +### 概念 + +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 + + +### 功能描述 + +- 测试服务 + + 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 + +- 调测环境登录 + + Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 + +- 测试结果分析 + + Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 + +- 辅助定位 + + Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 + +### 应用场景 + +- 聚合开发者测试用例 + + 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 + +- 登录环境随时调测 + + 测试过程中，发现有 Bug 时，可随时提供调测资源服务，登录到环境进行复现、调试。 + +- 快照数据分析对比 + + 测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 + +- 辅助定界 + + 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题 Commit 信息。 + + + +## 使用 Compass-CI +> 当您注册之后，便可以编写 yaml 文件并通过我们的工具上传任务以进行自定义测试，测试功能将尽快上线。 + + +- 注册自己的仓库 + + 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 + ```bash + git clone https://gitee.com/wu_fengguang/upstream-repos.git + less upstream-repos/README.md + ``` + +### 编写 job yaml 文件 + +#### job yaml 简介 +job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAML_for_ruby.html)的格式编写，所有 job 文件位于```$LKP_SRC/jobs```路径下。[$LKP_SRC](https://gitee.com/wu_fengguang/lkp-te… + +#### job yaml 的结构 + +- 标识头部 Header（必选） + + 每一个 job 文件的开始部分，都有一些基础的描述信息，称之为 yaml 的 Header。头部主要有 suite 和 category，suite 是测试套的名称。category 是测试类型，包括 benchmarch（性能测试）/functional（功能测试）/noise-benchmark（不常用）。 + + ```yaml + suite: netperf \\测试套名称 + category: benchmark \\测试类型：benchmarch|functional|noise-benchmark + ``` + +- 测试脚本和参数（必选） + + - 测试脚本位于```$LKP_SRC/tests```，```$LKP_SRC/daemon```路径下。job yaml 中写入与上述路径中脚本相匹配的文件，将其视为可执行脚本。 + + ``` + $LKP_SRC/setup + $LKP_SRC/monitors + $LKP_SRC/daemon + $LKP_SRC/tests + ``` + - 参数值作为环境变量传递，每个测试脚本将在其文件头的注释中记录可接受的参数。 + + ```yaml + netserver: + netperf: + runtime: 10s + test: + - TCP_STREAM + - UDP_STREAM + send_size: + - 1024 + - 2048 + ``` +- 测试资源相关变量（必选） + + SUT/schduler/部署等资源相关变量。 + ```yaml + testbox: vm-2p8g + os: openeuler + os_version: 20.03 + os_arch: aarch64 + ``` +- 系统设置脚本（可选） + + 位于```$LKP_SRC/setup```路径下。设置脚本将在测试脚本之前运行，主要是用于启停一些依赖性服务或工具，或者配置测试所需的参数等。 + + ```yaml + cgroup2: #$LKP_SRC/setup/cgroup2 executable script + memory.high: 90% + memory.low: 50% + memory.max: max + memory.swap.max: + io.max: + io.weight: + rdma.max: + ``` +- 监控脚本（可选） + + 位于```$LKP_SRC/monitors```路径下。监视器在运行基准测试时可以捕获性能统计数据，对性能分析和回归根源有价值。 + + ```yaml + proc-vmstats: + nterval: 5 + ``` + +**了解更多 job yaml 相关知识** [Job definition to execution](https://gitee.com/wu_fengguang/lkp-tests/blob/master/jobs/README… + + +### 提交测试任务 + +### 查看测试结果 + +### 比较测试结果 + +### 提交 borrow 任务 + +### 提交 bisect 任务 + +## 高级功能 + +### 添加 OS 支持 + +制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 + +#### 操作步骤 + + 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） + - 通过 docker 获取 rootfs + + 1) 下载 openEuler 官方提供的 docker 镜像压缩包 + + ```bash + wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… + ``` + 2) 加载 docker 镜像 + ```bash + docker load -i openEuler-docker.aarch64 + ``` + + 3) 启动 openEuler 容器 + ```bash + docker run -id openeuler-20.03-lts + ``` + 4) 拷贝 docker 的 rootfs + ```bash + docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs + ``` + - 通过 qemu.img(qcow2格式)获取 rootfs (openEuler为例) + + + 1) 下载 openEule r官方网站提供的 qcow2 格式镜像 + ```bash + wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… + ``` + 2) 使用{compass-ci}/container/qcow2rootfs 制作rootfs + ```bash + cd {compass-ci}/container/qcow2rootfs + ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs + ``` + 2. 定制rootfs + 1. 使用chroot命令切换到 rootfs (此步骤需要 root 权限) + ```bash + chroot openEuler-rootfs + ``` + 2. 根据个人需要安装并配置服务 + + a. 修改 root 密码 + b. 配置 ssh 服务 + c. 检查系统时间 + d. 如果使用 docker 制作 osimage 还需要以下操作： + > 1. 安装所需版本内核 + > 2. 从 centos 官方网站下载内核rpm包 + > 3. 使用 yum 进行安装 + > 4. 删除 docker 环境变量文件 + + 3. 退出 rootfs，并打包 + ```bash + cd $rootfs + find . | coip -o -Hnewc |gzip -9 > $os_name.cgz + ``` +#### FAQ +1. 日志报错 “Unable to mount root fs on unknown-block” + - 问题现象 + ```bash + [ 0.390437] List of all partitions: + [ 0.390806] No filesystem could mount root, tried: + [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) + ... + [ 0.399404] Memory Limit: none + [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- + ``` + - 解决方法 + + 1）启动内存不足，增加内存可解决。 + 2）内核文件权限不足，给予 644 权限。 + +2. 系统运行缓慢 + + - 问题现象 + + 打包镜像体积过大，会消耗很大内存 + - 解决方法 + + 建议用户根据具体需要对 rootfs 进行裁剪。 + + +### 添加测试用例 + 参考文档详见：[添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/ad… + + +### PKGBUILD 构建 + +1. 概述： + + 使用 PKGBUILD 完成上游项目源代码的编译构建，同一项目可以使用不同 PKGBUILD 完成项目编译构建过程。 + PKGBUILD 相关参考文档：https://wiki.archlinux.org/index.php/PKGBUILD。 + +2. 步骤： + + 1) 注册仓库 + 如果您想在 git push 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 + ```bash + git clone https://gitee.com/wu_fengguang/upstream-repos.git + less upstream-repos/README.md + ``` + + 2) 执行构建测试 + 项目代码更新后自动触发构建任务，无需其他操作。 + + 3) 查看结果 + web: https://compass-ci.openeuler.org/jobs。 + + 4) 备注 + 目前提供 archlinux 下已有 PKGBUILD 项目构建测试，暂不支持自主导入 PKGBUILD 文件。 + PKGBUILD 存放路径： /srv/git/archlinux/*/*/PKGBUILD。 + + +### depends + +- 依赖包构建 + 流程: 以下默认使用 debian:sid 版本进行,如果使用其他OS跑测试用例，则需要在 lkp-tests/distro/adaptation/ 下将包名以键值对的形式写入对应 OS 的文件中 + > 例： `echo 'apache2: httpd' >> lkp-tests/distro/adaptation/openeuler` + + 1) 在lkp-tests/distro/depends/下新建一个与测试用例同名的文件 + `vim lkp-tests/distro/depends/test` + 2) 将测试用例所需debian下的包名填入创建的文件中 + `echo "apache2" >> lkp-tests/distro/depends/test` + 3) 生成lkp-aarch64.cgz + `./crystal-ci/container/lkp-initrd/run` + 4) 提交测试用例 + `submit test.yaml` + 5) 查看结果 + https://compass-ci.openeuler.org/jobs + + + +### 本地搭建 compass-ci 服务器节点 + +在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 + +#### 准备工作 +- 硬件 + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + + >**说明：** + >划分最小独立分区请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/master/s… + >划分较大独立分区请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/master/s… + + +- 软件 + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + + >**说明：** + >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + + +#### 操作指导 +1. 登录 openEuler 系统 + +2. 创建工作目录并设置文件权限 + ```bash + mkdir demo && cd demo && umask 002 + ``` +3. 克隆 compass-ci 项目代码到 demo 目录 + ```bash + git clone https://gitee.com/wu_fengguang/compass-ci.git + ``` +4. 执行一键部署脚本 install-tiny + ```bash + cd compass-ci/sparrow && ./install-tiny + ``` + + +## 呼吁合作 + - 增强 git bisect 能力 + - 增强数据分析能力 + - 增强数据结果可视化能力 + +## 下一步计划 + +1. 优化 git bisect 精准率及效率。 +2. 优化数据分析联动能力，让数据分析更加聚焦。 +3. 数据可视化优化，更加友好展示数据比较结果。 +4. 增强登录认证机制，如：GPG 认证。 +5. 优化部署效率。 -- 2.23.0

1 1

[PATCH v8 compass-ci] fix bugs and optimize
by Luan Shengde 30 Nov '20

30 Nov '20

1. set default option '-h' [why] the tool will show help message if no option specified the tool will get no email address without both '-e' and '-f' [how] add '-h' as default option if no option specified options will be reset to '-h' if both '-e' and '-f' are not specified 2. fix undefined method 'empty' [why] the original value is defined 'nil' for my_email when execute answerback-email with no '-e' or '-f', it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) [how] use '.nil?' instead of '.empty?' 3. option parse error [why] if add no required value for option, the option will use the field that follows as its value. example: when execute: ./answerback-email.rb -e -n name '-n' will be used as the value to '-e' [how] add value check for options that need a value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 56 ++++++++++++++++++-- 1 file changed, 51 insertions(+), 5 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..0ecd213 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -80,7 +80,7 @@ end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|n] [--update]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [-l|--login y|n] [-u|--update]\n" opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' opts.banner += "[-g|--gen-sshkey] [--login y|n] [--update]\n" opts.banner += " -e|-f is required when applying account or updating account\n" @@ -94,19 +94,45 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + unless email_address =~ /[^@]+@[\d\w]+\.[\w\d]+/ + message = "Not a standard format email: #{email_address}.\n\n" + puts message + + return false + end + my_info['my_email'] = email_address end opts.on('-n name', '--name name', 'appoint name') do |name| + unless name =~ /^\w[\w\d ]+/ + message = "Name should only contains letters, digits and spaces\n\n" + puts message + + return false + end + stdin_info['my_name'] = name end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| + unless File.exist? pub_key_file + message = "File not found: #{pub_key_file}.\n\n" + puts message + + return false + end stdin_info['new_ssh_pubkey'] = File.read(pub_key_file).strip end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + unless File.exist? email_file + message = "Email file not found: #{email_file}.\n\n" + puts message + + return false + end mail_content = Mail.read(email_file) init_info(mail_content, email_info, my_info) end @@ -121,12 +147,16 @@ options = OptionParser.new do |opts| end opts.on('-l value', '--login value', 'enable/disable login, value: y|n') do |value| - if value.downcase == 'y' + case value + when 'y', 'Y' conf_info['enable_login'] = true - elsif value.downcase == 'n' + when 'n', 'N' conf_info['enable_login'] = false else - raise 'invalid parameter, please use y|n' + message = "-l: bad value #{value}, please use y|n\n\n" + puts message + + return false end end @@ -136,6 +166,15 @@ options = OptionParser.new do |opts| end end +# if no option specified, set default option '-h' +# if both '-e' and '-f' are not specified, will clear the ARGV, +# and use '-h' instead +if ARGV.empty? + ARGV << '-h' +elsif (['-e', '-f'] - ARGV).eql? ['-e', '-f'] + ARGV.clear + ARGV << '-h' +end options.parse!(ARGV) def apply_account(my_info, conf_info) @@ -152,7 +191,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) @@ -180,7 +219,14 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] end +def check_server + message = 'please run the tool on z9 server' + + raise message unless ENV['HOSTNAME'] == 'z9' +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) + check_server check_my_email(my_info) my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/4] refactor: user-client/maintain: walk-os-iperf-test
by Liu Yinsi 30 Nov '20

30 Nov '20

1. submit job; 2. get job result by monitor and es query; 3. write job result into report; 4. send report mail; 5. cron task, test once an hours, mail once a week. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 151 +++++++++++------- 1 file changed, 91 insertions(+), 60 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 21e1428..091b01b 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -1,62 +1,93 @@ -#!/bin/bash -e +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -# os test rounds -test_times=1 - -test_logs=walk-test.report -test_yaml=iperf-walk-os.yaml -test_os=( - 'openeuler aarch64 20' - 'openeuler aarch64 20.03' - 'centos aarch64 7.6' - 'centos aarch64 7.8' - 'centos aarch64 8.1.1911' - 'debian aarch64 sid' -) - -trap 'echo User Cancelled! && exit' SIGINT - -sec2date() { - date -d@$1 +"%Y%m%d_%H:%M:%S" -} - -logging() { - if [ 'new' == "$1" ] ; then - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" ${@:2} > $test_logs - else - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" $@ >> $test_logs - fi -} - -create_test_yaml() { - local hwarch os os_arch os_ver cur_os - hwarch=$(arch) - cur_os="$@" - os=$(echo $cur_os |cut -d' ' -f1) - os_arch=$(echo $cur_os |cut -d' ' -f2) - os_ver=$(echo $cur_os |cut -d' ' -f3) - sed "s/USER/$USER/" $test_yaml > test.yaml - sed -i "s/OS_ARCH/$os_arch/" test.yaml - sed -i "s/OS_VER/$os_ver/" test.yaml - sed -i "s/OS/$os/" test.yaml - echo "$hwarch $os/$os_arch/$os_ver" -} - -job_test() { - local job_id bef aft - job_id=$(submit test.yaml |awk '{print $NF}') - bef=$(date +%s) - ./my-qemu.sh - aft=$(date +%s) - logging "$1" "$2" "$3" "$(sec2date $bef)" "$(sec2date $aft)" "$((aft - bef))" "$job_id" -} - -logging new "Index" "hwArch" "OS" "Begin" "End" "Cost/sec" "Job_ID" -for round in $(seq $test_times) -do - for idx in "${!test_os[@]}" - do - job_test $round $(create_test_yaml "${test_os[$idx]}") - done -done +# rootfs test via job, including cifs, initramfs files +# boot way kvm. +# input: job yaml, os, os_version, os_arch, os_mount, testbox. +# output: a test report, including log print time, job ID, os, runtime, job state and url to check logs, and send mail. +# Eg: test report format: +# [2020-11-26 19:22:22 +0800]: ["crystal.143338", "centos", "aarch64", "7.6.1810", "cifs", 2020-11-26 19:21:12 +0800, +# 2020-11-26 19:22:21 +0800, 69, "finished", "/result/iperf-walk-os-test/vm-2p8g/2020-11-26/crystal.143338"] + +require "#{ENV['CCI_SRC']}/lib/es_query" +require "#{ENV['LKP_SRC']}/lib/monitor" +require "#{ENV['CCI_SRC']}/lib/mail_client" + +require 'time' +require 'logger' +require 'rufus-scheduler' + +TEST_OS = ['openeuler aarch64 20.03', + 'centos aarch64 7.6.1810', + 'centos aarch64 7.8.2003', + 'centos aarch64 8.1.1911', + 'debian aarch64 sid', + 'archlinux aarch64 2020-11-12'].freeze + +TEST_YAML = 'iperf-walk-os.yaml' + +TESTBOX = 'vm-2p8g' + +LOG = 'walk-test.report' + +OS_MOUNT = %w[cifs initramfs].freeze + +EMAIL_ADDRESS = (ENV['USER']).to_s.freeze + +def run_qemu + Process.fork do + %x(#{ENV['CCI_SRC']}/providers/my-qemu.sh >/dev/null 2>&1) + end + +end + +def submit_job(os_str, os_mount) + `submit --my-queue 'os os_arch os_version=#{os_str}' 'testbox=#{TESTBOX}' 'os_mount=#{os_mount}' #{TEST_YAML}` +end + +def traversal_os_mount + OS_MOUNT.each do |os_mount| + traversal_test_os(os_mount) + end + +end + +def traversal_test_os(os_mount) + TEST_OS.each do |os_str| + test_rootfs(os_str, os_mount) + end + +end + +def test_rootfs(os_str, os_mount) + message = submit_job(os_str, os_mount).chomp + job_id = message.split('=')[1] + run_qemu + + job_result = if message.include? 'error' + [] + else + find_by_id(job_id) + end + + report_all = job_id.split(', ') + os_str.split + os_mount.split + job_result + write_report(report_all) +end + +def cron_job + test = Rufus::Scheduler.new + test.cron '0 8-22 * * *' do + traversal_os_mount + end + + mail = Rufus::Scheduler.new + mail.cron '0 8 * * 1' do + mail_report + end + + [test, mail].each(&:join) +end + +cron_job -- 2.23.0

2 2

[PATCH compass-ci 3/3] send-internet-mail: update start
by Luan Shengde 30 Nov '20

30 Nov '20

add new environment variable MAILDIR HOST_SERVER used to check whether need to run store-email attach MAILDIR to the container used to store the mails have been sent MAILDIR/sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 483f01a..3803127 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -12,11 +12,14 @@ names = Set.new %w[ ROBOT_EMAIL_ADDRESS ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT + MAILDIR ] defaults = relevant_defaults(names) defaults['CCI_SRC'] = '/c/compass-ci' +defaults['MAILDIR'] ||= '/srv/cci/Maildir/.compass-ci' +defaults['HOST_SERVER'] = ENV['HOSTNAME'] env = docker_env(defaults) @@ -27,10 +30,10 @@ cmd = %W[ -d ] + env + %W[ -p #{defaults['SEND_MAIL_PORT']}:#{defaults['SEND_MAIL_PORT']} + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw -v #{ENV['CCI_SRC']}:#{defaults['CCI_SRC']} -v /etc/localtime:/etc/localtime:ro -w #{defaults['CCI_SRC']}/container/send-internet-mail - --log-driver json-file send-internet-mail ] -- 2.23.0

1 0

[PATCH compass-ci 2/3] send-internet-mail/mail-post: call function to store email
by Luan Shengde 30 Nov '20

30 Nov '20

call function store_email to store email SEND_MAIL_PORT: 49000 # port for send internet mail HOST_SERVER: z9 # only on z9 it works Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 0b1f94a..cac7a76 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -51,5 +51,11 @@ def send_mail(mail_info) to mail_info['to'] body mail_info['body'] end + mail.deliver! + + return unless ENV['SEND_MAIL_PORT'].to_s == '49000' + return unless ENV['HOST_SERVER'] == 'z9' + + store_email(mail) end -- 2.23.0

1 0

[PATCH compass-ci 1/3] send-internet-mail: add store email
by Luan Shengde 30 Nov '20

30 Nov '20

add function: store_email store email to MAILDIR/sent file name: ${email_addr}_${time_now} Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 3b1d159..851904c 100644 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -19,3 +19,13 @@ smtp = { } Mail.defaults { delivery_method :smtp, smtp } + +def store_email(mail) + time_now = Time.new.strftime('%Y%m%d%H%M%S') + file_name = [mail.to[0], time_now].join('_') + file_full_name = File.join(ENV['MAILDIR'], 'sent', file_name) + File.open(file_full_name, 'w') do |f| + f.puts mail + end + FileUtils.chown_R(1144, 1110, file_full_name) +end -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-account.sh: rename
by Lu Weitao 30 Nov '20

30 Nov '20

[Why] keep same style with es-jobs-mapping.sh, es-repo-mapping.sh [How] es-account.sh => es-accounts-mapping.sh Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/{es-account.sh => es-accounts-mapping.sh} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename sbin/{es-account.sh => es-accounts-mapping.sh} (100%) diff --git a/sbin/es-account.sh b/sbin/es-accounts-mapping.sh similarity index 100% rename from sbin/es-account.sh rename to sbin/es-accounts-mapping.sh -- 2.23.0

1 0

[PATCH lkp-tests] jobs/borrow-1h.yaml: rm my_email field
by Zhang Yale 30 Nov '20

30 Nov '20

[why] The "my_email" field must be written in "~/.config/compass-ci/default/$user.yaml", If It it not written, the job fails to be submitted. Therefore, this field is not used in borrow-1h.yaml. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-1h.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/jobs/borrow-1h.yaml b/jobs/borrow-1h.yaml index 6c2284f3..be22b94a 100644 --- a/jobs/borrow-1h.yaml +++ b/jobs/borrow-1h.yaml @@ -9,6 +9,5 @@ sshd: nil end %> - my_email: # sleep at the bottom sleep: 1h -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.sh: add function run_target_stats
by Wei Jihui 30 Nov '20

30 Nov '20

if user define his own stats and puts his stats file to tbox by use submit -a or other method, he should exec stats in test scripts by himself. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.sh | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/job.sh b/lib/job.sh index b84e6b18..03c49e0a 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -313,3 +313,10 @@ run_test() fi } +# user can run self-define stats by: +# run_target_stats $(basename $0) +run_target_stats() +{ + local script_name=$1 + $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.json +} -- 2.23.0

1 0

[PATCH v3 lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 30 Nov '20

30 Nov '20

[why] Originally submit_job is too complicated, it need to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/job_on_fail/job-3000.yaml | 52 +++++++++++ .../{job_on_fail.yaml => job.yaml} | 1 + .../job_on_fail/job_on_fail-3000.yaml | 22 ----- spec/submit/link_jobs/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 29 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 29 ------ .../link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 29 ------ spec/submit/link_jobs/redis.yaml | 1 - .../link_jobs1/job-10000-2-100000ss-add.yaml | 57 ++++++++++++ spec/submit/link_jobs1/job.yaml | 8 ++ ...68000000-1-65535-set_get-never-never.yaml} | 90 +++++++------------ spec/submit/link_jobs2/job.yaml | 26 ++++++ .../matrix/job-1-300-whetstone-double.yaml | 55 ++++++++++++ spec/submit/matrix/job-3-300-shell1.yaml | 55 ++++++++++++ .../matrix/{unixbench-spec.yaml => job.yaml} | 5 +- ...unixbench-spec-1-300-whetstone-double.yaml | 25 ------ .../unixbench-spec-3-300-shell1.yaml | 25 ------ spec/submit/merge_yaml/job-defaults.yaml | 49 ++++++++++ spec/submit/merge_yaml/job.yaml | 1 + .../merge_yaml/merge_included_files.yaml | 1 - .../merge_included_files-defaults.yaml | 19 ---- spec/submit/merge_yaml/template_yaml | 2 +- .../ftq/ftq-1000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 ------ .../job-1000-2-100000ss-add.yaml | 57 ++++++++++++ .../job-10000-2-100000ss-add.yaml | 57 ++++++++++++ .../separate_yaml/job-20-2-6000ss-cache.yaml | 57 ++++++++++++ .../separate_yaml/{ftq.yaml => job.yaml} | 0 spec/submit_spec.rb | 41 ++------- 30 files changed, 520 insertions(+), 355 deletions(-) create mode 100644 spec/submit/job_on_fail/job-3000.yaml rename spec/submit/job_on_fail/{job_on_fail.yaml => job.yaml} (97%) delete mode 100644 spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml delete mode 120000 spec/submit/link_jobs/ftq.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml delete mode 120000 spec/submit/link_jobs/redis.yaml create mode 100644 spec/submit/link_jobs1/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/link_jobs1/job.yaml rename spec/submit/{link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml => link_jobs2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml} (56%) create mode 100644 spec/submit/link_jobs2/job.yaml create mode 100644 spec/submit/matrix/job-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrix/job-3-300-shell1.yaml rename spec/submit/matrix/{unixbench-spec.yaml => job.yaml} (54%) delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml create mode 100644 spec/submit/merge_yaml/job-defaults.yaml create mode 100644 spec/submit/merge_yaml/job.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml create mode 100644 spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml rename spec/submit/separate_yaml/{ftq.yaml => job.yaml} (100%) diff --git a/spec/submit/job_on_fail/job-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml new file mode 100644 index 00000000..5f41e885 --- /dev/null +++ b/spec/submit/job_on_fail/job-3000.yaml @@ -0,0 +1,52 @@ +--- + +#! spec/submit/job_on_fail/job.yaml +suite: atomic +on_fail: + sleep: 3000 +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/job_on_fail/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + sleep: 3000 + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/job_on_fail/job_on_fail.yaml b/spec/submit/job_on_fail/job.yaml similarity index 97% rename from spec/submit/job_on_fail/job_on_fail.yaml rename to spec/submit/job_on_fail/job.yaml index bf68ff40..5aaedf30 100644 --- a/spec/submit/job_on_fail/job_on_fail.yaml +++ b/spec/submit/job_on_fail/job.yaml @@ -2,4 +2,5 @@ suite: atomic on_fail: sleep: 3000 + atomic: diff --git a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml b/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml deleted file mode 100644 index 3d7ad9b5..00000000 --- a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml +++ /dev/null @@ -1,22 +0,0 @@ ---- -suite: atomic -on_fail: - sleep: 3000 -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - sleep: 3000 - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq.yaml b/spec/submit/link_jobs/ftq.yaml deleted file mode 120000 index b472f871..00000000 --- a/spec/submit/link_jobs/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index 9a613c00..00000000 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 8bf52d0f..00000000 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index 8cb78287..00000000 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/redis.yaml b/spec/submit/link_jobs/redis.yaml deleted file mode 120000 index 438bdab7..00000000 --- a/spec/submit/link_jobs/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit/link_jobs1/job-10000-2-100000ss-add.yaml b/spec/submit/link_jobs1/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..a36d77de --- /dev/null +++ b/spec/submit/link_jobs1/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_jobs1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_jobs1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_jobs1/job.yaml b/spec/submit/link_jobs1/job.yaml new file mode 100644 index 00000000..5624b122 --- /dev/null +++ b/spec/submit/link_jobs1/job.yaml @@ -0,0 +1,8 @@ +suite: ftq + +nr_task: 100% +samples: 100000ss + +ftq: + test: add + freq: 10000 diff --git a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 56% rename from spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_jobs2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 466a2054..6e023d71 100644 --- a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_jobs2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -1,7 +1,7 @@ --- + +#! spec/submit/link_jobs2/job.yaml suite: redis -testcase: redis -category: benchmark need_memory: 62G nr_threads: 2 numactl: @@ -16,7 +16,7 @@ cluster: cs-localhost cpu_node_bind: even nr_processes: 4 if role server: - redis-server: + redis-server: if role client: redis: test: set,get @@ -27,43 +27,43 @@ if role client: key_len: 68000000 lab: spec_lab testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_jobs2/job.yaml tbox_group: vm-2p8g--spec_submit node_roles: server client + +#! hosts/vm-2p8g provider: qemu template: kvm nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -kmsg: -boot-time: -uptime: -iostat: -heartbeat: -vmstat: -numa-numastat: -numa-vmstat: -numa-meminfo: -proc-vmstat: -proc-stat: -meminfo: -slabinfo: -interrupts: -lock_stat: -latency_stats: -softirqs: -bdi_dev_mapping: -diskstats: -nfsstat: -cpuidle: -cpufreq-stats: -sched_debug: -perf-stat: -mpstat: -perf-profile: -cpufreq_governor: + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() pp: numactl: all: 1 @@ -85,31 +85,7 @@ pp: key_len: 68000000 nr_processes: 4 cpu_node_bind: even - cpufreq_governor: + cpufreq_governor: + +#! auto generated by add_monitors() monitors: - kmsg: - boot-time: - uptime: - iostat: - heartbeat: - vmstat: - numa-numastat: - numa-vmstat: - numa-meminfo: - proc-vmstat: - proc-stat: - meminfo: - slabinfo: - interrupts: - lock_stat: - latency_stats: - softirqs: - bdi_dev_mapping: - diskstats: - nfsstat: - cpuidle: - cpufreq-stats: - sched_debug: - perf-stat: - mpstat: - perf-profile: diff --git a/spec/submit/link_jobs2/job.yaml b/spec/submit/link_jobs2/job.yaml new file mode 100644 index 00000000..bf96e654 --- /dev/null +++ b/spec/submit/link_jobs2/job.yaml @@ -0,0 +1,26 @@ +suite: redis + +need_memory: 62G +nr_threads: 100% +numactl: + all: 1 +sysctl: + sc_overcommit_memory: 1 + sc_somaxconn: 65535 +transparent_hugepage: + thp_enabled: never + thp_defrag: never +cluster: cs-localhost +cpu_node_bind: even +nr_processes: 4 + +if role server: + redis-server: +if role client: + redis: + test: set,get + data_size: 1024 + n_client: 5 + requests: 68000000 + n_pipeline: 3 + key_len: 68000000 diff --git a/spec/submit/matrix/job-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml new file mode 100644 index 00000000..ae23f10c --- /dev/null +++ b/spec/submit/matrix/job-1-300-whetstone-double.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/job-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml new file mode 100644 index 00000000..90e20c6b --- /dev/null +++ b/spec/submit/matrix/job-3-300-shell1.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/job.yaml similarity index 54% rename from spec/submit/matrix/unixbench-spec.yaml rename to spec/submit/matrix/job.yaml index e17322b4..75864574 100644 --- a/spec/submit/matrix/unixbench-spec.yaml +++ b/spec/submit/matrix/job.yaml @@ -1,9 +1,8 @@ suite: unixbench runtime: 300s - nr_task test: - - 1 whetstone-double - - 3 shell1 +- 1 whetstone-double +- 3 shell1 unixbench: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml deleted file mode 100644 index 9896bcef..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '1' -test: whetstone-double -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: whetstone-double - nr_task: '1' - cpufreq_governor: -monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml deleted file mode 100644 index 912d10ee..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '3' -test: shell1 -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: shell1 - nr_task: '3' - cpufreq_governor: -monitors: diff --git a/spec/submit/merge_yaml/job-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml new file mode 100644 index 00000000..4168a6ef --- /dev/null +++ b/spec/submit/merge_yaml/job-defaults.yaml @@ -0,0 +1,49 @@ +--- + +#! spec/submit/merge_yaml/job.yaml +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/merge_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/merge_yaml/job.yaml b/spec/submit/merge_yaml/job.yaml new file mode 100644 index 00000000..6d64c381 --- /dev/null +++ b/spec/submit/merge_yaml/job.yaml @@ -0,0 +1 @@ +<< : template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml deleted file mode 100644 index 8d459a32..00000000 --- a/spec/submit/merge_yaml/merge_included_files.yaml +++ /dev/null @@ -1 +0,0 @@ -<<: template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml deleted file mode 100644 index 42b53c13..00000000 --- a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml +++ /dev/null @@ -1,19 +0,0 @@ ---- -suite: atomic -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/merge_yaml/template_yaml b/spec/submit/merge_yaml/template_yaml index 331cb470..a4805648 100644 --- a/spec/submit/merge_yaml/template_yaml +++ b/spec/submit/merge_yaml/template_yaml @@ -1,3 +1,3 @@ suite: atomic -atomic: +atomic: diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index f3e2c553..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 63bac5cf..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index a802dc1b..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..4dcd179d --- /dev/null +++ b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..d5566714 --- /dev/null +++ b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..dd177379 --- /dev/null +++ b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/job.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq.yaml rename to spec/submit/separate_yaml/job.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..312400a4 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -14,46 +14,23 @@ def stable_yaml_file(yaml_file) save_yaml(job, yaml_file) end -def traverse_file(output_dir) - Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - stable_yaml_file(yaml_file) - end +def traverse_file(yaml_file) + stable_yaml_file(yaml_file) unless File.basename(yaml_file) == 'job.yaml' end - -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) + +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*").each do |dir| + yaml_file = "#{dir}/job.yaml" submit_cmd = [ "#{LKP_SRC}/sbin/submit", - '-o', output_dir, + '-o', dir, '-s', 'lab: spec_lab', '-s', 'testbox: vm-2p8g--spec_submit', yaml_file ] system(*submit_cmd) - traverse_file(output_dir) + traverse_file(yaml_file) end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: delete kernel_append_root in job.yaml
by Wu Zhende 30 Nov '20

30 Nov '20

kernel_params contains kernel_append_root delete kernel_append_root in job.yaml Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 2 -- src/scheduler/kernel_params.cr | 6 +++--- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 55bac7f..6f81172 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -92,7 +92,6 @@ class Job lkp_initrd_user user_lkp_src kernel_uri - kernel_append_root kernel_params docker_image kernel_version @@ -151,7 +150,6 @@ class Job set_kernel_version() set_initrds_uri() set_kernel_uri() - set_kernel_append_root() set_kernel_params() set_lkp_server() set_sshr_info() diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 2cc8172..0afe692 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -11,7 +11,7 @@ class Job return @hash["kernel_custom_params"] if @hash["kernel_custom_params"]? end - private def set_kernel_append_root + def kernel_append_root os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") fs2root = { @@ -22,7 +22,7 @@ class Job "container" => "", } - self["kernel_append_root"] = fs2root[os_mount] + fs2root[os_mount] end private def kernel_console @@ -30,7 +30,7 @@ class Job end private def set_kernel_params - self["kernel_params"] = " #{kernel_common_params()} #{kernel_custom_params()} #{kernel_append_root} #{kernel_console()}" + self["kernel_params"] = " #{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" end end -- 2.23.0

1 0

[PATCH v3 compass-ci] service/scheduler: initialize a Sched for each API
by Wu Zhende 30 Nov '20

30 Nov '20

So can use Sched's instance variable in Sched's any interface Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 7 +++--- src/scheduler/find_job_boot.cr | 6 ++--- src/scheduler/find_next_job_boot.cr | 6 ++--- src/scheduler/request_cluster_state.cr | 14 ++++++------ src/scheduler/scheduler.cr | 31 +++++++++++++++----------- src/scheduler/submit_job.cr | 4 ++-- src/scheduler/update_job_parameter.cr | 6 ++--- 7 files changed, 40 insertions(+), 34 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 2be7d41..26ddd92 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -25,12 +25,13 @@ class Sched property redis property block_helper - def initialize + def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new @redis = Redis::Client.new @task_queue = TaskQueueAPI.new @block_helper = BlockHelper.new @rgc = RemoteGitClient.new + @env = env end def normalize_mac(mac : String) @@ -67,7 +68,7 @@ class Sched extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end - def update_tbox_wtmp(env : HTTP::Server::Context) + def update_tbox_wtmp testbox = "" hash = Hash(String, String).new @@ -75,7 +76,7 @@ class Sched hash["time"] = time %w(mac ip job_id tbox_name tbox_state).each do |parameter| - if (value = env.params.query[parameter]?) + if (value = @env.params.query[parameter]?) case parameter when "tbox_name" testbox = value diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b5a23c5..9805464 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -2,9 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def find_job_boot(env : HTTP::Server::Context) - value = env.params.url["value"] - boot_type = env.params.url["boot_type"] + def find_job_boot + value = @env.params.url["value"] + boot_type = @env.params.url["boot_type"] case boot_type when "ipxe" diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 807a9ae..09abbd5 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -2,9 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def find_next_job_boot(env) - hostname = env.params.query["hostname"]? - mac = env.params.query["mac"]? + def find_next_job_boot + hostname = @env.params.query["hostname"]? + mac = @env.params.query["mac"]? if !hostname && mac hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) end diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 72e3810..79a13cc 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -6,9 +6,9 @@ class Sched # all request states: # wait_ready | abort | failed | finished | wait_finish | # write_state | roles_ip - def request_cluster_state(env) - request_state = env.params.query["state"] - job_id = env.params.query["job_id"] + def request_cluster_state + request_state = @env.params.query["state"] + job_id = @env.params.query["job_id"] cluster_id = @redis.hash_get("sched/id2cluster", job_id).not_nil! cluster_state = "" @@ -41,10 +41,10 @@ class Sched return cluster_state when "write_state" - node_roles = env.params.query["node_roles"] - node_ip = env.params.query["ip"] - direct_ips = env.params.query["direct_ips"] - direct_macs = env.params.query["direct_macs"] + node_roles = @env.params.query["node_roles"] + node_ip = @env.params.query["ip"] + direct_ips = @env.params.query["direct_ips"] + direct_macs = @env.params.query["direct_macs"] job_info = {"roles" => node_roles, "ip" => node_ip, diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 5f97567..98b8e2e 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -30,12 +30,17 @@ require "../lib/sched" module Scheduler VERSION = "0.2.0" - sched = Sched.new + add_context_storage_type(Sched) before_all do |env| + env.set "sched", Sched.new(env) env.response.headers["Connection"] = "close" end + def self.get_sched_from_env(env) + env.get("sched").as(Sched) + end + # for debug (maybe kemal debug|logger does better) def self.debug_message(env, response) puts %({"from": "#{env.request.remote_address}", "response": #{response.to_json}}) @@ -53,7 +58,7 @@ module Scheduler # /boot.xxx/host/${hostname} # /boot.yyy/mac/${mac} get "/boot.:boot_type/:parameter/:value" do |env| - response = sched.find_job_boot(env) + response = get_sched_from_env(env).find_job_boot job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? puts %({"job_id": "#{job_id}", "job_state": "boot"}) if job_id @@ -63,7 +68,7 @@ module Scheduler # /~lkp/cgi-bin/gpxelinux.cgi?hostname=:hostname&mac=:mac&last_kernel=:last_kernel get "/~lkp/cgi-bin/gpxelinux.cgi" do |env| - response = sched.find_next_job_boot(env) + response = get_sched_from_env(env).find_next_job_boot job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? puts %({"job_id": "#{job_id}", "job_state": "boot"}) if job_id @@ -75,7 +80,7 @@ module Scheduler # - echo job_id to caller # -- job_id = "0" ? means failed post "/submit_job" do |env| - job_messages = sched.submit_job(env) + job_messages = get_sched_from_env(env).submit_job job_messages.each do |job_message| puts job_message.to_json @@ -103,7 +108,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/set_host_mac?hostname=wfg&mac=00-01-02-03-04-05" put "/set_host_mac" do |env| if (client_hostname = env.params.query["hostname"]?) && (client_mac = env.params.query["mac"]?) - sched.set_host_mac(client_mac, client_hostname) + get_sched_from_env(env).set_host_mac(client_mac, client_hostname) "Done" else @@ -114,7 +119,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" put "/set_host2queues" do |env| if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) - sched.set_host2queues(client_host, client_queues) + get_sched_from_env(env).set_host2queues(client_host, client_queues) "Done" else @@ -125,7 +130,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| if client_mac = env.params.query["mac"]? - sched.del_host_mac(client_mac) + get_sched_from_env(env).del_host_mac(client_mac) "Done" else @@ -136,7 +141,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" put "/del_host2queues" do |env| if client_host = env.params.query["host"]? - sched.del_host2queues(client_host) + get_sched_from_env(env).del_host2queues(client_host) "Done" else @@ -150,7 +155,7 @@ module Scheduler # ?job_file=/lkp/scheduled/job.yaml&job_state=post_run&job_id=10 # ?job_file=/lkp/scheduled/job.yaml&loadavg=0.28 0.82 0.49 1/105 3389&start_time=1587725398&end_time=1587725698&job_id=10 get "/~lkp/cgi-bin/lkp-jobfile-append-var" do |env| - puts sched.update_job_parameter(env) + puts get_sched_from_env(env).update_job_parameter "Done" end @@ -173,7 +178,7 @@ module Scheduler # response: get "server ip" from cluster state, # return "server=<server ip>". get "/~lkp/cgi-bin/lkp-cluster-sync" do |env| - response = sched.request_cluster_state(env) + response = get_sched_from_env(env).request_cluster_state debug_message(env, response) @@ -187,14 +192,14 @@ module Scheduler # get job_id from request job_id = env.params.query["job_id"]? if job_id - puts sched.close_job(job_id) + puts get_sched_from_env(env).close_job(job_id) end "Done" end get "/~lkp/cgi-bin/lkp-wtmp" do |env| - puts sched.update_tbox_wtmp(env) + puts get_sched_from_env(env).update_tbox_wtmp "Done" end @@ -205,7 +210,7 @@ module Scheduler job_id = env.params.query["job_id"].to_s if testbox && ssh_port - sched.report_ssh_port(testbox, ssh_port) + get_sched_from_env(env).report_ssh_port(testbox, ssh_port) end puts %({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"}) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..4e78102 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -2,8 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def submit_job(env : HTTP::Server::Context) - body = env.request.body.not_nil!.gets_to_end + def submit_job + body = @env.request.body.not_nil!.gets_to_end job_content = JSON.parse(body) job = Job.new(job_content, job_content["id"]?) diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index 0692025..cee8b9b 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -2,8 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def update_job_parameter(env : HTTP::Server::Context) - job_id = env.params.query["job_id"]? + def update_job_parameter + job_id = @env.params.query["job_id"]? if !job_id return false end @@ -13,7 +13,7 @@ class Sched job_content["id"] = job_id (%w(start_time end_time loadavg job_state)).each do |parameter| - value = env.params.query[parameter]? + value = @env.params.query[parameter]? if !value || value == "" next end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] git_mirror.rb: change git_repo of upstream-repos
by Li Yuanchao 30 Nov '20

30 Nov '20

Change git_repo of upstream-repos from 'upstream-repos/upstream-repos' to 'u/upstream-repos/upstream-repos'. It's to keep style the same with other repos. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5b6b889..edee331 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -174,7 +174,7 @@ class MirrorMain update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - return reload_fork_info if git_repo == 'upstream-repos/upstream-repos' + return reload_fork_info if git_repo == 'u/upstream-repos/upstream-repos' new_refs = check_new_refs(git_repo) return if new_refs[:heads].empty? @@ -254,7 +254,7 @@ class MirrorMain end def reload_fork_info - upstream_repos = 'upstream-repos/upstream-repos' + upstream_repos = 'u/upstream-repos/upstream-repos' if @git_info[upstream_repos][:cur_refs].empty? @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) else -- 2.23.0

1 0

[PATCH v2 compass-ci] container/netdata-slave: monitoring HW server
by Xiao Shenwei 30 Nov '20

30 Nov '20

[monitored object] file-server : master crystal.ci : slave z9 : slave ibmc-mgnt : slave backup-server: slave jumper-server: slave startup by multi-qemu or multi-docker job [function] slave node will collect host machine metrics, send to master slave node does not provider data storage, alarm, web services. [usage] config/stream.conf destination = master-HOST[:PORT] api key = refer-to: https://github.com/netdata/netdata/tree/master/streaming --- container/netdata-slave/Dockerfile | 10 ++++++ container/netdata-slave/build | 5 +++ container/netdata-slave/config/netdata.conf | 5 +++ container/netdata-slave/config/stream.conf | 10 ++++++ container/netdata-slave/start | 34 +++++++++++++++++++++ 5 files changed, 64 insertions(+) create mode 100755 container/netdata-slave/Dockerfile create mode 100755 container/netdata-slave/build create mode 100644 container/netdata-slave/config/netdata.conf create mode 100644 container/netdata-slave/config/stream.conf create mode 100755 container/netdata-slave/start diff --git a/container/netdata-slave/Dockerfile b/container/netdata-slave/Dockerfile new file mode 100755 index 0000000..5c2c02f --- /dev/null +++ b/container/netdata-slave/Dockerfile @@ -0,0 +1,10 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +FROM netdata/netdata + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN usermod -u 1090 netdata && \ + groupmod -g 1090 netdata diff --git a/container/netdata-slave/build b/container/netdata-slave/build new file mode 100755 index 0000000..1a84550 --- /dev/null +++ b/container/netdata-slave/build @@ -0,0 +1,5 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +docker build -t netdata/netdata-slave . diff --git a/container/netdata-slave/config/netdata.conf b/container/netdata-slave/config/netdata.conf new file mode 100644 index 0000000..456fe02 --- /dev/null +++ b/container/netdata-slave/config/netdata.conf @@ -0,0 +1,5 @@ +[global] + memory mode = none + +[health] + enabled = no diff --git a/container/netdata-slave/config/stream.conf b/container/netdata-slave/config/stream.conf new file mode 100644 index 0000000..ac91a7d --- /dev/null +++ b/container/netdata-slave/config/stream.conf @@ -0,0 +1,10 @@ +[stream] + enabled = yes + destination = + api key = + timeout seconds = 60 + default port = 19999 + send charts matching = * + buffer size bytes = 1048576 + reconnect delay seconds = 5 + initial clock resync iterations = 60 diff --git a/container/netdata-slave/start b/container/netdata-slave/start new file mode 100755 index 0000000..e790298 --- /dev/null +++ b/container/netdata-slave/start @@ -0,0 +1,34 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +. $CCI_SRC/container/defconfig.sh + +docker_rm netdata-slave + +DIR=$(dirname $(realpath $0)) + +cmd=( + docker run -d + --restart=unless-stopped + --hostname=$HOSTNAME + --name netdata-slave + -p 19999:19999 + -v $DIR/config/netdata.conf:/etc/netdata/netdata.conf + -v $DIR/config/stream.conf:/etc/netdata/stream.conf + -v /etc/passwd:/host/etc/passwd:ro + -v /etc/group:/host/etc/group:ro + -v /proc:/host/proc:ro + -v /sys:/host/sys:ro + -v /etc/localtime:/etc/localtime:ro + -v /etc/os-release:/host//etc/os-release:ro + -v /var/run/docker.sock:/var/run/docker.sock:ro + -v /srv/cache/netdata_cache:/var/cache/netdata + -v /srv/cache/netdata_lib:/var/lib/netdata + --log-driver json-file + --log-opt max-size=1g + --cap-add SYS_PTRACE + --security-opt apparmor=unconfined + netdata/netdata-slave +) +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: change git-repo of upstream-repos
by Li Yuanchao 30 Nov '20

30 Nov '20

Change git_repo of upstream-repos from 'upstream-repos/upstream-repos' to 'u/upstream-repos/upstream-repos'. It's to keep style the same with other repos. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5b6b889..edee331 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -174,7 +174,7 @@ class MirrorMain update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - return reload_fork_info if git_repo == 'upstream-repos/upstream-repos' + return reload_fork_info if git_repo == 'u/upstream-repos/upstream-repos' new_refs = check_new_refs(git_repo) return if new_refs[:heads].empty? @@ -254,7 +254,7 @@ class MirrorMain end def reload_fork_info - upstream_repos = 'upstream-repos/upstream-repos' + upstream_repos = 'u/upstream-repos/upstream-repos' if @git_info[upstream_repos][:cur_refs].empty? @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) else -- 2.23.0

1 0

[PATCH compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 30 Nov '20

30 Nov '20

Add a document for introducing the arch, nr_node, nr_cpu, memory.md fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..2a4336c --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,16 @@ +# arch, nr_node, nr_cpu, memory + +Meaning: +- Each test machine has some basic information, such as arch, memory, nr_cpu, nr_node + these fields mean: + arch: the architecture of the machine. + nr_node: number of NUMA nodes. + nr_cpu: number of logical CPUs. + memory: the storage space of the machine. +- These fields are from the file which in $LAB_SRC/hosts. +- Here is an example of these fields in the host file: + arch: aarch64 + nr_node: 4 + nr_cpu: 96 + memory: 256G + ... -- 2.23.0

1 0

[PATCH lkp-tests] job.rb: disable submit warn
by Wei Jihui 30 Nov '20

30 Nov '20

[why] it has a warn when submit job: WARN -- skip non-executable */lkp-tests/monitors/plain/wait [how] lkp-tests/monitors/plain/wait is link to bin/event/wakeup but bin/event/wakeup does not exist. so we will do not check this link file. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 8d1f7469..c485da58 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -115,6 +115,8 @@ def __create_programs_hash(glob, lkp_src) next if path =~ /\.[0-9]+$/ unless File.executable?(path) + next if File.symlink?(path) && File.readlink(path) == '../../bin/event/wakeup' + log_warn "skip non-executable #{path}" unless path =~ /\.cr$/ next end -- 2.23.0

1 0

[PATCH compass-ci] modify READNE.md
by Li Ping 30 Nov '20

30 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.md | 162 ++++++++---------------------------------------------- 1 file changed, 24 insertions(+), 138 deletions(-) diff --git a/README.md b/README.md index 644e9ba..80e0766 100644 --- a/README.md +++ b/README.md @@ -1,74 +1,32 @@ -# Compass-CI 用户指南 +## Compass-CI -## 简介 +### 关于Compass-CI -### 概念 +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务--。Compass-CI 基于开源软件PR进行自动化测试(包括构建测试，软件包自带用例测试等)，共同构建一个开放、完整的开源软件生态测试系统。 -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 - -### 功能描述 +### 功能介绍 - **测试服务** - 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 + 使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - **调测环境登录** - Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 + 使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - **测试结果分析** - Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 - -- **辅助定位** - - Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 - - -### 架构简介 -![Compass-CI架构](doc/pictures/compass-ci-architecture.png) -- **接入层** - - 成为衔接 Compass-CI 服务于开发者的纽带，同时提供内部设备管理的平台，提供 web 门户查阅测试结果、项目清单、ssh 环境登录。 - -- **服务层** - - 提供测试服务、环境登录、结果分析、辅助定界、用户项目注册能力，为本项目的主要功能。 - -- **支撑层** - - 作为项目主体功能实现，详细描述资源编排、部署、构建、测试等详细实现。 - -- **数据层** - - 为项目提供数据处理服务，包括测试结果格式化及归档。 - -- **资源层** - - 提供硬件设备，作为Compass-CI 服务部署的设备以及测试的物理设备、虚拟机、容器资源；提供测试过程中依赖软件仓库及依赖仓库、部署服务器。 - + 通过 [web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 -### 应用场景 - -- **聚合开发者测试用例** - - 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 +- **测试结果复现** -- **登录环境随时调测** - - 测试过程中，发现有 Bug 时，可随时提供调测资源服务，登录到环境进行复现、调试。 - -- **快照数据分析对比** - - 测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 - -- **辅助定界** + Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 - 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题的 Commit 信息。 - +- **辅助定位** + Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 ## 使用 Compass-CI > 您只需要先注册自己的仓库，当您的仓库有 commit 提交时，构建测试会自动执行，并且可在我们的网站中查看结果。 @@ -89,97 +47,25 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 web: https://compass-ci.openeuler.org/jobs +## Source +我们最新的，最完整的源码都存放在 [Gitee](https://gitee.com/wu_fengguang/compass-ci.git) 上，Fork 我们吧！ -## 高级功能 - -### 添加 OS 支持 - -制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 - -#### 操作步骤 - - 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） - - 通过 docker 获取 rootfs - - 1) 下载 openEuler 官方提供的 docker 镜像压缩包 - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) 加载 docker 镜像 - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) 启动 openEuler 容器 - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) 拷贝 docker 的 rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - 通过 qemu.img(qcow2格式)获取 rootfs (以 openEuler 为例) - - - 1) 下载 openEuler 官方网站提供的 qcow2 格式镜像 - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) 使用 {compass-ci}/container/qcow2rootfs 制作 rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. 定制 rootfs - 1. 使用 chroot 命令切换到 rootfs 中(此步骤需要 root 权限) - ```bash - chroot openEuler-rootfs - ``` - 2. 根据个人需要安装并配置服务 - - a. 修改 root 密码 - b. 配置 ssh 服务 - c. 检查系统时间 - d. 如果使用 docker 制作 osimage 还需要以下操作： - > 1. 安装所需版本内核 - > 2. 从 centos 官方网站下载内核 rpm 包 - > 3. 使用 yum 进行安装 - > 4. 删除 docker 环境变量文件 - - 3. 退出 rootfs，并打包 - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` -#### FAQ -1. 日志报错 “Unable to mount root fs on unknown-block” - - 问题现象 - ```bash - [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: - [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) - ... - [ 0.399404] Memory Limit: none - [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- - ``` - - 解决方法 - - 1）启动内存不足，增加内存可解决。 - 2）内核文件权限不足，给予 644 权限。 - -2. 系统运行缓慢 - - - 问题现象 +## Contributing to Compass-CI + +我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导 + +## Website - 打包镜像体积过大，会消耗很大内存。 - - 解决方法 +所有的测试结果，已加入的开源软件清单，历史测试查询比较都可以在我们的官网 [Website](https://compass-ci.openeuler.org) 上找到。 - 建议用户根据具体需要对 rootfs 进行裁剪。 +## 加入我们 +您可以通过以下的方式加入我们： + - 您可以加入我们的 [mailing list](https://mailweb.openeuler.org/postorius/lists/compass-ci.openeuler.or… + - 您可以加入我们每周四下午四点 16：00 的项目例会（会议链接将会通过邮件发送给您） -## 呼吁合作 +欢迎您跟我们一起： - 增强 git bisect 能力 - 增强数据分析能力 - 增强数据结果可视化能力 -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: fix: useless condition
by Ren Wen 30 Nov '20

30 Nov '20

'queue' will always have the value after initializing the Job. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..c8d0875 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -108,11 +108,6 @@ class Sched # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) queue = job.queue - return [{ - "job_id" => "0", - "message" => "get queue failed", - "job_state" => "submit", - }] unless queue # only single job will has "idle job" and "execute rate limiter" if job["idle_job"].empty? -- 2.23.0

1 0

[PATCH compass-ci] compare_matrixes.rb: put compare matrix with more members first
by Li Yuanchao 30 Nov '20

30 Nov '20

As hash table is disordered, the output of compare matrixes sometime would put one member matrix first, that is not a good comparation. We should put good comparation first. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare_matrixes.rb | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d8112e9..652d8f0 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -383,24 +383,33 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group +# the result with more comparison objects first def compare_group_matrices(group_matrices, suites_list, options) result_str = '' - group_matrices.each do |k, v| - matrices_list = [] - matrices_titles = [] - v.each do |dim, matrix| - matrices_titles << dim - matrices_list << matrix - end - if options[:no_print] - result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - else - print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - end + group_matrices_array = sort_by_matrice_size(group_matrices) + group_matrices_array.each do |matrice_kv| + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) end result_str end +def get_matrix_str(matrice_key, matrice_value, suites_list, options) + m_list = [] + m_titles = [] + matrice_value.each do |dim, matrix| + m_titles << dim + m_list << matrix + end + return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + + print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) +end + +#big size first +def sort_by_matrice_size(group_matrices) + group_matrices.sort { |a, b| b[1].size <=> a[1].size } +end + # input: groups_matrices # { # group_key_1 => { -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: field 'queue' always has the value
by Ren Wen 30 Nov '20

30 Nov '20

Field 'queue' will always have a value when initializing a Job. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..c8d0875 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -108,11 +108,6 @@ class Sched # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) queue = job.queue - return [{ - "job_id" => "0", - "message" => "get queue failed", - "job_state" => "submit", - }] unless queue # only single job will has "idle job" and "execute rate limiter" if job["idle_job"].empty? -- 2.23.0

1 1

[PATCH compass-ci] mail-robot: add backup email
by Luan Shengde 30 Nov '20

30 Nov '20

[why] backup email file for successfully applied account [how] backup email after build email message for successfully applied account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..2d5631d 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -98,12 +98,23 @@ class ApplyAccount es.put_source_by_id(@my_info['my_email'], @my_info) end + def store_email(email_message) + time_now = Time.new.strftime('%Y%m%d%H%M%S') + file_name = [@my_info['my_email'], time_now].join '_' + file_full_name = File.join(MAILDIR, 'sent', file_name) + f = File.new(file_full_name, 'w') + f.puts email_message + f.close + FileUtils.chown_R(1144, 1110, file_full_name) + end + def send_mail(error_message) - email_message = if error_message.empty? - build_apply_account_email(@my_info) - else - build_apply_account_fail_email(@my_info, error_message) - end + if error_message.empty? + email_message = build_apply_account_email(@my_info) + store_email(email_message) + else + email_message = build_apply_account_fail_email(@my_info, error_message) + end %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") end -- 2.23.0

2 2

[PATCH v2 compass-ci] modify README
by Li Ping 30 Nov '20

30 Nov '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.md | 162 ++++++++---------------------------------------------- 1 file changed, 24 insertions(+), 138 deletions(-) diff --git a/README.md b/README.md index 644e9ba..80e0766 100644 --- a/README.md +++ b/README.md @@ -1,74 +1,32 @@ -# Compass-CI 用户指南 +## Compass-CI -## 简介 +### 关于Compass-CI -### 概念 +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务--。Compass-CI 基于开源软件PR进行自动化测试(包括构建测试，软件包自带用例测试等)，共同构建一个开放、完整的开源软件生态测试系统。 -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 - -### 功能描述 +### 功能介绍 - **测试服务** - 支持开发者基于本地设备开发，往 github 提交代码，Compass-CI 自动获取代码开展测试，并向开发者反馈测试结果。 + 使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - **调测环境登录** - Compass-CI 提供 SSH 登录能力，测试过程中如果测出问题，开发者可根据需要登录环境进行调测。 + 使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - **测试结果分析** - Compass-CI 记录历史测试结果，对外提供 web 及命令行接口，支持开发者针对已有的测试结果进行分析，挖掘影响测试结果的因素。 - -- **辅助定位** - - Compass-CI 在测试过程中可以自动识别错误信息，触发基于 git tree 的测试，找出引入问题模块的变化点。 - - -### 架构简介 -![Compass-CI架构](doc/pictures/compass-ci-architecture.png) -- **接入层** - - 成为衔接 Compass-CI 服务于开发者的纽带，同时提供内部设备管理的平台，提供 web 门户查阅测试结果、项目清单、ssh 环境登录。 - -- **服务层** - - 提供测试服务、环境登录、结果分析、辅助定界、用户项目注册能力，为本项目的主要功能。 - -- **支撑层** - - 作为项目主体功能实现，详细描述资源编排、部署、构建、测试等详细实现。 - -- **数据层** - - 为项目提供数据处理服务，包括测试结果格式化及归档。 - -- **资源层** - - 提供硬件设备，作为Compass-CI 服务部署的设备以及测试的物理设备、虚拟机、容器资源；提供测试过程中依赖软件仓库及依赖仓库、部署服务器。 - + 通过 [web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 -### 应用场景 - -- **聚合开发者测试用例** - - 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 +- **测试结果复现** -- **登录环境随时调测** - - 测试过程中，发现有 Bug 时，可随时提供调测资源服务，登录到环境进行复现、调试。 - -- **快照数据分析对比** - - 测试过程中，全面监控系统运行信息（CPU/MEM/IO/网络等），对测试过程中的数据做快照归档，提供多次测试之间快照数据分析对比能力，协助开发者对测试结果开展分析，找出影响测试结果的因素。 - -- **辅助定界** + Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 - 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题的 Commit 信息。 - +- **辅助定位** + Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 ## 使用 Compass-CI > 您只需要先注册自己的仓库，当您的仓库有 commit 提交时，构建测试会自动执行，并且可在我们的网站中查看结果。 @@ -89,97 +47,25 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 web: https://compass-ci.openeuler.org/jobs +## Source +我们最新的，最完整的源码都存放在 [Gitee](https://gitee.com/wu_fengguang/compass-ci.git) 上，Fork 我们吧！ -## 高级功能 - -### 添加 OS 支持 - -制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 - -#### 操作步骤 - - 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） - - 通过 docker 获取 rootfs - - 1) 下载 openEuler 官方提供的 docker 镜像压缩包 - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) 加载 docker 镜像 - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) 启动 openEuler 容器 - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) 拷贝 docker 的 rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - 通过 qemu.img(qcow2格式)获取 rootfs (以 openEuler 为例) - - - 1) 下载 openEuler 官方网站提供的 qcow2 格式镜像 - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) 使用 {compass-ci}/container/qcow2rootfs 制作 rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. 定制 rootfs - 1. 使用 chroot 命令切换到 rootfs 中(此步骤需要 root 权限) - ```bash - chroot openEuler-rootfs - ``` - 2. 根据个人需要安装并配置服务 - - a. 修改 root 密码 - b. 配置 ssh 服务 - c. 检查系统时间 - d. 如果使用 docker 制作 osimage 还需要以下操作： - > 1. 安装所需版本内核 - > 2. 从 centos 官方网站下载内核 rpm 包 - > 3. 使用 yum 进行安装 - > 4. 删除 docker 环境变量文件 - - 3. 退出 rootfs，并打包 - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` -#### FAQ -1. 日志报错 “Unable to mount root fs on unknown-block” - - 问题现象 - ```bash - [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: - [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) - ... - [ 0.399404] Memory Limit: none - [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- - ``` - - 解决方法 - - 1）启动内存不足，增加内存可解决。 - 2）内核文件权限不足，给予 644 权限。 - -2. 系统运行缓慢 - - - 问题现象 +## Contributing to Compass-CI + +我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导 + +## Website - 打包镜像体积过大，会消耗很大内存。 - - 解决方法 +所有的测试结果，已加入的开源软件清单，历史测试查询比较都可以在我们的官网 [Website](https://compass-ci.openeuler.org) 上找到。 - 建议用户根据具体需要对 rootfs 进行裁剪。 +## 加入我们 +您可以通过以下的方式加入我们： + - 您可以加入我们的 [mailing list](https://mailweb.openeuler.org/postorius/lists/compass-ci.openeuler.or… + - 您可以加入我们每周四下午四点 16：00 的项目例会（会议链接将会通过邮件发送给您） -## 呼吁合作 +欢迎您跟我们一起： - 增强 git bisect 能力 - 增强数据分析能力 - 增强数据结果可视化能力 -- 2.23.0

1 0

[PATCH compass-ci] doc: how to apply account
by Wu Fengguang 30 Nov '20

30 Nov '20

- fix title - fix format - improve logic - fix broken link Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/manual/apply-account.md | 40 +++++++++++++++++++++++++++++++++ doc/manual/apply-ssh-account.md | 38 ------------------------------- 2 files changed, 40 insertions(+), 38 deletions(-) create mode 100644 doc/manual/apply-account.md delete mode 100644 doc/manual/apply-ssh-account.md diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md new file mode 100644 index 0000000..804b6ca --- /dev/null +++ b/doc/manual/apply-account.md @@ -0,0 +1,40 @@ +# apply account + +## step 1: send email to apply account + +Email template: + +--- + From: {{ David Rientjes <rientjes(a)google.com> }} + To: compass-ci(a)qq.com + Subject: apply account + + my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} + + {{ attach your ssh pubkey file(s), e.g. ~/.ssh/id_rsa.pub }} + +--- + +- please replace the 3 parts in {{ }} with your information. +- my oss commit: a git commit URL that has your contribution. + +## step 2: receive an email + +Which contains account information for you: + + - my_email + - my_name + - my_uuid + - SCHED_HOST + - SCHED_PORT + +## step 3: one-time setup + +Follow the email to setup your local environment: + + - clone git repos + - write ~/.config/compass-ci/default/account.yaml + +## step 4: try it out + +Now try [submitting a job to compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… diff --git a/doc/manual/apply-ssh-account.md b/doc/manual/apply-ssh-account.md deleted file mode 100644 index 160657e..0000000 --- a/doc/manual/apply-ssh-account.md +++ /dev/null @@ -1,38 +0,0 @@ -# apply account - -## steps -1. send apply account email - - send apply account email to: compass-ci(a)qq.com - attention: - email subject: - apply account - email body: - example: - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… - attachment: - ssh pubkey file - - - email example: - - To: compass-ci(a)qq.com - Subject: apply account - - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… - - attach your ssh pub key to the email as attachment - -2. receive email from compass-ci(a)qq.com - the email contains following information: - - my_email - - my_name - - my_uuid - - SCHED_HOST - - SCHED_PORT - -3. local environment configuration - follow steps in the email to finish the following configuration - - setup default yaml - ~/.config/compass-ci/default/account.yaml - -for more information: how to use submit - https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/compass-ci… -- 2.23.0

2 3

[PATCH v3 compass-ci] scheduler: fix the paths to find cluster config file
by Ren Wen 30 Nov '20

30 Nov '20

To find cluster config file at *lab-z9/cluster/* first, *lkp-tests/cluster/* otherwise. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 3 +++ src/scheduler/submit_job.cr | 14 +++++++++----- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 1b37333..1afdda0 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -24,6 +24,9 @@ OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i3 SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" +CCI_REPOS = "/c" +LAB_REPO = "lab-z9" + INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c8d0875..7e1ac27 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -15,7 +15,7 @@ class Sched else cluster_config = get_cluster_config(cluster_file, job.lkp_initrd_user, - job.os_arch) + job.os_arch).not_nil! return submit_cluster_job(job, cluster_config) end rescue ex @@ -147,12 +147,16 @@ class Sched @es.set_job_content(job) end - # get cluster config using own lkp_src cluster file, - # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) - cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - return YAML.parse(File.read(cluster_file_path)) + + cluster_file_paths = [ + Path.new(CCI_REPOS, LAB_REPO, "cluster", cluster_file), + Path.new(lkp_src, "cluster", cluster_file) + ] + cluster_file_paths.each do |f| + return YAML.parse(File.read(f)) if File.file?(f) + end end def get_commit_date(job) -- 2.23.0

1 0

[PATCH lkp-tests] job.rb: disable submit warn
by Wei Jihui 30 Nov '20

30 Nov '20

[why] it has a warn when submit job: WARN -- skip non-executable */lkp-tests/monitors/plain/wait [how] lkp-tests/monitors/plain/wait is link to bin/event/wakeup but bin/event/wakeup does not exist. so we will do not check link file. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 8d1f7469..78604d64 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -115,6 +115,8 @@ def __create_programs_hash(glob, lkp_src) next if path =~ /\.[0-9]+$/ unless File.executable?(path) + next if File.symlink?(path) + log_warn "skip non-executable #{path}" unless path =~ /\.cr$/ next end -- 2.23.0

2 1

[PATCH v7 compass-ci] answerback-email: fix bugs and optimize
by Luan Shengde 30 Nov '20

30 Nov '20

1. set default option '-h' [why] the tool will show help message if no option specified the tool will get no email address without both '-e' and '-f' [how] add '-h' as default option if no option specified options will be reset to '-h' if both '-e' and '-f' are not specified 2. fix undefined method 'empty' [why] the original value is defined 'nil' for my_email when execute answerback-email with no '-e' or '-f', it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) [how] use '.nil?' instead of '.empty?' 3. option parse error [why] if add no required value for option, the option will use the field that follows as its value. example: ./answerback-email.rb -e -n name Not a standard format email: -n. [how] add value check for options that need a value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 49 ++++++++++++++++++-- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..825d815 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -80,7 +80,7 @@ end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|n] [--update]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [-l|--login y|n] [-u|--update]\n" opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' opts.banner += "[-g|--gen-sshkey] [--login y|n] [--update]\n" opts.banner += " -e|-f is required when applying account or updating account\n" @@ -94,19 +94,45 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + unless email_address =~ /[^@]+@[\d\w]+\.[\w\d]+/ + message = "Not a standard format email: #{email_address}.\n\n" + puts message + + return false + end + my_info['my_email'] = email_address end opts.on('-n name', '--name name', 'appoint name') do |name| + unless name =~ /^\w[\w\d ]+/ + message = "Name should only contains letters, digits and spaces\n\n" + puts message + + return false + end + stdin_info['my_name'] = name end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| + unless File.exist? pub_key_file + message = "File not found: #{pub_key_file}.\n\n" + puts message + + return false + end stdin_info['new_ssh_pubkey'] = File.read(pub_key_file).strip end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + unless File.exist? email_file + message = "Email file not found: #{email_file}.\n\n" + puts message + + return false + end mail_content = Mail.read(email_file) init_info(mail_content, email_info, my_info) end @@ -121,12 +147,16 @@ options = OptionParser.new do |opts| end opts.on('-l value', '--login value', 'enable/disable login, value: y|n') do |value| - if value.downcase == 'y' + case value + when 'y', 'Y' conf_info['enable_login'] = true - elsif value.downcase == 'n' + when 'n', 'N' conf_info['enable_login'] = false else - raise 'invalid parameter, please use y|n' + message = "-l: bad value #{value}, please use y|n\n\n" + puts message + + return false end end @@ -136,6 +166,15 @@ options = OptionParser.new do |opts| end end +# if no option specified, set default option '-h' +# if both '-e' and '-f' are not specified, will clear the ARGV, +# and use '-h' instead +if ARGV.empty? + ARGV << '-h' +elsif (['-e', '-f'] - ARGV).eql? ['-e', '-f'] + ARGV.clear + ARGV << '-h' +end options.parse!(ARGV) def apply_account(my_info, conf_info) @@ -152,7 +191,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 0

[PATCH v2 lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 30 Nov '20

30 Nov '20

[why] Originally submit_job is too complicated, it need to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit_spec.rb | 32 +++++++------------------------- 1 file changed, 7 insertions(+), 25 deletions(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..2ea689ec 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -20,10 +20,12 @@ def traverse_file(output_dir) end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*/*.yaml").each do |yaml_file| + next unless File.exist?(yaml_file) + + path = File.dirname(yaml_file) + output_dir = File.join(path, File.basename(yaml_file, '.yaml')) submit_cmd = [ "#{LKP_SRC}/sbin/submit", '-o', output_dir, @@ -36,24 +38,4 @@ def submit_job(submit_item) end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0

2 1

[PATCH v6 compass-ci] answerback-email: fix bugs and optimizi
by Luan Shengde 30 Nov '20

30 Nov '20

1. set default option -h for tool [why] the tool will show help message if no option specified the tool will get no email address without both '-e' and '-f' [how] add '-h' as default option if no option specified options will be reset to '-h' if both '-e' and '-f' are not specified 2. fix undefined method 'empty' [why] the original value is defined nil for my_email when execute answerback-email with no -e or -f, it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) [how] use '.nil?' instead of '.empty?' 3. option parse error [why] if add no required value for option, it will use the field that follow as the value, example: ./answerback-email.rb -e -n name Not a standard format email: -n. [how] add value check for options that need a value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 49 ++++++++++++++++++-- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..825d815 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -80,7 +80,7 @@ end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|n] [--update]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [-l|--login y|n] [-u|--update]\n" opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' opts.banner += "[-g|--gen-sshkey] [--login y|n] [--update]\n" opts.banner += " -e|-f is required when applying account or updating account\n" @@ -94,19 +94,45 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + unless email_address =~ /[^@]+@[\d\w]+\.[\w\d]+/ + message = "Not a standard format email: #{email_address}.\n\n" + puts message + + return false + end + my_info['my_email'] = email_address end opts.on('-n name', '--name name', 'appoint name') do |name| + unless name =~ /^\w[\w\d ]+/ + message = "Name should only contains letters, digits and spaces\n\n" + puts message + + return false + end + stdin_info['my_name'] = name end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| + unless File.exist? pub_key_file + message = "File not found: #{pub_key_file}.\n\n" + puts message + + return false + end stdin_info['new_ssh_pubkey'] = File.read(pub_key_file).strip end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + unless File.exist? email_file + message = "Email file not found: #{email_file}.\n\n" + puts message + + return false + end mail_content = Mail.read(email_file) init_info(mail_content, email_info, my_info) end @@ -121,12 +147,16 @@ options = OptionParser.new do |opts| end opts.on('-l value', '--login value', 'enable/disable login, value: y|n') do |value| - if value.downcase == 'y' + case value + when 'y', 'Y' conf_info['enable_login'] = true - elsif value.downcase == 'n' + when 'n', 'N' conf_info['enable_login'] = false else - raise 'invalid parameter, please use y|n' + message = "-l: bad value #{value}, please use y|n\n\n" + puts message + + return false end end @@ -136,6 +166,15 @@ options = OptionParser.new do |opts| end end +# if no option specified, set default option '-h' +# if both '-e' and '-f' are not specified, will clear the ARGV, +# and use '-h' instead +if ARGV.empty? + ARGV << '-h' +elsif (['-e', '-f'] - ARGV).eql? ['-e', '-f'] + ARGV.clear + ARGV << '-h' +end options.parse!(ARGV) def apply_account(my_info, conf_info) @@ -152,7 +191,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 0

[PATCH v3 compass-ci 4/4] user-client/maintain: send report mail
by Liu Yinsi 30 Nov '20

30 Nov '20

call $CCI_SRC/lib/mail_client.rb to mail. 'LOG' is a file => walk-test.report, job result will write in it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 42a8b97..2ec95ce 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,17 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def mail_report + message = File.read(LOG).chomp + data = " + subject: os rootfs test report + to: #{EMAIL_ADDRESS} + body: #{message}" + + email = MailClient.new + email.send_mail(data) +end + def write_report(report_all) File.open(LOG, 'a') do |f| log = Logger.new(f, 'weekly') -- 2.23.0

1 0

[PATCH v3 compass-ci 3/4] user-client/maintain: write job result into report
by Liu Yinsi 30 Nov '20

30 Nov '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 4f3bc88..42a8b97 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,16 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def write_report(report_all) + File.open(LOG, 'a') do |f| + log = Logger.new(f, 'weekly') + log.formatter = proc { |_severity, datetime, _progname, msg| + "[#{datetime}]: #{msg}\n" + } + log.info(report_all) + end +end + def monitor(query, actions, timeout) monitor = Monitor.new monitor.overrides = query -- 2.23.0

1 0

[PATCH v3 compass-ci 2/4] user-client/maintain: get job result by monitor and es query
by Liu Yinsi 30 Nov '20

30 Nov '20

1. monitor job, start time => job_state is 'boot', end time => job_state is 'extract_finished'; 2. query es by field 'job_state' and 'result_root'. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 40 +++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 091b01b..4f3bc88 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -36,6 +36,46 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def monitor(query, actions, timeout) + monitor = Monitor.new + monitor.overrides = query + monitor.action = actions + monitor.run(timeout) + monitor +end + +def monitor_state(job_id) + time_list = [] + + %w[boot extract_finished].each do |state| + result_array = monitor({ 'job_id' => job_id, 'job_state' => state }, { 'stop' => true }, '1800').result + time_list.push(Time.parse(result_dict[0]['time'])) unless result_array.empty? + end + + time_list +end + +def es_search(job_id) + es_list = [] + + es = ESQuery.new + result_dict = es.query_by_id(job_id) + %w[job_state result_root].each do |field| + es_list.push(result_dict[field]) unless result_dict.nil? + end + + es_list +end + +def find_by_id(job_id) + start_time, end_time = monitor_state(job_id) + cost_time = end_time - start_time unless end_time.nil? + + job_state, result_root = es_search(job_id) + + return start_time, end_time, cost_time.to_i.floor, job_state, result_root +end + def run_qemu Process.fork do %x(#{ENV['CCI_SRC']}/providers/my-qemu.sh >/dev/null 2>&1) -- 2.23.0

1 0

[PATCH v2 compass-ci] service/scheduler: initialize a Sched for each API
by Wu Zhende 30 Nov '20

30 Nov '20

So can use Sched's instance variable in Sched's any interface Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 7 +++--- src/scheduler/find_job_boot.cr | 6 ++--- src/scheduler/find_next_job_boot.cr | 6 ++--- src/scheduler/request_cluster_state.cr | 14 ++++++------ src/scheduler/scheduler.cr | 31 +++++++++++++++----------- src/scheduler/submit_job.cr | 4 ++-- src/scheduler/update_job_parameter.cr | 6 ++--- 7 files changed, 40 insertions(+), 34 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 2be7d41..26ddd92 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -25,12 +25,13 @@ class Sched property redis property block_helper - def initialize + def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new @redis = Redis::Client.new @task_queue = TaskQueueAPI.new @block_helper = BlockHelper.new @rgc = RemoteGitClient.new + @env = env end def normalize_mac(mac : String) @@ -67,7 +68,7 @@ class Sched extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end - def update_tbox_wtmp(env : HTTP::Server::Context) + def update_tbox_wtmp testbox = "" hash = Hash(String, String).new @@ -75,7 +76,7 @@ class Sched hash["time"] = time %w(mac ip job_id tbox_name tbox_state).each do |parameter| - if (value = env.params.query[parameter]?) + if (value = @env.params.query[parameter]?) case parameter when "tbox_name" testbox = value diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b5a23c5..9805464 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -2,9 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def find_job_boot(env : HTTP::Server::Context) - value = env.params.url["value"] - boot_type = env.params.url["boot_type"] + def find_job_boot + value = @env.params.url["value"] + boot_type = @env.params.url["boot_type"] case boot_type when "ipxe" diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 807a9ae..09abbd5 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -2,9 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def find_next_job_boot(env) - hostname = env.params.query["hostname"]? - mac = env.params.query["mac"]? + def find_next_job_boot + hostname = @env.params.query["hostname"]? + mac = @env.params.query["mac"]? if !hostname && mac hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) end diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 72e3810..79a13cc 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -6,9 +6,9 @@ class Sched # all request states: # wait_ready | abort | failed | finished | wait_finish | # write_state | roles_ip - def request_cluster_state(env) - request_state = env.params.query["state"] - job_id = env.params.query["job_id"] + def request_cluster_state + request_state = @env.params.query["state"] + job_id = @env.params.query["job_id"] cluster_id = @redis.hash_get("sched/id2cluster", job_id).not_nil! cluster_state = "" @@ -41,10 +41,10 @@ class Sched return cluster_state when "write_state" - node_roles = env.params.query["node_roles"] - node_ip = env.params.query["ip"] - direct_ips = env.params.query["direct_ips"] - direct_macs = env.params.query["direct_macs"] + node_roles = @env.params.query["node_roles"] + node_ip = @env.params.query["ip"] + direct_ips = @env.params.query["direct_ips"] + direct_macs = @env.params.query["direct_macs"] job_info = {"roles" => node_roles, "ip" => node_ip, diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 5f97567..98b8e2e 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -30,12 +30,17 @@ require "../lib/sched" module Scheduler VERSION = "0.2.0" - sched = Sched.new + add_context_storage_type(Sched) before_all do |env| + env.set "sched", Sched.new(env) env.response.headers["Connection"] = "close" end + def self.get_sched_from_env(env) + env.get("sched").as(Sched) + end + # for debug (maybe kemal debug|logger does better) def self.debug_message(env, response) puts %({"from": "#{env.request.remote_address}", "response": #{response.to_json}}) @@ -53,7 +58,7 @@ module Scheduler # /boot.xxx/host/${hostname} # /boot.yyy/mac/${mac} get "/boot.:boot_type/:parameter/:value" do |env| - response = sched.find_job_boot(env) + response = get_sched_from_env(env).find_job_boot job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? puts %({"job_id": "#{job_id}", "job_state": "boot"}) if job_id @@ -63,7 +68,7 @@ module Scheduler # /~lkp/cgi-bin/gpxelinux.cgi?hostname=:hostname&mac=:mac&last_kernel=:last_kernel get "/~lkp/cgi-bin/gpxelinux.cgi" do |env| - response = sched.find_next_job_boot(env) + response = get_sched_from_env(env).find_next_job_boot job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? puts %({"job_id": "#{job_id}", "job_state": "boot"}) if job_id @@ -75,7 +80,7 @@ module Scheduler # - echo job_id to caller # -- job_id = "0" ? means failed post "/submit_job" do |env| - job_messages = sched.submit_job(env) + job_messages = get_sched_from_env(env).submit_job job_messages.each do |job_message| puts job_message.to_json @@ -103,7 +108,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/set_host_mac?hostname=wfg&mac=00-01-02-03-04-05" put "/set_host_mac" do |env| if (client_hostname = env.params.query["hostname"]?) && (client_mac = env.params.query["mac"]?) - sched.set_host_mac(client_mac, client_hostname) + get_sched_from_env(env).set_host_mac(client_mac, client_hostname) "Done" else @@ -114,7 +119,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" put "/set_host2queues" do |env| if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) - sched.set_host2queues(client_host, client_queues) + get_sched_from_env(env).set_host2queues(client_host, client_queues) "Done" else @@ -125,7 +130,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| if client_mac = env.params.query["mac"]? - sched.del_host_mac(client_mac) + get_sched_from_env(env).del_host_mac(client_mac) "Done" else @@ -136,7 +141,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" put "/del_host2queues" do |env| if client_host = env.params.query["host"]? - sched.del_host2queues(client_host) + get_sched_from_env(env).del_host2queues(client_host) "Done" else @@ -150,7 +155,7 @@ module Scheduler # ?job_file=/lkp/scheduled/job.yaml&job_state=post_run&job_id=10 # ?job_file=/lkp/scheduled/job.yaml&loadavg=0.28 0.82 0.49 1/105 3389&start_time=1587725398&end_time=1587725698&job_id=10 get "/~lkp/cgi-bin/lkp-jobfile-append-var" do |env| - puts sched.update_job_parameter(env) + puts get_sched_from_env(env).update_job_parameter "Done" end @@ -173,7 +178,7 @@ module Scheduler # response: get "server ip" from cluster state, # return "server=<server ip>". get "/~lkp/cgi-bin/lkp-cluster-sync" do |env| - response = sched.request_cluster_state(env) + response = get_sched_from_env(env).request_cluster_state debug_message(env, response) @@ -187,14 +192,14 @@ module Scheduler # get job_id from request job_id = env.params.query["job_id"]? if job_id - puts sched.close_job(job_id) + puts get_sched_from_env(env).close_job(job_id) end "Done" end get "/~lkp/cgi-bin/lkp-wtmp" do |env| - puts sched.update_tbox_wtmp(env) + puts get_sched_from_env(env).update_tbox_wtmp "Done" end @@ -205,7 +210,7 @@ module Scheduler job_id = env.params.query["job_id"].to_s if testbox && ssh_port - sched.report_ssh_port(testbox, ssh_port) + get_sched_from_env(env).report_ssh_port(testbox, ssh_port) end puts %({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"}) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..4e78102 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -2,8 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def submit_job(env : HTTP::Server::Context) - body = env.request.body.not_nil!.gets_to_end + def submit_job + body = @env.request.body.not_nil!.gets_to_end job_content = JSON.parse(body) job = Job.new(job_content, job_content["id"]?) diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index 0692025..cee8b9b 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -2,8 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def update_job_parameter(env : HTTP::Server::Context) - job_id = env.params.query["job_id"]? + def update_job_parameter + job_id = @env.params.query["job_id"]? if !job_id return false end @@ -13,7 +13,7 @@ class Sched job_content["id"] = job_id (%w(start_time end_time loadavg job_state)).each do |parameter| - value = env.params.query[parameter]? + value = @env.params.query[parameter]? if !value || value == "" next end -- 2.23.0

2 1

[PATCH v5 compass-ci] answerback-email: fix bugs and optimizi
by Luan Shengde 30 Nov '20

30 Nov '20

1. set default option -h for tool [why] the tool will show help message if no option specified the tool will get no email address without both '-e' and '-f' [how] add '-h' as default option if no option specified options will be reset to '-h' if both '-e' and '-f' are not specified 2. fix undefined method 'empty' [why] the original value is defined nil for my_email when execute answerback-email with no -e or -f, it will throw error as: raceback (most recent call last): 2: from ./answerback-email.rb:213:in `<main>' 1: from ./answerback-email.rb:184:in `send_account' ./answerback-email.rb:155:in `check_my_email': undefined method `empty?' for nil:NilClass (NoMethodError) [how] use '.nil?' instead of '.empty?' 3. option parse error [why] if add no required value for option, it will use the field that follow as the value, example: ./answerback-email.rb -e -n name Not a standard format email: -n. [how] add value check for options that need a value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 51 +++++++++++++++++--- 1 file changed, 45 insertions(+), 6 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c6bfdd3..ec1f5b7 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -80,7 +80,7 @@ end options = OptionParser.new do |opts| opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|n] [--update]\n" + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [-l|--login y|n] [-u|--update]\n" opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' opts.banner += "[-g|--gen-sshkey] [--login y|n] [--update]\n" opts.banner += " -e|-f is required when applying account or updating account\n" @@ -94,19 +94,45 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + unless email_address =~ /[^@]+@[\d\w]+\.[\w\d]+/ + message = "Not a standard format email: #{email_address}.\n\n" + puts message + + return false + end + my_info['my_email'] = email_address end opts.on('-n name', '--name name', 'appoint name') do |name| + unless name =~ /^\w[\w\d ]+/ + message = "Name should only contains letters, digits and spaces\n\n" + puts message + + return false + end + stdin_info['my_name'] = name end opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| + unless File.exist? pub_key_file + message = "File not found: #{pub_key_file}.\n\n" + puts message + + return false + end stdin_info['new_ssh_pubkey'] = File.read(pub_key_file).strip end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + unless File.exist? email_file + message = "Email file not found: #{email_file}.\n\n" + puts message + + return false + end mail_content = Mail.read(email_file) init_info(mail_content, email_info, my_info) end @@ -121,12 +147,16 @@ options = OptionParser.new do |opts| end opts.on('-l value', '--login value', 'enable/disable login, value: y|n') do |value| - if value.downcase == 'y' + case value + when 'y', 'Y' conf_info['enable_login'] = true - elsif value.downcase == 'n' + when 'n', 'N' conf_info['enable_login'] = false else - raise 'invalid parameter, please use y|n' + message = "-l: bad value #{value}, please use y|n\n\n" + puts message + + return false end end @@ -136,6 +166,15 @@ options = OptionParser.new do |opts| end end +# if no option specified, set default option '-h' +# if both '-e' and '-f' are not specified, will clear the ARGV, +# and use '-h' instead +if ARGV.empty? + ARGV << '-h' +elsif (['-e', '-f'] - ARGV).eql? ['-e', '-f'] + ARGV.clear + ARGV << '-h' +end options.parse!(ARGV) def apply_account(my_info, conf_info) @@ -152,7 +191,7 @@ def check_my_email(my_info) message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' - raise message if my_info['my_email'].empty? + raise message if my_info['my_email'].nil? end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) @@ -210,4 +249,4 @@ def store_account_info(my_info) es.put_source_by_id(my_info['my_email'], my_info) end -send_account(my_info, conf_info, email_info, my_info_es, stdin_info) +# send_account(my_info, conf_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 1

[PATCH compass-ci 4/5] user-client/maintain: write job result into report
by Liu Yinsi 30 Nov '20

30 Nov '20

close the file every time after open and write it, avoid duplicate printing of logs. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- user-client/maintain/walk-os-test/walk-os-iperf-test | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index d9d6bd6..7ed07c5 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -34,6 +34,16 @@ OS_MOUNT = %w[cifs initramfs].freeze EMAIL_ADDRESS = (ENV['USER']).to_s.freeze +def write_report(report_all) + file = File.open(LOG, 'a') + log = Logger.new(file, 'weekly') + log.formatter = proc { |_severity, datetime, _progname, msg| + "[#{datetime}]: #{msg}\n" + } + log.info(report_all) + file.close +end + def monitor(query, actions, timeout) monitor = Monitor.new monitor.overrides = query -- 2.23.0

3 6