- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] doc/manual: update test-oss-project.zh.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.zh.md | 146 ++++++++++++++++-------------- 1 file changed, 78 insertions(+), 68 deletions(-) diff --git a/doc/manual/test-oss-project.zh.md b/doc/manual/test-oss-project.zh.md index aa2cd15..b455c32 100644 --- a/doc/manual/test-oss-project.zh.md +++ b/doc/manual/test-oss-project.zh.md @@ -1,68 +1,78 @@ -# 使用 compass-ci 平台测试开源项目 - -本文介绍了使用 compass-ci 平台测试开源项目的操作。 - -### 添加待测试仓库 URL 到 upstream-repos 仓库 - -执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 - -1. Fork 要测试的代码仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 - -![](./../pictures/fork_blacklight.png) - -2. 执行以下命令，以首字母和仓库名创建文件路径。 - - ``` - mkdir -p b/backlight - ``` - -3. 执行以下命令，在该目录下新建同名文件 backlight。 - ``` - cd b/backlight - touch backlight - ``` - -4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 - - ``` - vim backlight - ``` - 内容格式为 - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - >![](./../public_sys-resources/icon-notice.gif) **注意：** - > - >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 - -5. 通过 Pull Request 命令将新增的 backlight 文件提交到 blacklight 仓库。 - - -### 提交测试任务到 compass-ci 平台 - -1. 准备测试用例 - - 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 - - * 使用仓库中已经适配好的测试用例 - 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： - iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 - - * 编写测试用例并添加到仓库 - - 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) - -2. 配置 auto_submit.yaml 文件，提交测试任务 - - 你只需要在 compass-ci 仓库下面的 sbin/auto_submit.yaml 文件中添加配置信息，如： - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - 通过 Pull Request 的方式将修改好的 auto_submit.yaml 文件提交到 compass-ci 仓库，就可以使用 compass-ci 测试你的项目了。 - - auto_submit.yaml 文件的参数配置请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 +# 使用 compass-ci 平台测试开源项目 + +本文介绍了使用 compass-ci 平台测试开源项目的操作。 + +### 添加待测试仓库 URL 到 upstream-repos 仓库 + +执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 + +1. Fork upstream-repos 仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 + +![](./../pictures/fork_blacklight.png) + +2. 执行以下命令，以首字母和仓库名创建文件路径。 + + ``` + mkdir -p b/backlight + ``` + +3. 执行以下命令，在该目录下新建同名文件 backlight。 + ``` + cd b/backlight + touch backlight + ``` + +4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 + + ``` + vim backlight + ``` + 内容格式为 + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + >![](./../public_sys-resources/icon-notice.gif) **注意：** + > + >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 + +5. 通过 Pull Request 命令将新增的 backlight 文件提交到 upstream-repos 仓库。 + + +### 提交测试任务到 compass-ci 平台 + +1. 准备测试用例 + + 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 + + * 使用仓库中已经适配好的测试用例 + 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： + iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 + + * 编写测试用例并添加到仓库 + + 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) + +2. 配置 upstream-repos仓库中的DEFAULTS文件，提交测试任务 + + 你只需要在上述backlight文件所在目录增加 DEFAULTS 文件并添加配置信息，如： + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + 通过 Pull Request 的方式将修改好的 DEFAULTS 文件提交到 upstream-repos 仓库，就可以使用 compass-ci 测试你的项目了。 + + 详细配置方式请参考 https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md 。 + + 命令参数意义及作用请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] container/dracut-initrd: add some tools into dracut to enable initrd
by Xu Xijian 24 Mar '21

24 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index b8eb6f5..a57d681 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -7,7 +7,7 @@ export DEBIAN_FRONTEND=noninteractive apt-get update apt-get install -y --fix-missing --no-install-recommends -qq apt-utils \ nfs-common netbase cifs-utils kmod rsync dracut dracut-network xz-utils \ -dracut-config-generic +dracut-config-generic lvm2 xz-utils systemd-sysv apt-get clean rm -rf /var/lib/apt/lists/* @@ -17,7 +17,8 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -sed -i "/install() {/a\ inst /sbin/e2fsck /sbin/mke2fs /usr/bin/basename" /usr/lib/dracut/modules.d/00bash/module-setup.sh +tools_00bash="/sbin/e2fsck /sbin/mke2fs /usr/bin/basename /sbin/lvm /sbin/reboot" +sed -i "/install() {/a\ inst $tools_00bash" /usr/lib/dracut/modules.d/00bash/module-setup.sh pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" [ "$(sed -n '$p' $pre_mount_file)" = "exit 0" ] && sed -i '$d' "$pre_mount_file" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/dracut-initrd: use rootfs_src to replace root in local mount
by Xu Xijian 24 Mar '21

24 Mar '21

$rootfs_src is what we want to get timestamp and os_info, $rootfs_src is fixed formatted, $root can have several other format. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 240551a..835bdd2 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -4,7 +4,7 @@ analyse_kernel_cmdline_params() { rootfs="$(getarg root=)" # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') - timestamp="$(basename $root)" + timestamp="$(basename $rootfs_src)" os="$(echo $rootfs_src | awk -F '/|-' '{print $2}')" os_arch="$(echo $rootfs_src | awk -F '/|-' '{print $3}')" os_version="$(echo $rootfs_src | awk -F '/|-' '{print $4}')" -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/job.rb: fix host file not exist
by Li Ping 24 Mar '21

24 Mar '21

[why]: if the hosts_file 2288hv5-2s44p-38g--a6 already in lkp-tests/hosts/ submit borrow-10d.yaml testbox=2288hv5-2s44p-38g--a6 Traceback (most recent call last): 7: from /home/liping/lkp-tests/sbin/submit:136:in `<main>' 6: from /home/liping/lkp-tests/sbin/submit:136:in `each' 5: from /home/liping/lkp-tests/sbin/submit:150:in `block in <main>' 4: from /home/liping/lkp-tests/lib/job.rb:743:in `each_jobs' 3: from /home/liping/lkp-tests/lib/job.rb:447:in `load_defaults' 2: from /home/liping/lkp-tests/lib/job.rb:432:in `load_self_config' 1: from /home/liping/lkp-tests/lib/job.rb:351:in `load_hosts_config' /home/liping/lkp-tests/lib/job.rb:378:in `get_hosts_file': hosts file not exist: 2288hv5-2s44p-38g, maybe need check testbox field (ArgumentError) [how] add check host file at: 1. lab repo + tbox_group 2. lkp_src repo + testbox --- lib/job.rb | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 7fc7cb5b6..8ac5d7a98 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -366,14 +366,17 @@ class Job end def get_hosts_file - testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" - return testbox_file if File.file?(testbox_file) - + lab_repo = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}" tbox_group = @job['tbox_group'].split(/\.|--/)[0] - tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" - return tbox_group_file if File.file?(tbox_group_file) - raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" + ["#{lab_repo}/hosts/#{@job['testbox']}", + "#{LKP_SRC}/hosts/#{@job['testbox']}", + "#{lab_repo}/hosts/#{tbox_group}", + "#{LKP_SRC}/hosts/#{tbox_group}"].each do |file| + return file if File.file?(file) + end + + raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end def include_files -- 2.23.0

1 1

[PATCH v3 lkp-tests] lib/job.rb: fix host file not exist
by Li Ping 24 Mar '21

24 Mar '21

[why]: if the hosts_file 2288hv5-2s44p-38g--a6 already in lkp-tests/hosts/ submit borrow-10d.yaml testbox=2288hv5-2s44p-38g--a6 Traceback (most recent call last): 7: from /home/liping/lkp-tests/sbin/submit:136:in `<main>' 6: from /home/liping/lkp-tests/sbin/submit:136:in `each' 5: from /home/liping/lkp-tests/sbin/submit:150:in `block in <main>' 4: from /home/liping/lkp-tests/lib/job.rb:743:in `each_jobs' 3: from /home/liping/lkp-tests/lib/job.rb:447:in `load_defaults' 2: from /home/liping/lkp-tests/lib/job.rb:432:in `load_self_config' 1: from /home/liping/lkp-tests/lib/job.rb:351:in `load_hosts_config' /home/liping/lkp-tests/lib/job.rb:378:in `get_hosts_file': hosts file not exist: 2288hv5-2s44p-38g, maybe need check testbox field (ArgumentError) [how] add check host file at: 1. lab repo + tbox_group 2. lkp_src repo + testbox --- lib/job.rb | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 7fc7cb5b6..064a2b1dd 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -366,12 +366,15 @@ class Job end def get_hosts_file - testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" - return testbox_file if File.file?(testbox_file) - + lab_repo = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}" tbox_group = @job['tbox_group'].split(/\.|--/)[0] - tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" - return tbox_group_file if File.file?(tbox_group_file) + + ["#{lab_repo}/hosts/#{@job['testbox']}", + "#{LKP_SRC}/hosts/#{@job['testbox']}", + "#{lab_repo}/hosts/#{tbox_group}", + "#{LKP_SRC}/hosts/#{tbox_group}"].each do |file| + return file if File.file?(file) + end raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end -- 2.23.0

1 0

[PATCH compass-ci] container/dracut-initrd: use rootfs_src to replace root in local mount
by Xu Xijian 24 Mar '21

24 Mar '21

$rootfs_src is what we want to get timestamp and os_info, $rootfs_src is fixed formatted, $root can have several other format. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 240551a..835bdd2 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -4,7 +4,7 @@ analyse_kernel_cmdline_params() { rootfs="$(getarg root=)" # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') - timestamp="$(basename $root)" + timestamp="$(basename $rootfs_src)" os="$(echo $rootfs_src | awk -F '/|-' '{print $2}')" os_arch="$(echo $rootfs_src | awk -F '/|-' '{print $3}')" os_version="$(echo $rootfs_src | awk -F '/|-' '{print $4}')" -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: fix host file not exist
by Wei Jihui 24 Mar '21

24 Mar '21

add check host file at: 1. lab repo + tbox_group 2. lkp_src repo + testbox Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 7fc7cb5b..c1df0f5b 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -366,12 +366,16 @@ class Job end def get_hosts_file - testbox_file = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}/hosts/#{@job['testbox']}" - return testbox_file if File.file?(testbox_file) - + testbox = @job['testbox'] tbox_group = @job['tbox_group'].split(/\.|--/)[0] - tbox_group_file = "#{LKP_SRC}/hosts/#{tbox_group}" - return tbox_group_file if File.file?(tbox_group_file) + lab_repo = "#{ENV['CCI_REPOS']}/lab-#{@job['lab']}" + + ["#{lab_repo}/hosts/#{testbox}", + "#{lab_repo}/hosts/#{tbox_group}", + "#{LKP_SRC}/hosts/#{testbox}", + "#{LKP_SRC}/hosts/#{tbox_group}"].each do |file| + return file if File.file?(file) + end raise ArgumentError, "hosts file not exist: #{tbox_group}, maybe need check testbox field" end -- 2.23.0

2 1

[PATCH v2 compass-ci 3/3] container/mail-robot: optimize parsing email
by Luan Shengde 24 Mar '21

24 Mar '21

optimize code: use gem resource ruby-git instead of system command Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index e6c35f7..ce01fec 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -3,9 +3,13 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +# set default system encoding +Encoding.default_external = Encoding::UTF_8 + require 'json' require 'mail' require 'rest-client' +require 'git' # used to parse mail_content for my_commit_url and my_ssh_pubkey # be called by AssignAccount when it needs to extract required data: @@ -110,14 +114,15 @@ class ParseApplyAccountEmail repo_dir = repo_url.split('/')[-1] commit_id = url.split('/')[-1] - %x(/usr/bin/git -C /tmp clone --bare #{repo_url} #{repo_dir}) + Git.clone(repo_url, repo_dir, path: '/tmp/', bare: true) # get all commit IDs and check commit id exists - repo_commits = %x(/usr/bin/git -C /tmp/#{repo_dir} log --pretty=format:'%H').split(/\n/) + repo = Git.bare("/tmp/#{repo_dir}") + repo_commits = repo.lib.log_commits check_commit_exist(commit_id, repo_commits) # get the auther's email for the commit - author_email = %x(/usr/bin/git -C /tmp/#{repo_dir} show -s --format=%aE #{commit_id}).chomp + author_email = repo.gcommit(commit_id).author.email check_commit_email(author_email) FileUtils.rm_rf "/tmp/#{repo_dir}" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container/mail-robot: add use ENV variable
by Luan Shengde 24 Mar '21

24 Mar '21

use ENV variable instead of full-path string enable modify the path without re-edit the code. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 2 +- container/mail-robot/start | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 17705ec..97c3989 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -56,7 +56,7 @@ class ApplyAccount @es_port = ES_PORT # email info file for account issuers. - @account_issuer = YAML.load_file '/etc/compass-ci/account-issuer.yaml' + @account_issuer = YAML.load_file ENV['ACCOUNT_ISSUER'] @my_info = {} end diff --git a/container/mail-robot/start b/container/mail-robot/start index bd2a62d..ce18498 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -16,6 +16,7 @@ names = Set.new %w[ SCHED_HTTPS_PORT ES_PORT MAILDIR + ACCOUNT_ISSUER lab ] -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/mail-robot: add install ruby-git
by Luan Shengde 24 Mar '21

24 Mar '21

add install ruby-git in dockerfile Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile index e4b3e7c..41056a5 100644 --- a/container/mail-robot/Dockerfile +++ b/container/mail-robot/Dockerfile @@ -10,7 +10,7 @@ ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y git uuid-runtime ruby-listen ruby-json ruby-mail ruby-rest-client + apt-get install -y git uuid-runtime ruby-listen ruby-json ruby-mail ruby-rest-client ruby-git RUN gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ -- 2.23.0

1 0