February 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] constants.rb: fix warning: already initialized constant
by Cao Xueliang 07 Feb '21

07 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/constants.rb | 22 +++++++++++----------- src/delimiter/constants.rb | 16 ++++++++-------- 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/lib/constants.rb b/lib/constants.rb index 27f37cb..5ed4d57 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -5,19 +5,19 @@ require_relative '../container/defconfig.rb' config = cci_defaults -ES_HOST = config['ES_HOST'] || '172.17.0.1' -ES_PORT = config['ES_PORT'] || 9200 +ES_HOST ||= config['ES_HOST'] || '172.17.0.1' +ES_PORT ||= config['ES_PORT'] || 9200 -SEND_MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' -SEND_MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_HOST ||= config['SEND_MAIL_HOST'] || '172.17.0.1' +SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 49000 -SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' -SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 +SRV_HTTP_HOST ||= config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' +SRV_HTTP_PORT ||= config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 -SRV_HTTP_DOMAIN = config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' +SRV_HTTP_DOMAIN ||= config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' -ASSIST_RESULT_HOST = config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' -ASSIST_RESULT_PORT = config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 +ASSIST_RESULT_HOST ||= config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' +ASSIST_RESULT_PORT ||= config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 -SCHED_HOST = config['SCHED_HOST'] || '172.17.0.1' -SCHED_PORT = config['SCHED_PORT'] || 3000 +SCHED_HOST ||= config['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT ||= config['SCHED_PORT'] || 3000 diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 95d1acd..a790736 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -2,14 +2,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' -GIT_MIRROR_HOST = ENV['GIT_MIRROR_HOST'] || '172.17.0.1' -MONITOR_HOST = ENV['MONITOR_HOST'] || '172.17.0.1' -MONITOR_PORT = ENV['MONITOR_PORT'] || '11310' -DELIMITER_EMAIL = ENV['DELIMITER_EMAIL'] || 'delimiter@localhost' -TMEP_GIT_BASE = '/c/public_git' -DELIMITER_TASK_QUEUE = 'delimiter' +LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' +MONITOR_HOST ||= ENV['MONITOR_HOST'] || '172.17.0.1' +MONITOR_PORT ||= ENV['MONITOR_PORT'] || '11310' +GIT_MIRROR_HOST ||= ENV['GIT_MIRROR_HOST'] || '172.17.0.1' +DELIMITER_EMAIL ||= ENV['DELIMITER_EMAIL'] || 'delimiter@localhost' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" # The files which are in this dir can be uploaded by lkp-tests -TMP_RESULT_ROOT = ENV['TMP_RESULT_ROOT'] || '/tmp/lkp/result' +TMP_RESULT_ROOT ||= ENV['TMP_RESULT_ROOT'] || '/tmp/lkp/result' +TMEP_GIT_BASE = '/c/public_git' PROCESS_JOB_YAML = "/tmp/process.yaml" +DELIMITER_TASK_QUEUE = 'delimiter' -- 2.23.0

1 0

[PATCH compass-ci] container/dracut-initrd: change filename from add-cifs.conf to add-modules.conf
by Xu Xijian 07 Feb '21

07 Feb '21

since finename add-cifs.conf has changed to add-modules.conf, here modify the source file of COPY in Dockerfile. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/Dockerfile b/container/dracut-initrd/Dockerfile index 4208b98..7f8448f 100644 --- a/container/dracut-initrd/Dockerfile +++ b/container/dracut-initrd/Dockerfile @@ -7,7 +7,7 @@ MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> ADD bin /usr/local/bin COPY conf/sources.list* /etc/apt/ -COPY conf/add-cifs.conf /etc/dracut.conf.d/ +COPY conf/add-modules.conf /etc/dracut.conf.d/ WORKDIR /usr/local/bin RUN setup-dracut.sh -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] container/register-accounts: fix failed to register account
by Liu Yinsi 07 Feb '21

07 Feb '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/register-accounts# ./start register-accounts register-accounts /c/compass-ci/container/register-accounts # ./register-accounts awk: /etc/compass-ci/defaults/*.yaml: No such file or directory "option: {\"my_email\"=>\"\", \"my_name\"=>\"\", \"lab\"=>\"\"}" Traceback (most recent call last): 10: from ./register-accounts:8:in `<main>' 9: from ./register-accounts:8:in `require_relative' 8: from /c/compass-ci/sbin/build-my-info.rb:57:in `<top (required)>' 7: from /c/compass-ci/sbin/build-my-info.rb:48:in `build_my_info' 6: from /c/compass-ci/lib/build_my_info_client.rb:30:in `search_my_info' 5: from /c/compass-ci/lib/es_query.rb:58:in `query_by_id' 4: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-api-7.10.1/lib/elasticsearch/api/actions/search.rb:103:in `search' 3: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/client.rb:182:in `perform_request' 2: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request' 1: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/base.rb:347:in `perform_request' /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error': [400] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: {\\n \\"bool\\" : {\\n \\"must\\" : [\\n {\\n \\"term\\" : {\\n \\"_id\\" : {\\n \\"value\\" : \\"\\",\\n \\"boost\\" : 1.0\\n }\\n }\\n }\\n ],\\n \\"adjust_pure_negative\\" : true,\\n \\"boost\\" : 1.0\\n }\\n}","index_uuid":"MH_AlDBvTYa9sQhwsESXgQ","index":"accounts"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"accounts","node":"I1usO2axQ9-XZhzcX3oHxw","reason":{"type":"query_shard_exception","reason":"failed to create query: {\\n \\"bool\\" : {\\n \\"must\\" : [\\n {\\n \\"term\\" : {\\n \\"_id\\" : {\\n \\"value\\" : \\"\\",\\n \\"boost\\" : 1.0\\n }\\n }\\n }\\n ],\\n \\"adjust_pure_negative\\" : true,\\n \\"boost\\" : 1.0\\n }\\n}","index_uuid":"MH_AlDBvTYa9sQhwsESXgQ","index":"accounts","caused_by":{"type":"illegal_argument_exception","reason":"Ids can't be empty"}}}]},"status":400} (Elasticsearch::Transport::Transport::Errors::BadRequest) refer code: /c/compass-ci/sbin/build-my-info.rb: ... option = { 'my_email' => `git config --global user.email`.chomp, 'my_name' => `git config --global user.name`.chomp, 'lab' => `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp } ... build_my_info(option) ... [why] in a new running container environment, not exists git config (user email, user name) and config yaml (/etc/compass-ci/defaults/*.yaml), so option is empty, then failed to register account. check option['my_email'] whether empty or not is necessary, if option['my_email'] is null, not call function build_my_info(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/build-my-info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index dbe093d..1b0df0d 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -52,4 +52,4 @@ def build_my_info(option) build_my_info.config_my_info(info_es) end -build_my_info(option) +build_my_info(option) unless option['my_email'].empty? -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/register-accounts: fix env lab not exist
by Liu Yinsi 07 Feb '21

07 Feb '21

container/register-accounts: ... my_info['lab'] = ENV['lab'] ... [why] when use env lab in container, must add into env in start script, or can't get env lab. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/register-accounts/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/register-accounts/start b/container/register-accounts/start index b1ebe98..c2acc75 100755 --- a/container/register-accounts/start +++ b/container/register-accounts/start @@ -9,6 +9,7 @@ require_relative '../defconfig.rb' names = Set.new %w[ ES_HOST ES_PORT + lab ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: add to adjust local mount when get_initrds
by Xu Xijian 07 Feb '21

07 Feb '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index c013aa7..89c92c1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -509,7 +509,7 @@ class Job if "#{os_mount}" == "initramfs" temp_initrds.concat(initramfs_initrds()) - elsif "#{os_mount}" == "nfs" || "#{os_mount}" == "cifs" + elsif "#{os_mount}" == "nfs" || "#{os_mount}" == "cifs" || "#{os_mount}" == "local" temp_initrds.concat(nfs_cifs_initrds()) end -- 2.23.0

2 2

[PATCH v2 compass-ci] doc/manual: add document apply-account.zh.md
by Luan Shengde 07 Feb '21

07 Feb '21

add Chinese document for applying account. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/apply-account.zh.md | 58 ++++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 doc/manual/apply-account.zh.md diff --git a/doc/manual/apply-account.zh.md b/doc/manual/apply-account.zh.md new file mode 100644 index 0000000..5e6605d --- /dev/null +++ b/doc/manual/apply-account.zh.md @@ -0,0 +1,58 @@ +# 账号申请 + +首先您需要一个账号，可以用来提交job，同时跟踪您的资源消费情况。 +因为目前我们的测试机资源池有限，因此需要做一些使用限制。 + +我们为平台测试所覆盖的开源软件（OSS）的贡献者提供了免费的账号。 +你可以通过在下面邮件中提供‘my oss commit’来验明正身您是一个开源软件（OSS）的贡献者。 + +我们同时也为我们的合作者提供了免费的账号。 +这种情况需要提供用的邮件，名字，ssh公钥，以及申请测试机的目的。 + +## 1. 发送邮件进行账号申请 + +邮件模板： + +--- + 发件人: {{ David Rientjes <rientjes(a)google.com> }} + 收件人: compass-ci-robot(a)qq.com + 邮件标题: apply account + + my oss commit: {{ https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… }} + + {{ 将您的ssh公钥添加为邮件附件，例如 ~/.ssh/id_rsa.pub }} + +--- + +- 使用您的信息替换{{ }}中的内容 +- 邮件名应该使用英文或中文拼音，例如模板中的“David Rientjes” +- my oss commit: 使用您的邮件和名字提交的代码的地址。 + + + 您的仓库需要在平台测试覆盖的范围内，并且注册到： + [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). + 如果贡献的OSS工程仓库不在upstream-repos中，请参考： + [test-oss-project](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/… + 将仓库注册到compass-ci测试池中。 + +## 2. 接收邮件 + +申请邮件成功后，您将收到反馈邮件，邮件内容包括以下信息： + + my_name + my_email + my_token + lab + +## 3. 一次性配置 + +根据步骤2收到邮件的提示，配置您的本地环境： + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + setup ~/.config/compass-ci/defaults/account.yaml + ~/.config/compass-ci/include/lab/{{ lab }}.yaml + +## 4. 现在开始 + +[提交job到compass-ci](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submit-job.en.md) + -- 2.23.0

1 0

[PATCH compass-ci] lib/git_bisect.rb: delete --first-parent in git start bisect command
by Cao Xueliang 07 Feb '21

07 Feb '21

The option will follow only the first parent commit upon seeing a merge commit, but we need bisect the commits in the merge, so, delete the option. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 40f6b49..dd0b782 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -89,7 +89,7 @@ class GitBisect puts "bad_commit: #{@bad_commit}" puts "good_commit: #{@good_commit}" - result = `git -C #{@work_dir} bisect start --first-parent #{@bad_commit} #{@good_commit}` + result = `git -C #{@work_dir} bisect start #{@bad_commit} #{@good_commit}` temp = result.split(/\n/) if temp[0].include? 'Bisecting' result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} #{(a)error_id.inspect} #{@work_dir}` -- 2.23.0

1 0

[PATCH v4 compass-ci] sbin: add script for download os rootfs
by Liu Yinsi 07 Feb '21

07 Feb '21

For locally deploy compass-ci, download os rootfs to run qemu test, the default download os rootfs is openeuler aarch64 20.03, user can also use this script to download other os rootfs if they need. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 sbin/download-rootfs diff --git a/sbin/download-rootfs b/sbin/download-rootfs new file mode 100755 index 0000000..208ac7d --- /dev/null +++ b/sbin/download-rootfs @@ -0,0 +1,40 @@ +#!/bin/bash +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# download os rootfs. +# 1. download default os rootfs openeuler aarch64 20.03 +# example usage: +# ./download-rootfs +# 2. download os rootfs you want by add params $os, $os_arch, $os_version. +# example usage: +# ./download-rootfs debian aarch64 sid + +os=${1:-openeuler} +os_arch=${2:-aarch64} +os_version=${3:-20.03} + +. ${CCI_SRC}/container/defconfig.sh +. ${CCI_SRC}/lib/log.sh + +load_cci_defaults + +rootfs_dir=${os}/${os_arch}/${os_version} + +download_os_rootfs() +{ + mkdir -p /srv/os/${rootfs_dir} + cd /srv/os/$rootfs_dir + + download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${rootfs_dir}.cgz" + wget $download_url + + [ "$?" -ne 0 ] && { + log_error "failed to download os rootfs: $download_url" + exit 1 + } + + gzip -dc ${os_version}.cgz | cpio -id + rm -rf ${os_version}.cgz +} + +download_os_rootfs -- 2.23.0

4 8

[PATCH compass-ci] container/openresty-proxy-cache: fix build error
by Liu Yinsi 07 Feb '21

07 Feb '21

when execute openresty-proxy-cache/build, if openresty already exists [error] fatal: destination path 'openresty' already exists and is not an empty directory. [how] delete openresty after finish build container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/openresty-proxy-cache/build | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/openresty-proxy-cache/build b/container/openresty-proxy-cache/build index 0846197..e77c19e 100755 --- a/container/openresty-proxy-cache/build +++ b/container/openresty-proxy-cache/build @@ -5,3 +5,5 @@ git clone https://gitee.com/cuiyili/openresty.git || exit 1 docker build -t openresty:proxy_cache . -f openresty/Dockerfile + +rm -rf openresty -- 2.23.0

3 3

[PATCH compass-ci 1/4] container/defconfig: add function docker_skip_rebuild
by Liu Yinsi 07 Feb '21

07 Feb '21

when execute `sparrow/4-docker/buildall run-only`, if caller script export env DOCKER_SKIP_REBUILD, then image already is exist, just exit build script, not rebuild container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 6 ++++++ container/defconfig.sh | 9 +++++++++ 2 files changed, 15 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index f9b0824..1a655a3 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -47,3 +47,9 @@ def get_available_memory # take the middle value according to the system memory size. [1024, 30720, Math.sqrt(memtotal) * 1024].sort[1].to_i end + +def docker_skip_rebuild(tag) + if ENV.include? 'DOCKER_SKIP_REBUILD' + exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" + end +end diff --git a/container/defconfig.sh b/container/defconfig.sh index 54e65b5..7612e2d 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -41,3 +41,12 @@ push_image() docker tag "$src_tag" "$dst_tag" docker push "$dst_tag" } + +docker_skip_rebuild() +{ + tag=$1 + [ $DOCKER_SKIP_REBUILD ] && { + docker image inspect $tag > /dev/null 2>&1 + [ "$?" == "0" ] && exit 1 + } +} -- 2.23.0

2 2