- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] extract-stats: wait_for refresh doc to ES DB
by Lu Weitao 12 Apr '21

12 Apr '21

[Why] We need update doc(job) to ES completed, then send delimiter task to task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 45b75a3..56dad9e 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -62,6 +62,7 @@ class StatsWorker @es.@client.update( { :index => "jobs", :type => "_doc", + :refresh => "wait_for", :id => job_id, :body => {:doc => update_content}, } -- 2.23.0

2 1

[PATCH compass-ci] doc: update invalid URL
by Liu Yinsi 12 Apr '21

12 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- doc/manual/apply-account.md | 2 +- doc/manual/apply-account.zh.md | 2 +- doc/tutorial.md | 43 +--------------------------------- 3 files changed, 3 insertions(+), 44 deletions(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 542251d..69e678a 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -33,7 +33,7 @@ Email template: The git repo should be in our test coverage and registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). If your contributed OSS project is not in upstream-repos, you may refer to - [test-oss-project](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/… + [test-oss-project](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… to add it to compass-ci testing. ## step 2: receive an email diff --git a/doc/manual/apply-account.zh.md b/doc/manual/apply-account.zh.md index 849364f..d177a76 100644 --- a/doc/manual/apply-account.zh.md +++ b/doc/manual/apply-account.zh.md @@ -32,7 +32,7 @@ 您的仓库需要在平台测试覆盖的范围内，并且注册到： [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). 如果贡献的OSS工程仓库不在upstream-repos中，请参考： - [test-oss-project](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/… + [test-oss-project](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 将仓库注册到compass-ci测试池中。 ## 2. 接收邮件 diff --git a/doc/tutorial.md b/doc/tutorial.md index b04bf70..cd72d5c 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -280,48 +280,7 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM -### 本地搭建 compass-ci 服务器节点 - -在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 - -#### 准备工作 -- 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 32GB - CPU：64 nuclear (建议) - 硬盘：>= 500G (建议划分独立分区) - - >**说明：** - >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - -- 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 - - >**说明：** - >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… - - -#### 操作指导 -1. 登录 openEuler 系统 - -2. 创建工作目录并设置文件权限 - ```bash - mkdir demo && cd demo && umask 002 - ``` -3. 克隆 compass-ci 项目代码到 demo 目录 - ```bash - git clone https://gitee.com/wu_fengguang/compass-ci.git - ``` -4. 执行一键部署脚本 install-tiny - ```bash - cd compass-ci/sparrow && ./install-tiny - ``` - +### [本地搭建 compass-ci 服务器节点](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/README… ## 呼吁合作 - 增强 git bisect 能力 -- 2.23.0

1 0

[PATCH lkp-tests] tests/openeuler_docker: optimize file name error
by Liu Shaofei 12 Apr '21

12 Apr '21

[why] ./git--github.com-peternav-myrepo.git/z9.2062565/dockerfile:from_openeuler_1 => ./git--github.com-peternav-myrepo.git/z9.2062565/peternav-myrepo_1 Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/openeuler_docker | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tests/openeuler_docker b/tests/openeuler_docker index 3ea57e2c9..906d6c88c 100755 --- a/tests/openeuler_docker +++ b/tests/openeuler_docker @@ -19,8 +19,10 @@ declare -i dockerfile_count=0 fit_repo() { yum reinstall -y git - git clone "$dockerfile_url" "$BENCHMARK_ROOT/src/" - dockerfile_path="$(find "$BENCHMARK_ROOT/src/" -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]')" + project_name="${dockerfile_url##*/}" + project_name="${project_name%.git}" + git clone "$dockerfile_url" "$BENCHMARK_ROOT/{project_name}/" + dockerfile_path="$(find "$BENCHMARK_ROOT/{project_name}/" -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]')" dockerfile_count=$(( $(echo $dockerfile_path | awk '{print NF}') / 2 )) [ -n "$dockerfile_path" ] || { echo "current-repo-not-support-centos: 1" @@ -45,7 +47,7 @@ build_centos() build_openeuler() { - docker build -f "$dockerfile_name" -t "${image}_$1_$index" . >> $TMP_RESULT_ROOT/${image}_$1_$index 2>&1 + docker build -f "$dockerfile_name" -t "${image}_$1_$index" . >> $TMP_RESULT_ROOT/${image}_$index 2>&1 [ $? != 0 ] && { let nr_openeuler_fail++ echo "${image}_$index.build.fail: 1" @@ -60,7 +62,7 @@ docker_run() { docker_run_cmd="docker run -d --name ${image}_$1_$index ${image}_$1_$index" [ $1 == "openeuler" ] && { - docker_run_cmd="${docker_run_cmd} >> $TMP_RESULT_ROOT/${image}_$1_$index 2>&1" + docker_run_cmd="${docker_run_cmd} >> $TMP_RESULT_ROOT/${image}_$index 2>&1" } eval $docker_run_cmd @@ -84,6 +86,9 @@ handle_build_result() elif [ "$nr_openeuler_success" == "$dockerfile_count" ]; then echo "openeuler_valid_url: $dockerfile_url" exit 0 + else + echo "Build CentOS images failed." + exit 0 fi } @@ -91,7 +96,7 @@ build() { dockerfile_name="${file##*/}" dockerfile_name="${dockerfile_name%:*}" - image=$(echo "$file" | awk -F '/' '{print $5}' | tr 'A-Z' 'a-z') + image=$(echo "$file" | awk -F '/' '$5~/Dockerfile/ {print $4} $5!~/Dockerfile/ {print $5}' | tr 'A-Z' 'a-z') build_centos "centos" > /dev/null 2>&1 || return 1 handle_dockerfile "$dockerfile_name" -- 2.23.0

2 2

git count
by Wu Fengguang 12 Apr '21

12 Apr '21

Cao Xueliang is our code star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./git-count 14 Cao Xueliang 8 Wu Zhende 6 Lu Weitao 5 Li Ping 3 Liu Shaofei 3 Li Yuanchao 2 Bai Jing 2 Wei Jihui 2 Cui Yili 2 Wang Yong 2 Lin Jiaxin 1 Xu Xijian 1 Wang Chenglong 1 Luan Shengde 1 Zhang Dewan 0 Du Kaitian 0 Liu Yinsi 0 Wu Fengguang 0 Zhang Yale 0 Yu Chuan

1 0

[PATCH compass-ci] sparrow: fix docker pull local repos failed
by Cao Xueliang 12 Apr '21

12 Apr '21

[Error] docker pull 172.168.131.2:5001/openeuler:20.03-LTS Error response from daemon: Get https://172.168.177.252:5001/v2/: http:server gave HTTP response to HTTPS client [why] There is no DOCKER_REGISTRY_HOST and DOCKER_REGISTRY_PORT in mutli-docker job, so use localhost value as the DOCKER_REGISTRY_HOST. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index fd529e7..3aeba90 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -149,6 +149,8 @@ ASSIST_RESULT_HOST: ASSIST_RESULT_PORT: RESULT_WEBDAV_HOST: RESULT_WEBDAV_PORT: +DOCKER_REGISTRY_HOST: +DOCKER_REGISTRY_PORT: EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib: add check_public_network.sh
by Liu Yinsi 12 Apr '21

12 Apr '21

Check public network ok or not during locally deploy Compass CI, because users need to sparrow in a pure intranet environment. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/check_public_network.sh | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 lib/check_public_network.sh diff --git a/lib/check_public_network.sh b/lib/check_public_network.sh new file mode 100644 index 0000000..19a0f02 --- /dev/null +++ b/lib/check_public_network.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +public_network_ok() +{ + ping -c 1 -w 1 114.114.114.114 >/dev/null 2>&1 +} -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow: call public_network_ok
by Liu Yinsi 12 Apr '21

12 Apr '21

[why] For user locally deploy Compass CI, if public network not ok, they will install pkg by themselves, in order to fix error message, so call function public_network_ok(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 8 ++++++-- sparrow/0-package/os/openEuler | 14 +++++++++----- sparrow/5-build/os-ready | 4 +++- 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 9fdb4be..eb86b24 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,8 +2,12 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && { + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ + gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +} grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 9cb7b04..e742fa1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. $CCI_SRC/lib/check_public_network.sh + pkgs=( vim git @@ -26,10 +28,12 @@ pkgs=( sudo ) -yum install -y --skip-broken "${pkgs[@]}" +public_network_ok && { + yum install -y --skip-broken "${pkgs[@]}" -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm -rm -rf *.rpm + rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm + rm -rf *.rpm +} diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready index 48a740b..5e1acc4 100755 --- a/sparrow/5-build/os-ready +++ b/sparrow/5-build/os-ready @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -$CCI_SRC/sbin/download-rootfs +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && $CCI_SRC/sbin/download-rootfs -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit_spec.rb: delete submit_id when do submit spec
by Wei Jihui 12 Apr '21

12 Apr '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index bc411325..233aea26 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -3,7 +3,7 @@ require "#{LKP_SRC}/lib/yaml" def stable_yaml_file(yaml_file) job = load_yaml(yaml_file) - delete_job_key = %w(LKP_DEBUG_PREFIX lkp_initrd_user job_origin my_token my_email my_name) + delete_job_key = %w(LKP_DEBUG_PREFIX lkp_initrd_user job_origin my_token my_email my_name submit_id) job.delete_if do |key, _| delete_job_key.include?(key) || key.to_s.start_with?('#! ') || -- 2.23.0

1 0

[PATCH compass-ci 2/2] service/scheduler: set time when update job
by Wu Zhende 12 Apr '21

12 Apr '21

Can use time in kibana to limit the search scope Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 4 ++++ src/scheduler/elasticsearch_client.cr | 3 +++ 2 files changed, 7 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8691082..b6bb730 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -315,6 +315,10 @@ class Job return package_dir end + def set_time + self["time"] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") + end + def set_upload_dirs self["upload_dirs"] = "#{result_root}#{get_package_dir}" end diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 58bed08..0e83ff1 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -36,6 +36,9 @@ class Elasticsearch::Client # caller should judge response["_id"] != nil def set_job_content(job : Job) + # time indicates the update time of each job event + job.set_time + response = get_job_content(job.id) if response["id"]? response = update(job.dump_to_json_any, job.id) -- 2.23.0

1 0

[PATCH v1 compass-ci] extract-stats: add Fiber.yield after spawn operate
by Cao Xueliang 12 Apr '21

12 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/extract_stats.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index c35d3ec..89cd89e 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -42,8 +42,7 @@ module ExtractStats task = tasks.delete_at(0) spawn { StatsWorker.new.handle(task.key, channel) } - - sleep 1 + Fiber.yield end end @@ -63,8 +62,7 @@ module ExtractStats while true key = channel.receive spawn { StatsWorker.new.handle(key, channel) } - - sleep 1 + Fiber.yield end end end -- 2.23.0

1 0