- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] doc: update invalid URL
by Liu Yinsi 12 Apr '21

12 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- doc/manual/apply-account.md | 2 +- doc/manual/apply-account.zh.md | 2 +- doc/tutorial.md | 43 +--------------------------------- 3 files changed, 3 insertions(+), 44 deletions(-) diff --git a/doc/manual/apply-account.md b/doc/manual/apply-account.md index 542251d..69e678a 100644 --- a/doc/manual/apply-account.md +++ b/doc/manual/apply-account.md @@ -33,7 +33,7 @@ Email template: The git repo should be in our test coverage and registered in [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). If your contributed OSS project is not in upstream-repos, you may refer to - [test-oss-project](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/… + [test-oss-project](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… to add it to compass-ci testing. ## step 2: receive an email diff --git a/doc/manual/apply-account.zh.md b/doc/manual/apply-account.zh.md index 849364f..d177a76 100644 --- a/doc/manual/apply-account.zh.md +++ b/doc/manual/apply-account.zh.md @@ -32,7 +32,7 @@ 您的仓库需要在平台测试覆盖的范围内，并且注册到： [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos). 如果贡献的OSS工程仓库不在upstream-repos中，请参考： - [test-oss-project](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/… + [test-oss-project](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc… 将仓库注册到compass-ci测试池中。 ## 2. 接收邮件 diff --git a/doc/tutorial.md b/doc/tutorial.md index b04bf70..cd72d5c 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -280,48 +280,7 @@ job yaml 是测试描述和执行的基本单元，以[YAML](http://yaml.org/YAM -### 本地搭建 compass-ci 服务器节点 - -在 openEuler 系统一键部署 compass-ci 环境，当前已支持 openEuler-aarch64-20.03-LTS 系统环境，以下配置仅供参考。 - -#### 准备工作 -- 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 32GB - CPU：64 nuclear (建议) - 硬盘：>= 500G (建议划分独立分区) - - >**说明：** - >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - -- 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 - - >**说明：** - >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… - - -#### 操作指导 -1. 登录 openEuler 系统 - -2. 创建工作目录并设置文件权限 - ```bash - mkdir demo && cd demo && umask 002 - ``` -3. 克隆 compass-ci 项目代码到 demo 目录 - ```bash - git clone https://gitee.com/wu_fengguang/compass-ci.git - ``` -4. 执行一键部署脚本 install-tiny - ```bash - cd compass-ci/sparrow && ./install-tiny - ``` - +### [本地搭建 compass-ci 服务器节点](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/README… ## 呼吁合作 - 增强 git bisect 能力 -- 2.23.0

1 0

[PATCH lkp-tests] tests/openeuler_docker: optimize file name error
by Liu Shaofei 12 Apr '21

12 Apr '21

[why] ./git--github.com-peternav-myrepo.git/z9.2062565/dockerfile:from_openeuler_1 => ./git--github.com-peternav-myrepo.git/z9.2062565/peternav-myrepo_1 Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/openeuler_docker | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tests/openeuler_docker b/tests/openeuler_docker index 3ea57e2c9..906d6c88c 100755 --- a/tests/openeuler_docker +++ b/tests/openeuler_docker @@ -19,8 +19,10 @@ declare -i dockerfile_count=0 fit_repo() { yum reinstall -y git - git clone "$dockerfile_url" "$BENCHMARK_ROOT/src/" - dockerfile_path="$(find "$BENCHMARK_ROOT/src/" -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]')" + project_name="${dockerfile_url##*/}" + project_name="${project_name%.git}" + git clone "$dockerfile_url" "$BENCHMARK_ROOT/{project_name}/" + dockerfile_path="$(find "$BENCHMARK_ROOT/{project_name}/" -type f -name "*Dockerfile*" | xargs grep -HE '^FROM centos:(centos|)[78]')" dockerfile_count=$(( $(echo $dockerfile_path | awk '{print NF}') / 2 )) [ -n "$dockerfile_path" ] || { echo "current-repo-not-support-centos: 1" @@ -45,7 +47,7 @@ build_centos() build_openeuler() { - docker build -f "$dockerfile_name" -t "${image}_$1_$index" . >> $TMP_RESULT_ROOT/${image}_$1_$index 2>&1 + docker build -f "$dockerfile_name" -t "${image}_$1_$index" . >> $TMP_RESULT_ROOT/${image}_$index 2>&1 [ $? != 0 ] && { let nr_openeuler_fail++ echo "${image}_$index.build.fail: 1" @@ -60,7 +62,7 @@ docker_run() { docker_run_cmd="docker run -d --name ${image}_$1_$index ${image}_$1_$index" [ $1 == "openeuler" ] && { - docker_run_cmd="${docker_run_cmd} >> $TMP_RESULT_ROOT/${image}_$1_$index 2>&1" + docker_run_cmd="${docker_run_cmd} >> $TMP_RESULT_ROOT/${image}_$index 2>&1" } eval $docker_run_cmd @@ -84,6 +86,9 @@ handle_build_result() elif [ "$nr_openeuler_success" == "$dockerfile_count" ]; then echo "openeuler_valid_url: $dockerfile_url" exit 0 + else + echo "Build CentOS images failed." + exit 0 fi } @@ -91,7 +96,7 @@ build() { dockerfile_name="${file##*/}" dockerfile_name="${dockerfile_name%:*}" - image=$(echo "$file" | awk -F '/' '{print $5}' | tr 'A-Z' 'a-z') + image=$(echo "$file" | awk -F '/' '$5~/Dockerfile/ {print $4} $5!~/Dockerfile/ {print $5}' | tr 'A-Z' 'a-z') build_centos "centos" > /dev/null 2>&1 || return 1 handle_dockerfile "$dockerfile_name" -- 2.23.0

2 2

git count
by Wu Fengguang 12 Apr '21

12 Apr '21

Cao Xueliang is our code star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./git-count 14 Cao Xueliang 8 Wu Zhende 6 Lu Weitao 5 Li Ping 3 Liu Shaofei 3 Li Yuanchao 2 Bai Jing 2 Wei Jihui 2 Cui Yili 2 Wang Yong 2 Lin Jiaxin 1 Xu Xijian 1 Wang Chenglong 1 Luan Shengde 1 Zhang Dewan 0 Du Kaitian 0 Liu Yinsi 0 Wu Fengguang 0 Zhang Yale 0 Yu Chuan

1 0

[PATCH compass-ci] sparrow: fix docker pull local repos failed
by Cao Xueliang 12 Apr '21

12 Apr '21

[Error] docker pull 172.168.131.2:5001/openeuler:20.03-LTS Error response from daemon: Get https://172.168.177.252:5001/v2/: http:server gave HTTP response to HTTPS client [why] There is no DOCKER_REGISTRY_HOST and DOCKER_REGISTRY_PORT in mutli-docker job, so use localhost value as the DOCKER_REGISTRY_HOST. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index fd529e7..3aeba90 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -149,6 +149,8 @@ ASSIST_RESULT_HOST: ASSIST_RESULT_PORT: RESULT_WEBDAV_HOST: RESULT_WEBDAV_PORT: +DOCKER_REGISTRY_HOST: +DOCKER_REGISTRY_PORT: EOF cat > /etc/profile.d/compass.sh <<'EOF' -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib: add check_public_network.sh
by Liu Yinsi 12 Apr '21

12 Apr '21

Check public network ok or not during locally deploy Compass CI, because users need to sparrow in a pure intranet environment. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/check_public_network.sh | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 lib/check_public_network.sh diff --git a/lib/check_public_network.sh b/lib/check_public_network.sh new file mode 100644 index 0000000..19a0f02 --- /dev/null +++ b/lib/check_public_network.sh @@ -0,0 +1,8 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +public_network_ok() +{ + ping -c 1 -w 1 114.114.114.114 >/dev/null 2>&1 +} -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow: call public_network_ok
by Liu Yinsi 12 Apr '21

12 Apr '21

[why] For user locally deploy Compass CI, if public network not ok, they will install pkg by themselves, in order to fix error message, so call function public_network_ok(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 8 ++++++-- sparrow/0-package/os/openEuler | 14 +++++++++----- sparrow/5-build/os-ready | 4 +++- 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 9fdb4be..eb86b24 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,8 +2,12 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && { + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ + gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch bunny +} grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 9cb7b04..e742fa1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -2,6 +2,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. $CCI_SRC/lib/check_public_network.sh + pkgs=( vim git @@ -26,10 +28,12 @@ pkgs=( sudo ) -yum install -y --skip-broken "${pkgs[@]}" +public_network_ok && { + yum install -y --skip-broken "${pkgs[@]}" -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… + wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… -rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm -rm -rf *.rpm + rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm + rm -rf *.rpm +} diff --git a/sparrow/5-build/os-ready b/sparrow/5-build/os-ready index 48a740b..5e1acc4 100755 --- a/sparrow/5-build/os-ready +++ b/sparrow/5-build/os-ready @@ -2,4 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -$CCI_SRC/sbin/download-rootfs +. $CCI_SRC/lib/check_public_network.sh + +public_network_ok && $CCI_SRC/sbin/download-rootfs -- 2.23.0

1 0

[PATCH lkp-tests] spec/submit_spec.rb: delete submit_id when do submit spec
by Wei Jihui 12 Apr '21

12 Apr '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit_spec.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index bc411325..233aea26 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -3,7 +3,7 @@ require "#{LKP_SRC}/lib/yaml" def stable_yaml_file(yaml_file) job = load_yaml(yaml_file) - delete_job_key = %w(LKP_DEBUG_PREFIX lkp_initrd_user job_origin my_token my_email my_name) + delete_job_key = %w(LKP_DEBUG_PREFIX lkp_initrd_user job_origin my_token my_email my_name submit_id) job.delete_if do |key, _| delete_job_key.include?(key) || key.to_s.start_with?('#! ') || -- 2.23.0

1 0

[PATCH compass-ci 2/2] service/scheduler: set time when update job
by Wu Zhende 12 Apr '21

12 Apr '21

Can use time in kibana to limit the search scope Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 4 ++++ src/scheduler/elasticsearch_client.cr | 3 +++ 2 files changed, 7 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8691082..b6bb730 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -315,6 +315,10 @@ class Job return package_dir end + def set_time + self["time"] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") + end + def set_upload_dirs self["upload_dirs"] = "#{result_root}#{get_package_dir}" end diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 58bed08..0e83ff1 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -36,6 +36,9 @@ class Elasticsearch::Client # caller should judge response["_id"] != nil def set_job_content(job : Job) + # time indicates the update time of each job event + job.set_time + response = get_job_content(job.id) if response["id"]? response = update(job.dump_to_json_any, job.id) -- 2.23.0

1 0

[PATCH v1 compass-ci] extract-stats: add Fiber.yield after spawn operate
by Cao Xueliang 12 Apr '21

12 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/extract_stats.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index c35d3ec..89cd89e 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -42,8 +42,7 @@ module ExtractStats task = tasks.delete_at(0) spawn { StatsWorker.new.handle(task.key, channel) } - - sleep 1 + Fiber.yield end end @@ -63,8 +62,7 @@ module ExtractStats while true key = channel.receive spawn { StatsWorker.new.handle(key, channel) } - - sleep 1 + Fiber.yield end end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] bin/es-jobs-mapping.sh: add time to mapping
by Wu Zhende 12 Apr '21

12 Apr '21

Add the time of the date type to the ES mapping. "time" indicates the last update time of the job. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/es-jobs-mapping.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 502e099..9926fd8 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -88,6 +88,9 @@ else "errid": { "type": "text" }, + "time": { + "type": "date" + }, "category": { "type": "keyword" }, -- 2.23.0

1 0

2024

2023

2022

2021

2020

Compass-ci