- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 3/3] container/logging-es: set available memory size
by Liu Yinsi 01 Dec '20

01 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 6 ++++-- container/logging-es/build | 18 ++++++++++-------- 2 files changed, 14 insertions(+), 10 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 6c01c43..97dd81c 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -4,6 +4,8 @@ ARG BASE_IMAGE FROM $BASE_IMAGE +ARG MEMORY + # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 MAINTAINER Wu Zhende <wuzhende666(a)163.com> @@ -14,8 +16,8 @@ RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elastic sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ - sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ - sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options + sed -i "s/-Xms1g/-Xms${MEMORY}g/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx${MEMORY}g/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/build b/container/logging-es/build index 451041b..bd05c9c 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,14 +1,16 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -declare -A BASE_IMAGE_DICT +require_relative '../defconfig.rb' -BASE_IMAGE_DICT=( - ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" - ["x86_64"]="elasticsearch:7.6.2" -) +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', + 'x86_64' => 'elasticsearch:7.6.2'}.freeze -BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . +available_memory = set_available_memory + +system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." -- 2.23.0

1 0

[PATCH compass-ci 2/3] container/es: set available memory size
by Liu Yinsi 01 Dec '20

01 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/Dockerfile | 6 ++++-- container/es/build | 8 ++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index e626dae..c54f3cf 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -3,6 +3,8 @@ FROM alpine:3.11 +ARG MEMORY + RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add --no-cache elasticsearch curl @@ -27,8 +29,8 @@ RUN rm -rf /etc/init.d/elasticsearch \ RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms20g/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx20g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xms256m/-Xms${MEMORY}g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}g/g" /usr/share/es/config/jvm.options WORKDIR /usr/share/es diff --git a/container/es/build b/container/es/build index d5607e0..7cc7d33 100755 --- a/container/es/build +++ b/container/es/build @@ -1,6 +1,10 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t es643b:alpine311 . --network=host +require_relative '../defconfig.rb' +available_memory = set_available_memory + +system "docker build -t es643b:alpine311 --build-arg MEMORY=#{availale_memory}. --network=host" -- 2.23.0

1 0

[PATCH v3 compass-ci] compare_matrixes.rb: put compare matrix with more members first
by Li Yuanchao 01 Dec '20

01 Dec '20

As hash table is disordered, the output of compare matrixes sometimes would put one member matrix first, that is not a good comparison. We should put good comparison first. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare_matrixes.rb | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d8112e9..9e4850d 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -383,24 +383,33 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group +# the result with more comparison objects first def compare_group_matrices(group_matrices, suites_list, options) result_str = '' - group_matrices.each do |k, v| - matrices_list = [] - matrices_titles = [] - v.each do |dim, matrix| - matrices_titles << dim - matrices_list << matrix - end - if options[:no_print] - result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - else - print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - end + group_matrices_array = sort_by_matrix_size(group_matrices) + group_matrices_array.each do |matrice_kv| + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) end result_str end +def get_matrix_str(matrice_key, matrice_value, suites_list, options) + m_list = [] + m_titles = [] + matrice_value.each do |dim, matrix| + m_titles << dim + m_list << matrix + end + return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + + print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) +end + +# big size first +def sort_by_matrix_size(group_matrices) + group_matrices.sort { |a, b| b[1].size <=> a[1].size } +end + # input: groups_matrices # { # group_key_1 => { -- 2.23.0

1 0

[PATCH compass-ci 4/4] container: add locate_files in assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

An interface is provided to support obtaining the real path of the /srv/initrd/deps and pkg program. Example: Input: curl -H 'Content-Type: Application/json' -XPOST '172.17.0.1:8101/locate_files' -d '{"deps_files": ["/srv/initrd/deps/nfs/openeuler/aarch64/20.03/perf/perf.cgz"], "pkg_files": ["/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/plzip/latest.cgz", "/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/stream/latest.cgz"]}' Output: {"deps_files":["/srv/initrd/deps/nfs/openeuler/aarch64/20.03/perf/perf_20201104.cgz"], "pkg_files":["/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/plzip/1-4.cgz", "/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/stream/1-1.cgz"]} Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/routes.rb | 31 +++++++++++++++++++++++ container/assistant/views/locate_files.rb | 21 +++++++++++++++ 2 files changed, 52 insertions(+) create mode 100755 container/assistant/routes.rb create mode 100644 container/assistant/views/locate_files.rb diff --git a/container/assistant/routes.rb b/container/assistant/routes.rb new file mode 100755 index 0000000..6623fd2 --- /dev/null +++ b/container/assistant/routes.rb @@ -0,0 +1,31 @@ +#!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'open3' + +require_relative './views/locate_files' + +set :bind, '0.0.0.0' +set :port, 8101 + +post '/locate_files' do + request.body.rewind + + begin + data = JSON.parse request.body.read + rescue JSON::ParserError + return [400, 'parse json params error'] + end + + begin + result = locate_files(data) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + [200, result.to_json] +end diff --git a/container/assistant/views/locate_files.rb b/container/assistant/views/locate_files.rb new file mode 100644 index 0000000..5769177 --- /dev/null +++ b/container/assistant/views/locate_files.rb @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def locate_files(data) + result = {} + + data.each do |key, value| + temp = [] + value.each do |val| + val = val.strip + if File.exist?(val) + temp << File.realpath(val) + end + end + + result.merge!({ "#{key}": temp }) + end + + return result +end -- 2.23.0

1 0

[PATCH compass-ci 3/4] container: start script for assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/start | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 container/assistant/start diff --git a/container/assistant/start b/container/assistant/start new file mode 100755 index 0000000..5f7822e --- /dev/null +++ b/container/assistant/start @@ -0,0 +1,21 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +docker_rm assistant + +cmd=( + docker run + --restart=always + --name assistant + -u nobody + -d + -p 8101:8101 + -v /srv/initrd:/srv/initrd + -v /etc/localtime:/etc/localtime:ro + debian:assistant +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci 2/4] container: build script for assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assistant/build diff --git a/container/assistant/build b/container/assistant/build new file mode 100755 index 0000000..7778c9f --- /dev/null +++ b/container/assistant/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t debian:assistant . -- 2.23.0

1 0

[PATCH compass-ci 1/4] container: assistant container Dockerfile
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/assistant/Dockerfile diff --git a/container/assistant/Dockerfile b/container/assistant/Dockerfile new file mode 100644 index 0000000..0287747 --- /dev/null +++ b/container/assistant/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y ruby-json ruby-sinatra curl + +COPY routes.rb /usr/local/bin/ +COPY views /usr/local/bin/views/ + +CMD ["/usr/local/bin/routes.rb"] -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] compare_matrixes.rb: put compare matrix with more members first
by Li Yuanchao 01 Dec '20

01 Dec '20

As hash table is disordered, the output of compare matrixes sometimes would put one member matrix first, that is not a good comparison. We should put good comparison first. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare_matrixes.rb | 33 +++++++++++++++++++++------------ 1 file changed, 21 insertions(+), 12 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index d8112e9..8232513 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -383,24 +383,33 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group +# the result with more comparison objects first def compare_group_matrices(group_matrices, suites_list, options) result_str = '' - group_matrices.each do |k, v| - matrices_list = [] - matrices_titles = [] - v.each do |dim, matrix| - matrices_titles << dim - matrices_list << matrix - end - if options[:no_print] - result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - else - print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) - end + group_matrices_array = sort_by_matrice_size(group_matrices) + group_matrices_array.each do |matrice_kv| + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) end result_str end +def get_matrix_str(matrice_key, matrice_value, suites_list, options) + m_list = [] + m_titles = [] + matrice_value.each do |dim, matrix| + m_titles << dim + m_list << matrix + end + return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + + print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) +end + +# big size first +def sort_by_matrice_size(group_matrices) + group_matrices.sort { |a, b| b[1].size <=> a[1].size } +end + # input: groups_matrices # { # group_key_1 => { -- 2.23.0

2 2

[PATCH lkp-tests 1/2] depends/lkp-dev: add package needed in docker
by Wang Yong 01 Dec '20

01 Dec '20

[error] /lkp/lkp/src/sbin/makepkg: line 403: file: command not found Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/lkp-dev | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/lkp-dev b/distro/depends/lkp-dev index 0e49807e8..555124f96 100644 --- a/distro/depends/lkp-dev +++ b/distro/depends/lkp-dev @@ -27,3 +27,4 @@ flex bison libssl-dev libipc-run-perl +file -- 2.23.0

2 2

[PATCH v2 compass-ci 1/2] delete source and add getting started
by Li Ping 01 Dec '20

01 Dec '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- README.md | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 80e0766..128eba0 100644 --- a/README.md +++ b/README.md @@ -10,50 +10,43 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 - **测试服务** - 使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… +使用Compass-CI 基于开源软件 PR 触发[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - **调测环境登录** - 使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - **测试结果分析** - 通过 [web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 +通过 [Web](https://compass-ci.openeuler.org) 接口，对历史测试结果进行分析和比较。 - **测试结果复现** - Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 +Compass-CI 把测试过程中的各项环境参数保存在任务结果输出的job.yaml文件中，开发者可以通过重新提交job.yaml复现测试结果。 - **辅助定位** - Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 +Compass-CI 可以识别自动化构建测试过程中的错误，触发基于 git tree 的测试，找出引入问题的commit。 -## 使用 Compass-CI -> 您只需要先注册自己的仓库，当您的仓库有 commit 提交时，构建测试会自动执行，并且可在我们的网站中查看结果。 +## Getting started -- 注册自己的仓库 +- **自动化测试** - 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 - ```bash - git clone https://gitee.com/wu_fengguang/upstream-repos.git - less upstream-repos/README.md - ``` +1. 添加待测试仓库 URL 到 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos.git) 仓库，[编写测试用例并添加到仓库](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/…, 详细流程请查看[这篇文档](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/man… -- `git push` - - 更新仓库，自动触发测试。 +2. git push 更新仓库，自动触发测试。 -- 在网页中搜索并查看结果 - - web: https://compass-ci.openeuler.org/jobs +3. 在网页中搜索并查看结果 web: https://compass-ci.openeuler.org/jobs -## Source +- **手动提交测试任务** -我们最新的，最完整的源码都存放在 [Gitee](https://gitee.com/wu_fengguang/compass-ci.git) 上，Fork 我们吧！ +1. [安装Compass-CI客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… +2. [编写测试用例](https/blob/master/doc/add-testcase.md), [手动提交测试任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… +3. 在网页中搜索并查看结果 web: https://compass-ci.openeuler.org/jobs ## Contributing to Compass-CI -我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导 +我们非常欢迎有新的贡献者，我们也很乐意为我们的贡献者提供一些指导，Compass-CI 主要是使用 Ruby 和 Crystal 开发的一个项目，我们使用 rubocop和 ameba 来规范我d的代码风格，具体说明可以查看 [rococop源码仓](https://github.com/rubocop-hq/rubocop)和[ameba源码仓](https://githu… ## Website -- 2.23.0

2 1