- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sched: result root add "." and replace "/" to "-"
by Cao Xueliang 02 Dec '20

02 Dec '20

Before: archlinuxaur-zzookeeper-HEAD After: archlinux-aur-z-zookeeper-HEAD Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/pp_params.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr index 28ec6ef..717f25b 100644 --- a/src/scheduler/pp_params.cr +++ b/src/scheduler/pp_params.cr @@ -51,8 +51,8 @@ class Job private def format_string(original_str) temp = [] of String original_str.each_char do |char| - if "#{char}" =~ /-|\w/ - temp << "#{char}" + if "#{char}" =~ /-|\.|\/|\w/ + temp << "#{char}".gusb("\/", "-") end end -- 2.23.0

2 2

[PATCH compass-ci] lib/job.cr: set "pub_key" field as level-1 parameter
by Zhang Yale 02 Dec '20

02 Dec '20

[why] The "pub_key" field in yaml will not be set under sshd, It should be set as level-1 parameter for password-free login, and "sshd" can be placed below the "on_fail" field or elsewhere. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4402ab..44a6ecf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -184,7 +184,7 @@ class Job end private def set_sshr_info - return unless self["sshd"]? + return unless @hash["pub_key"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] @@ -196,8 +196,7 @@ class Job end private def set_my_ssh_pubkey - sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() - pub_key = sshd["pub_key"]?.to_s + pub_key = @hash["pub_key"]?.to_s update_account_my_pub_key(pub_key) @hash["my_ssh_pubkey"] = @account_info["my_ssh_pubkey"] -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] container/logging-es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 6 ++++-- container/logging-es/build | 19 +++++++++++-------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 6c01c43..a0acf94 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -4,6 +4,8 @@ ARG BASE_IMAGE FROM $BASE_IMAGE +ARG MEMORY + # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 MAINTAINER Wu Zhende <wuzhende666(a)163.com> @@ -14,8 +16,8 @@ RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elastic sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ - sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ - sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options + sed -i "s/-Xms1g/-Xms${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/build b/container/logging-es/build index 451041b..6465ace 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,14 +1,17 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -declare -A BASE_IMAGE_DICT +require_relative '../defconfig.rb' -BASE_IMAGE_DICT=( - ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" - ["x86_64"]="elasticsearch:7.6.2" -) +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', + 'x86_64' => 'elasticsearch:7.6.2' +}.freeze -BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . +available_memory = get_available_memory + +system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container/es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/Dockerfile | 6 ++++-- container/es/build | 8 ++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index e626dae..ed02490 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -3,6 +3,8 @@ FROM alpine:3.11 +ARG MEMORY + RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add --no-cache elasticsearch curl @@ -27,8 +29,8 @@ RUN rm -rf /etc/init.d/elasticsearch \ RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms20g/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx20g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/es/config/jvm.options WORKDIR /usr/share/es diff --git a/container/es/build b/container/es/build index d5607e0..17a225c 100755 --- a/container/es/build +++ b/container/es/build @@ -1,6 +1,10 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t es643b:alpine311 . --network=host +require_relative '../defconfig.rb' +available_memory = get_available_memory + +system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] container/defconfig.rb: limit container memory usage
by Liu Yinsi 02 Dec '20

02 Dec '20

use Math.sqrt(memory) to get container occupied memory according to system memory size, instead of setting fixed size, more reasonable. In addition, '-Xms' not support floating number, so use to_i. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..dd52fe3 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + +def get_available_memory + memory = File.readlines('/proc/meminfo')[0].chomp.split[1].to_f / 1048576 + + # set container available memory size, minimum size is 1024m, maximum size is 30720m, + # take the middle value according to the system memory size. + [1024, 30720, Math.sqrt(memory) * 1024].sort[1].to_i +end -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] container/logging-es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 6 ++++-- container/logging-es/build | 19 +++++++++++-------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 6c01c43..a0acf94 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -4,6 +4,8 @@ ARG BASE_IMAGE FROM $BASE_IMAGE +ARG MEMORY + # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 MAINTAINER Wu Zhende <wuzhende666(a)163.com> @@ -14,8 +16,8 @@ RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elastic sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ - sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ - sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options + sed -i "s/-Xms1g/-Xms${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/build b/container/logging-es/build index 451041b..6465ace 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,14 +1,17 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -declare -A BASE_IMAGE_DICT +require_relative '../defconfig.rb' -BASE_IMAGE_DICT=( - ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" - ["x86_64"]="elasticsearch:7.6.2" -) +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', + 'x86_64' => 'elasticsearch:7.6.2' +}.freeze -BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . +available_memory = get_available_memory + +system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] container/es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/Dockerfile | 6 ++++-- container/es/build | 8 ++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index e626dae..ed02490 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -3,6 +3,8 @@ FROM alpine:3.11 +ARG MEMORY + RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add --no-cache elasticsearch curl @@ -27,8 +29,8 @@ RUN rm -rf /etc/init.d/elasticsearch \ RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms20g/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx20g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/es/config/jvm.options WORKDIR /usr/share/es diff --git a/container/es/build b/container/es/build index d5607e0..17a225c 100755 --- a/container/es/build +++ b/container/es/build @@ -1,6 +1,10 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t es643b:alpine311 . --network=host +require_relative '../defconfig.rb' +available_memory = get_available_memory + +system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] container/defconfig.rb: limit container memory usage
by Liu Yinsi 02 Dec '20

02 Dec '20

use Math.sqrt(memory) to get container occupied memory according to system memory size, instead of setting fixed size, more reasonable. In addition, '-Xms' not support floating number, so use to_i. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..e3ca89c 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + +def get_available_memory + memory = File.readlines('/proc/meminfo')[0].chomp.split[1].to_f/1048576 + + # set container available memory size, minimum size is 1024m, maximum size is 30720m, + # take the middle value according to the system memory size. + [1024, 30720, Math.sqrt(memory)*1024].sort[1].to_i +end -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: set "pub_key" field as level-1 parameter for password-free login
by Zhang Yale 02 Dec '20

02 Dec '20

[why] The "pub_key" field in yaml is used as the field to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4402ab..44a6ecf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -184,7 +184,7 @@ class Job end private def set_sshr_info - return unless self["sshd"]? + return unless @hash["pub_key"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] @@ -196,8 +196,7 @@ class Job end private def set_my_ssh_pubkey - sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() - pub_key = sshd["pub_key"]?.to_s + pub_key = @hash["pub_key"]?.to_s update_account_my_pub_key(pub_key) @hash["my_ssh_pubkey"] = @account_info["my_ssh_pubkey"] -- 2.23.0

2 2

[PATCH compass-ci] define_files.md: text format adapt to markdown
by Xu Xijian 02 Dec '20

02 Dec '20

In markdown documents, text for bash should be contained in block as belows: ```bash content ``` Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/define_files.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/job/define_files.md b/doc/job/define_files.md index 2c3457d..601d498 100644 --- a/doc/job/define_files.md +++ b/doc/job/define_files.md @@ -1,6 +1,7 @@ # define_files Meaning: +```bash - We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: distro/depends/$testcase distro/depends/$testcase-dev @@ -10,7 +11,10 @@ Meaning: jobs/$testcase.yaml - When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. - The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. +``` Usage example: +```bash - % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 - % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml +``` -- 2.23.0

2 3