December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] container/defconfig.rb: fix rubocop warning
by Liu Yinsi 02 Dec '20

02 Dec '20

defconfig.rb:23:12: Style/BlockDelimiters: Avoid using {...} for multi-line blocks. hash.map { |k, v| Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index dd52fe3..c0768ca 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -20,9 +20,7 @@ end def set_local_env hash = cci_defaults - hash.map { |k, v| - system "export #{k}=#{v}" - } + hash.map { |k, v| system "export #{k}=#{v}" } end def docker_env(hash) -- 2.23.0

1 0

[PATCH v1 compass-ci 1/2] sched: no need set kernel if os mount is container
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 44a6ecf..7758e78 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -146,16 +146,22 @@ class Job set_result_root() set_result_service() set_os_mount() + set_kernel() set_depends_initrd() - set_kernel_version() set_initrds_uri() - set_kernel_uri() - set_kernel_params() set_lkp_server() set_sshr_info() set_queue() end + private def set_kernel + return if os_mount == "container" + + set_kernel_version() + set_kernel_uri() + set_kernel_params() + end + private def append_init_field INIT_FIELD.each do |k, v| k = k.to_s -- 2.23.0

1 0

[PATCH compass-ci] sched: result root add "." and replace "/" to "-"
by Cao Xueliang 02 Dec '20

02 Dec '20

Before: archlinuxaur-zzookeeper-HEAD After: archlinux-aur-z-zookeeper-HEAD Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/pp_params.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr index 28ec6ef..717f25b 100644 --- a/src/scheduler/pp_params.cr +++ b/src/scheduler/pp_params.cr @@ -51,8 +51,8 @@ class Job private def format_string(original_str) temp = [] of String original_str.each_char do |char| - if "#{char}" =~ /-|\w/ - temp << "#{char}" + if "#{char}" =~ /-|\.|\/|\w/ + temp << "#{char}".gusb("\/", "-") end end -- 2.23.0

2 2

[PATCH compass-ci] lib/job.cr: set "pub_key" field as level-1 parameter
by Zhang Yale 02 Dec '20

02 Dec '20

[why] The "pub_key" field in yaml will not be set under sshd, It should be set as level-1 parameter for password-free login, and "sshd" can be placed below the "on_fail" field or elsewhere. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4402ab..44a6ecf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -184,7 +184,7 @@ class Job end private def set_sshr_info - return unless self["sshd"]? + return unless @hash["pub_key"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] @@ -196,8 +196,7 @@ class Job end private def set_my_ssh_pubkey - sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() - pub_key = sshd["pub_key"]?.to_s + pub_key = @hash["pub_key"]?.to_s update_account_my_pub_key(pub_key) @hash["my_ssh_pubkey"] = @account_info["my_ssh_pubkey"] -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] container/logging-es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 6 ++++-- container/logging-es/build | 19 +++++++++++-------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 6c01c43..a0acf94 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -4,6 +4,8 @@ ARG BASE_IMAGE FROM $BASE_IMAGE +ARG MEMORY + # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 MAINTAINER Wu Zhende <wuzhende666(a)163.com> @@ -14,8 +16,8 @@ RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elastic sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ - sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ - sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options + sed -i "s/-Xms1g/-Xms${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/build b/container/logging-es/build index 451041b..6465ace 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,14 +1,17 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -declare -A BASE_IMAGE_DICT +require_relative '../defconfig.rb' -BASE_IMAGE_DICT=( - ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" - ["x86_64"]="elasticsearch:7.6.2" -) +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', + 'x86_64' => 'elasticsearch:7.6.2' +}.freeze -BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . +available_memory = get_available_memory + +system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container/es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/Dockerfile | 6 ++++-- container/es/build | 8 ++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index e626dae..ed02490 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -3,6 +3,8 @@ FROM alpine:3.11 +ARG MEMORY + RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add --no-cache elasticsearch curl @@ -27,8 +29,8 @@ RUN rm -rf /etc/init.d/elasticsearch \ RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms20g/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx20g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/es/config/jvm.options WORKDIR /usr/share/es diff --git a/container/es/build b/container/es/build index d5607e0..17a225c 100755 --- a/container/es/build +++ b/container/es/build @@ -1,6 +1,10 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t es643b:alpine311 . --network=host +require_relative '../defconfig.rb' +available_memory = get_available_memory + +system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] container/defconfig.rb: limit container memory usage
by Liu Yinsi 02 Dec '20

02 Dec '20

use Math.sqrt(memory) to get container occupied memory according to system memory size, instead of setting fixed size, more reasonable. In addition, '-Xms' not support floating number, so use to_i. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..dd52fe3 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + +def get_available_memory + memory = File.readlines('/proc/meminfo')[0].chomp.split[1].to_f / 1048576 + + # set container available memory size, minimum size is 1024m, maximum size is 30720m, + # take the middle value according to the system memory size. + [1024, 30720, Math.sqrt(memory) * 1024].sort[1].to_i +end -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] container/logging-es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 6 ++++-- container/logging-es/build | 19 +++++++++++-------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 6c01c43..a0acf94 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -4,6 +4,8 @@ ARG BASE_IMAGE FROM $BASE_IMAGE +ARG MEMORY + # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 MAINTAINER Wu Zhende <wuzhende666(a)163.com> @@ -14,8 +16,8 @@ RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elastic sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ - sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ - sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options + sed -i "s/-Xms1g/-Xms${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/build b/container/logging-es/build index 451041b..6465ace 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,14 +1,17 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -declare -A BASE_IMAGE_DICT +require_relative '../defconfig.rb' -BASE_IMAGE_DICT=( - ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" - ["x86_64"]="elasticsearch:7.6.2" -) +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', + 'x86_64' => 'elasticsearch:7.6.2' +}.freeze -BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . +available_memory = get_available_memory + +system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] container/es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/Dockerfile | 6 ++++-- container/es/build | 8 ++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index e626dae..ed02490 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -3,6 +3,8 @@ FROM alpine:3.11 +ARG MEMORY + RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add --no-cache elasticsearch curl @@ -27,8 +29,8 @@ RUN rm -rf /etc/init.d/elasticsearch \ RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms20g/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx20g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/es/config/jvm.options WORKDIR /usr/share/es diff --git a/container/es/build b/container/es/build index d5607e0..17a225c 100755 --- a/container/es/build +++ b/container/es/build @@ -1,6 +1,10 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t es643b:alpine311 . --network=host +require_relative '../defconfig.rb' +available_memory = get_available_memory + +system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] container/defconfig.rb: limit container memory usage
by Liu Yinsi 02 Dec '20

02 Dec '20

use Math.sqrt(memory) to get container occupied memory according to system memory size, instead of setting fixed size, more reasonable. In addition, '-Xms' not support floating number, so use to_i. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..e3ca89c 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + +def get_available_memory + memory = File.readlines('/proc/meminfo')[0].chomp.split[1].to_f/1048576 + + # set container available memory size, minimum size is 1024m, maximum size is 30720m, + # take the middle value according to the system memory size. + [1024, 30720, Math.sqrt(memory)*1024].sort[1].to_i +end -- 2.23.0

1 0