- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci 1/3] container/defconfig.rb: limit container memory usage
by Liu Yinsi 02 Dec '20

02 Dec '20

use Math.sqrt(memory) to get container occupied memory according to system memory size, instead of setting fixed size, more reasonable. In addition, '-Xms' not support floating number, so use to_i. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..dd52fe3 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + +def get_available_memory + memory = File.readlines('/proc/meminfo')[0].chomp.split[1].to_f / 1048576 + + # set container available memory size, minimum size is 1024m, maximum size is 30720m, + # take the middle value according to the system memory size. + [1024, 30720, Math.sqrt(memory) * 1024].sort[1].to_i +end -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] container/logging-es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/logging-es/Dockerfile | 6 ++++-- container/logging-es/build | 19 +++++++++++-------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/container/logging-es/Dockerfile b/container/logging-es/Dockerfile index 6c01c43..a0acf94 100644 --- a/container/logging-es/Dockerfile +++ b/container/logging-es/Dockerfile @@ -4,6 +4,8 @@ ARG BASE_IMAGE FROM $BASE_IMAGE +ARG MEMORY + # docker image borrowed from hub.docker.com/r/gagara/elasticsearch-oss-arm64 MAINTAINER Wu Zhende <wuzhende666(a)163.com> @@ -14,8 +16,8 @@ RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elastic sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a indices.memory.index_buffer_size: 20%' /usr/share/elasticsearch/config/elasticsearch.yml && \ sed -i '$a thread_pool.write.queue_size: 2000' /usr/share/elasticsearch/config/elasticsearch.yml && \ - sed -i "s/-Xms1g/-Xms30g/g" /usr/share/elasticsearch/config/jvm.options && \ - sed -i "s/-Xmx1g/-Xmx30g/g" /usr/share/elasticsearch/config/jvm.options + sed -i "s/-Xms1g/-Xms${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx1g/-Xmx${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options RUN mkdir /usr/share/elasticsearch/tmp && \ chown -R 1090:1090 /usr/share/elasticsearch diff --git a/container/logging-es/build b/container/logging-es/build index 451041b..6465ace 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -1,14 +1,17 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -declare -A BASE_IMAGE_DICT +require_relative '../defconfig.rb' -BASE_IMAGE_DICT=( - ["aarch64"]="gagara/elasticsearch-oss-arm64:7.6.2" - ["x86_64"]="elasticsearch:7.6.2" -) +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', + 'x86_64' => 'elasticsearch:7.6.2' +}.freeze -BASE_IMAGE=${BASE_IMAGE_DICT[$(arch)]} +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE="$BASE_IMAGE" . +available_memory = get_available_memory + +system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] container/es: set available memory size
by Liu Yinsi 02 Dec '20

02 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/Dockerfile | 6 ++++-- container/es/build | 8 ++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index e626dae..ed02490 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -3,6 +3,8 @@ FROM alpine:3.11 +ARG MEMORY + RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add --no-cache elasticsearch curl @@ -27,8 +29,8 @@ RUN rm -rf /etc/init.d/elasticsearch \ RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms20g/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx20g/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/es/config/jvm.options +RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/es/config/jvm.options WORKDIR /usr/share/es diff --git a/container/es/build b/container/es/build index d5607e0..17a225c 100755 --- a/container/es/build +++ b/container/es/build @@ -1,6 +1,10 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t es643b:alpine311 . --network=host +require_relative '../defconfig.rb' +available_memory = get_available_memory + +system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] container/defconfig.rb: limit container memory usage
by Liu Yinsi 02 Dec '20

02 Dec '20

use Math.sqrt(memory) to get container occupied memory according to system memory size, instead of setting fixed size, more reasonable. In addition, '-Xms' not support floating number, so use to_i. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..e3ca89c 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + +def get_available_memory + memory = File.readlines('/proc/meminfo')[0].chomp.split[1].to_f/1048576 + + # set container available memory size, minimum size is 1024m, maximum size is 30720m, + # take the middle value according to the system memory size. + [1024, 30720, Math.sqrt(memory)*1024].sort[1].to_i +end -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: set "pub_key" field as level-1 parameter for password-free login
by Zhang Yale 02 Dec '20

02 Dec '20

[why] The "pub_key" field in yaml is used as the field to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4402ab..44a6ecf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -184,7 +184,7 @@ class Job end private def set_sshr_info - return unless self["sshd"]? + return unless @hash["pub_key"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] @@ -196,8 +196,7 @@ class Job end private def set_my_ssh_pubkey - sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() - pub_key = sshd["pub_key"]?.to_s + pub_key = @hash["pub_key"]?.to_s update_account_my_pub_key(pub_key) @hash["my_ssh_pubkey"] = @account_info["my_ssh_pubkey"] -- 2.23.0

2 2

[PATCH compass-ci] define_files.md: text format adapt to markdown
by Xu Xijian 02 Dec '20

02 Dec '20

In markdown documents, text for bash should be contained in block as belows: ```bash content ``` Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/define_files.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/job/define_files.md b/doc/job/define_files.md index 2c3457d..601d498 100644 --- a/doc/job/define_files.md +++ b/doc/job/define_files.md @@ -1,6 +1,7 @@ # define_files Meaning: +```bash - We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: distro/depends/$testcase distro/depends/$testcase-dev @@ -10,7 +11,10 @@ Meaning: jobs/$testcase.yaml - When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. - The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. +``` Usage example: +```bash - % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 - % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml +``` -- 2.23.0

2 3

[PATCH compass-ci] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't execute stuff from it cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

2 3

[PATCH v2 compass-ci 2/3] send-internet-mail/mail-post: call to store email
by Luan Shengde 02 Dec '20

02 Dec '20

call function store_email to store email to directory sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 0b1f94a..cac7a76 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -51,5 +51,11 @@ def send_mail(mail_info) to mail_info['to'] body mail_info['body'] end + mail.deliver! + + return unless ENV['SEND_MAIL_PORT'].to_s == '49000' + return unless ENV['HOST_SERVER'] == 'z9' + + store_email(mail) end -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: PKGBUILD: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't be executed directly cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't execute stuff from it cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

1 1