- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 1/3] container/defconfig.rb: limit container memory usage
by Liu Yinsi 02 Dec '20

02 Dec '20

use Math.sqrt(memory) to get container occupied memory according to system memory size, instead of setting fixed size, more reasonable. In addition, '-Xms' not support floating number, so use to_i. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..e3ca89c 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + +def get_available_memory + memory = File.readlines('/proc/meminfo')[0].chomp.split[1].to_f/1048576 + + # set container available memory size, minimum size is 1024m, maximum size is 30720m, + # take the middle value according to the system memory size. + [1024, 30720, Math.sqrt(memory)*1024].sort[1].to_i +end -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: set "pub_key" field as level-1 parameter for password-free login
by Zhang Yale 02 Dec '20

02 Dec '20

[why] The "pub_key" field in yaml is used as the field to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4402ab..44a6ecf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -184,7 +184,7 @@ class Job end private def set_sshr_info - return unless self["sshd"]? + return unless @hash["pub_key"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] @@ -196,8 +196,7 @@ class Job end private def set_my_ssh_pubkey - sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() - pub_key = sshd["pub_key"]?.to_s + pub_key = @hash["pub_key"]?.to_s update_account_my_pub_key(pub_key) @hash["my_ssh_pubkey"] = @account_info["my_ssh_pubkey"] -- 2.23.0

2 2

[PATCH compass-ci] define_files.md: text format adapt to markdown
by Xu Xijian 02 Dec '20

02 Dec '20

In markdown documents, text for bash should be contained in block as belows: ```bash content ``` Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/define_files.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/job/define_files.md b/doc/job/define_files.md index 2c3457d..601d498 100644 --- a/doc/job/define_files.md +++ b/doc/job/define_files.md @@ -1,6 +1,7 @@ # define_files Meaning: +```bash - We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: distro/depends/$testcase distro/depends/$testcase-dev @@ -10,7 +11,10 @@ Meaning: jobs/$testcase.yaml - When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. - The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. +``` Usage example: +```bash - % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 - % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml +``` -- 2.23.0

2 3

[PATCH compass-ci] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't execute stuff from it cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

2 3

[PATCH v2 compass-ci 2/3] send-internet-mail/mail-post: call to store email
by Luan Shengde 02 Dec '20

02 Dec '20

call function store_email to store email to directory sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 0b1f94a..cac7a76 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -51,5 +51,11 @@ def send_mail(mail_info) to mail_info['to'] body mail_info['body'] end + mail.deliver! + + return unless ENV['SEND_MAIL_PORT'].to_s == '49000' + return unless ENV['HOST_SERVER'] == 'z9' + + store_email(mail) end -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: PKGBUILD: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't be executed directly cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't execute stuff from it cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

1 1

[PATCH v3 compass-ci 2/2] container/netdata-slave: monitoring HW server
by Xiao Shenwei 02 Dec '20

02 Dec '20

[monitored object] file-server : master crystal.ci : slave z9 : slave ibmc-mgnt : slave backup-server: slave jumper-server: slave startup by multi-qemu or multi-docker machine [function] slave node will collect host machine metrics, send to master slave node does not provider data storage, alarm, web services. [usage] config/stream.conf destination = master-HOST[:PORT] api key = uuid refer-to: https://github.com/netdata/netdata/tree/master/streaming Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/netdata-slave/Dockerfile | 10 ++++++ container/netdata-slave/build | 5 +++ container/netdata-slave/config/netdata.conf | 5 +++ container/netdata-slave/config/stream.conf | 10 ++++++ container/netdata-slave/run | 34 +++++++++++++++++++++ 5 files changed, 64 insertions(+) create mode 100755 container/netdata-slave/Dockerfile create mode 100755 container/netdata-slave/build create mode 100644 container/netdata-slave/config/netdata.conf create mode 100644 container/netdata-slave/config/stream.conf create mode 100755 container/netdata-slave/run diff --git a/container/netdata-slave/Dockerfile b/container/netdata-slave/Dockerfile new file mode 100755 index 0000000..5c2c02f --- /dev/null +++ b/container/netdata-slave/Dockerfile @@ -0,0 +1,10 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +FROM netdata/netdata + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN usermod -u 1090 netdata && \ + groupmod -g 1090 netdata diff --git a/container/netdata-slave/build b/container/netdata-slave/build new file mode 100755 index 0000000..1a84550 --- /dev/null +++ b/container/netdata-slave/build @@ -0,0 +1,5 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +docker build -t netdata/netdata-slave . diff --git a/container/netdata-slave/config/netdata.conf b/container/netdata-slave/config/netdata.conf new file mode 100644 index 0000000..456fe02 --- /dev/null +++ b/container/netdata-slave/config/netdata.conf @@ -0,0 +1,5 @@ +[global] + memory mode = none + +[health] + enabled = no diff --git a/container/netdata-slave/config/stream.conf b/container/netdata-slave/config/stream.conf new file mode 100644 index 0000000..ac91a7d --- /dev/null +++ b/container/netdata-slave/config/stream.conf @@ -0,0 +1,10 @@ +[stream] + enabled = yes + destination = + api key = + timeout seconds = 60 + default port = 19999 + send charts matching = * + buffer size bytes = 1048576 + reconnect delay seconds = 5 + initial clock resync iterations = 60 diff --git a/container/netdata-slave/run b/container/netdata-slave/run new file mode 100755 index 0000000..e790298 --- /dev/null +++ b/container/netdata-slave/run @@ -0,0 +1,34 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +. $CCI_SRC/container/defconfig.sh + +docker_rm netdata-slave + +DIR=$(dirname $(realpath $0)) + +cmd=( + docker run -d + --restart=unless-stopped + --hostname=$HOSTNAME + --name netdata-slave + -p 19999:19999 + -v $DIR/config/netdata.conf:/etc/netdata/netdata.conf + -v $DIR/config/stream.conf:/etc/netdata/stream.conf + -v /etc/passwd:/host/etc/passwd:ro + -v /etc/group:/host/etc/group:ro + -v /proc:/host/proc:ro + -v /sys:/host/sys:ro + -v /etc/localtime:/etc/localtime:ro + -v /etc/os-release:/host//etc/os-release:ro + -v /var/run/docker.sock:/var/run/docker.sock:ro + -v /srv/cache/netdata_cache:/var/cache/netdata + -v /srv/cache/netdata_lib:/var/lib/netdata + --log-driver json-file + --log-opt max-size=1g + --cap-add SYS_PTRACE + --security-opt apparmor=unconfined + netdata/netdata-slave +) +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] sched: no need set kernel if os mount is container
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4402ab..5798ae3 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -146,16 +146,22 @@ class Job set_result_root() set_result_service() set_os_mount() + set_kernel() set_depends_initrd() - set_kernel_version() set_initrds_uri() - set_kernel_uri() - set_kernel_params() set_lkp_server() set_sshr_info() set_queue() end + private def set_kernel + return if os_mount == "container" + + set_kernel_version() + set_kernel_uri() + set_kernel_params() + end + private def append_init_field INIT_FIELD.each do |k, v| k = k.to_s -- 2.23.0

2 1

[PATCH v2 compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 01 Dec '20

01 Dec '20

Add a document for introducing the arch, nr_node, nr_cpu, memory fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..ba12368 --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,18 @@ +# arch, nr_node, nr_cpu, memory + +Meaning: +- Each test machine has some basic information, such as arch, nr_node, nr_cpu, memory + these fields mean: + arch: the architecture of the machine. + nr_node: number of NUMA nodes. + nr_cpu: number of logical CPUs. + memory: the storage space of the machine. + +- These fields are from the file which in $LAB_SRC/hosts. +- Here is an example of these fields in the host file: + arch: aarch64 + nr_node: 4 + nr_cpu: 96 + memory: 256G + ... + -- 2.23.0

2 1