December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/job.cr: set "pub_key" field as level-1 parameter for password-free login
by Zhang Yale 02 Dec '20

02 Dec '20

[why] The "pub_key" field in yaml is used as the field to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- src/lib/job.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4402ab..44a6ecf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -184,7 +184,7 @@ class Job end private def set_sshr_info - return unless self["sshd"]? + return unless @hash["pub_key"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] @@ -196,8 +196,7 @@ class Job end private def set_my_ssh_pubkey - sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() - pub_key = sshd["pub_key"]?.to_s + pub_key = @hash["pub_key"]?.to_s update_account_my_pub_key(pub_key) @hash["my_ssh_pubkey"] = @account_info["my_ssh_pubkey"] -- 2.23.0

2 2

[PATCH compass-ci] define_files.md: text format adapt to markdown
by Xu Xijian 02 Dec '20

02 Dec '20

In markdown documents, text for bash should be contained in block as belows: ```bash content ``` Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/define_files.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/job/define_files.md b/doc/job/define_files.md index 2c3457d..601d498 100644 --- a/doc/job/define_files.md +++ b/doc/job/define_files.md @@ -1,6 +1,7 @@ # define_files Meaning: +```bash - We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: distro/depends/$testcase distro/depends/$testcase-dev @@ -10,7 +11,10 @@ Meaning: jobs/$testcase.yaml - When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. - The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. +``` Usage example: +```bash - % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 - % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml +``` -- 2.23.0

2 3

[PATCH compass-ci] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't execute stuff from it cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

2 3

[PATCH v2 compass-ci 2/3] send-internet-mail/mail-post: call to store email
by Luan Shengde 02 Dec '20

02 Dec '20

call function store_email to store email to directory sent Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 0b1f94a..cac7a76 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -51,5 +51,11 @@ def send_mail(mail_info) to mail_info['to'] body mail_info['body'] end + mail.deliver! + + return unless ENV['SEND_MAIL_PORT'].to_s == '49000' + return unless ENV['HOST_SERVER'] == 'z9' + + store_email(mail) end -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: PKGBUILD: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't be executed directly cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] providers/docker: delete the noexec flag of tmpfs
by Xiao Shenwei 02 Dec '20

02 Dec '20

error messgae: ./configure permission denied [why] tmpfs default option (rw,nosuid,noexec,nodev) can't execute stuff from it cause of the noexec flag refer-to: https://github.com/docker/compose/issues/3425 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 9edad62..d532a56 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -23,7 +23,7 @@ cmd=( docker run --rm -m $memory - --mount type=tmpfs,destination=/tmp + --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/bin:ro -- 2.23.0

1 1

[PATCH v3 compass-ci 2/2] container/netdata-slave: monitoring HW server
by Xiao Shenwei 02 Dec '20

02 Dec '20

[monitored object] file-server : master crystal.ci : slave z9 : slave ibmc-mgnt : slave backup-server: slave jumper-server: slave startup by multi-qemu or multi-docker machine [function] slave node will collect host machine metrics, send to master slave node does not provider data storage, alarm, web services. [usage] config/stream.conf destination = master-HOST[:PORT] api key = uuid refer-to: https://github.com/netdata/netdata/tree/master/streaming Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/netdata-slave/Dockerfile | 10 ++++++ container/netdata-slave/build | 5 +++ container/netdata-slave/config/netdata.conf | 5 +++ container/netdata-slave/config/stream.conf | 10 ++++++ container/netdata-slave/run | 34 +++++++++++++++++++++ 5 files changed, 64 insertions(+) create mode 100755 container/netdata-slave/Dockerfile create mode 100755 container/netdata-slave/build create mode 100644 container/netdata-slave/config/netdata.conf create mode 100644 container/netdata-slave/config/stream.conf create mode 100755 container/netdata-slave/run diff --git a/container/netdata-slave/Dockerfile b/container/netdata-slave/Dockerfile new file mode 100755 index 0000000..5c2c02f --- /dev/null +++ b/container/netdata-slave/Dockerfile @@ -0,0 +1,10 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +FROM netdata/netdata + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN usermod -u 1090 netdata && \ + groupmod -g 1090 netdata diff --git a/container/netdata-slave/build b/container/netdata-slave/build new file mode 100755 index 0000000..1a84550 --- /dev/null +++ b/container/netdata-slave/build @@ -0,0 +1,5 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +docker build -t netdata/netdata-slave . diff --git a/container/netdata-slave/config/netdata.conf b/container/netdata-slave/config/netdata.conf new file mode 100644 index 0000000..456fe02 --- /dev/null +++ b/container/netdata-slave/config/netdata.conf @@ -0,0 +1,5 @@ +[global] + memory mode = none + +[health] + enabled = no diff --git a/container/netdata-slave/config/stream.conf b/container/netdata-slave/config/stream.conf new file mode 100644 index 0000000..ac91a7d --- /dev/null +++ b/container/netdata-slave/config/stream.conf @@ -0,0 +1,10 @@ +[stream] + enabled = yes + destination = + api key = + timeout seconds = 60 + default port = 19999 + send charts matching = * + buffer size bytes = 1048576 + reconnect delay seconds = 5 + initial clock resync iterations = 60 diff --git a/container/netdata-slave/run b/container/netdata-slave/run new file mode 100755 index 0000000..e790298 --- /dev/null +++ b/container/netdata-slave/run @@ -0,0 +1,34 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +. $CCI_SRC/container/defconfig.sh + +docker_rm netdata-slave + +DIR=$(dirname $(realpath $0)) + +cmd=( + docker run -d + --restart=unless-stopped + --hostname=$HOSTNAME + --name netdata-slave + -p 19999:19999 + -v $DIR/config/netdata.conf:/etc/netdata/netdata.conf + -v $DIR/config/stream.conf:/etc/netdata/stream.conf + -v /etc/passwd:/host/etc/passwd:ro + -v /etc/group:/host/etc/group:ro + -v /proc:/host/proc:ro + -v /sys:/host/sys:ro + -v /etc/localtime:/etc/localtime:ro + -v /etc/os-release:/host//etc/os-release:ro + -v /var/run/docker.sock:/var/run/docker.sock:ro + -v /srv/cache/netdata_cache:/var/cache/netdata + -v /srv/cache/netdata_lib:/var/lib/netdata + --log-driver json-file + --log-opt max-size=1g + --cap-add SYS_PTRACE + --security-opt apparmor=unconfined + netdata/netdata-slave +) +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] sched: no need set kernel if os mount is container
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4402ab..5798ae3 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -146,16 +146,22 @@ class Job set_result_root() set_result_service() set_os_mount() + set_kernel() set_depends_initrd() - set_kernel_version() set_initrds_uri() - set_kernel_uri() - set_kernel_params() set_lkp_server() set_sshr_info() set_queue() end + private def set_kernel + return if os_mount == "container" + + set_kernel_version() + set_kernel_uri() + set_kernel_params() + end + private def append_init_field INIT_FIELD.each do |k, v| k = k.to_s -- 2.23.0

2 1

[PATCH v2 compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 01 Dec '20

01 Dec '20

Add a document for introducing the arch, nr_node, nr_cpu, memory fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..ba12368 --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,18 @@ +# arch, nr_node, nr_cpu, memory + +Meaning: +- Each test machine has some basic information, such as arch, nr_node, nr_cpu, memory + these fields mean: + arch: the architecture of the machine. + nr_node: number of NUMA nodes. + nr_cpu: number of logical CPUs. + memory: the storage space of the machine. + +- These fields are from the file which in $LAB_SRC/hosts. +- Here is an example of these fields in the host file: + arch: aarch64 + nr_node: 4 + nr_cpu: 96 + memory: 256G + ... + -- 2.23.0

2 1

[PATCH v3 compass-ci] lib: add error_messages.rb
by Lin Jiaxin 01 Dec '20

01 Dec '20

provide error messages for git bisect Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 63 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 lib/error_messages.rb diff --git a/lib/error_messages.rb b/lib/error_messages.rb new file mode 100644 index 0000000..f59c73a --- /dev/null +++ b/lib/error_messages.rb @@ -0,0 +1,63 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' + +# obtain error messages hash +class ErrorMessages + def initialize(file) + @file = file + @seqno = '' + @in_stderr = false + @error_line = '' + @error_details = '' + @error_message = {} + @error_messages = Hash.new { |h, k| h[k] = Set.new } + end + + def obtain_error_messages + File.open((a)file).each_line do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'].downcase =~ /error|warning/ + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end + return @error_messages + end + + private + + def extract_error_message(line) + if line =~ /^ / + @error_details += line + else + update_error_message(line) + case line + when /^## ______________([0-9.]+):stderr$/ + @in_stderr = true + @seqno = $1 + return true + + when /^## ______________#{@seqno}:enderr$/ + @in_stderr = false + @seqno = '' + add_error_message + end + end + end + + def update_error_message(line) + @error_message['error_line'] = @error_line + @error_message['error_details'] = @error_details + @error_line = line + @error_details = '' + end + + def add_error_message + return unless @error_message['error_line'].downcase =~ /error|warning/ + + @error_messages[@error_message['error_line']] << @error_message['error_line'] + @error_message['error_details'] + end +end -- 2.23.0

2 1