- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/fluentd: use workers to improve concurrency
by Wu Zhende 20 Oct '20

20 Oct '20

[Why] Currently,only one worker is used to process all logs. As a result,logs can't be processed when there are a large number of concurrent logs. So use 10 workers. But tail plugin don't support multi-process. Use worker 0 for tail plugin. Use worker 1-9 for forward plugin. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/fluentd/docker-fluentd.conf | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/container/fluentd/docker-fluentd.conf b/container/fluentd/docker-fluentd.conf index ac73dfb..23c7585 100644 --- a/container/fluentd/docker-fluentd.conf +++ b/container/fluentd/docker-fluentd.conf @@ -1,8 +1,8 @@ -<source> - @type forward - bind 0.0.0.0 -</source> +<system> + workers 10 +</system> +<worker 0> <source> @type tail path /srv/cci/serial/logs/* @@ -37,6 +37,21 @@ </format> </store> </match> +</worker> + +<worker 1-9> +<source> + @type forward + bind 0.0.0.0 +</source> + +<filter **> + @type record_transformer + enable_ruby + <record> + time ${time.strftime('%Y-%m-%d %H:%M:%S%z')} + </record> +</filter> <filter **> @type parser @@ -74,6 +89,7 @@ </server> </store> </match> +</worker> <label @FLUENT_LOG> <match fluent.*> -- 2.23.0

1 0

[PATCH compass-ci] container/monitoring: fix Missing ENV key: "LKP_SRC"
by Cao Xueliang 20 Oct '20

20 Oct '20

[why] The monitor container require job.cr, and job.cr require jobfile_operate.cr which need to get the "LKP_SRC" from ENV. So, need add env variable "LKP_SRC" and mount lkp src to the monitor container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/monitoring/start | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/monitoring/start b/container/monitoring/start index 199f091..2e2946c 100755 --- a/container/monitoring/start +++ b/container/monitoring/start @@ -16,6 +16,7 @@ defaults = relevant_defaults(names) env = docker_env(defaults) DEFAULT_CCI = '/c/cci' +DEFAULT_LKP = '/c/lkp-tests' MONITOR_PORT = defaults['MONITOR_PORT'] || '11310' docker_rm "monitoring-#{MONITOR_PORT}" @@ -28,7 +29,9 @@ cmd = %W[ -p #{MONITOR_PORT}:#{MONITOR_PORT} ] + env + %W[ -e CCI_SRC=#{DEFAULT_CCI} + -e LKP_SRC=#{DEFAULT_LKP} -v /etc/localtime:/etc/localtime:ro + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v /srv/result:/srv/result monitoring-#{MONITOR_PORT} ] -- 2.23.0

1 0

[PATCH v4 compass-ci] service/scheduler: add sshr port to job content
by Wu Zhende 20 Oct '20

20 Oct '20

[Why] Network users need to use sshr to login testbox. So testbox should set sshr. And testbox get sshr info from job content. Scheduler needs to add sshr info to job content Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/start | 6 ++++++ src/lib/job.cr | 9 +++++++++ 2 files changed, 15 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 81e7139..305916e 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -24,10 +24,16 @@ names = Set.new %w[ GIT_SERVER FLUENTD_SERVER_HOST FLUENTD_SERVER_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN lab ] defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT_LEN'] ||= 2000 env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/lib/job.cr b/src/lib/job.cr index 15dc03e..12d3980 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -139,6 +139,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() + set_sshr_port() end private def append_init_field @@ -168,6 +169,14 @@ class Job end end + private def set_sshr_port + return unless self["sshd"]? + + self["sshr_port"] = ENV["SSHR_PORT"] + self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] + self["sshr_port_len"] = ENV["SSHR_PORT_LEN"] + end + private def set_os_dir self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end -- 2.23.0

1 0

[PATCH v3 compass-ci] auto_submit: add upstream_tag if the commit is tagged
by Li Yuanchao 20 Oct '20

20 Oct '20

sometimes the testcase needs a specific version software, add upstream_tag can help find the specific version. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index 1f55701..21958d3 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -34,6 +34,8 @@ class AutoSubmit commit_date = `git -C /srv/git/#{newrefs_info['git_repo']}.git log --format=%ct -1 #{commit_id}` real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") real_argvs.push("upstream_commit=#{commit_id}") + tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) + real_argvs.push("upstream_tag=#{tag}") unless tag.empty? real_argvs.push("commit_date=#{commit_date}") system(real_argvs.join(' ')) -- 2.23.0

1 0

[PATCH v3 compass-ci] service/scheduler: add sshr port to job content
by Wu Zhende 20 Oct '20

20 Oct '20

[Why] Network users need to use sshr to login testbox. So testbox should set sshr. And testbox get sshr info from job content. Scheduler needs to add sshr info to job content Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/start | 6 ++++++ src/lib/job.cr | 9 +++++++++ 2 files changed, 15 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 81e7139..7587e6c 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -24,10 +24,16 @@ names = Set.new %w[ GIT_SERVER FLUENTD_SERVER_HOST FLUENTD_SERVER_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN lab ] defaults = relevant_defaults(names) +defaults["SSHR_PORT"] = 5050 unless defaults["SSHR_PORT"] +defaults["SSHR_PORT_BASE"] = 50000 unless defaults["SSHR_PORT_BASE"] +defaults["SSHR_PORT_LEN"]= 2000 unless defaults["SSHR_PORT_LEN"] env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..6d52a21 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -139,6 +139,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() + set_sshr_port() end private def append_init_field @@ -168,6 +169,14 @@ class Job end end + private def set_sshr_port + return unless self["sshd"]? + + self["sshr_port"] = ENV["SSHR_PORT"] + self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] + self["sshr_port_len"] = ENV["SSHR_PORT_LEN"] + end + private def set_os_dir self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end -- 2.23.0

2 1

[PATCH v2 compass-ci] auto_submit: add upstream_tag if the commit is tagged
by Li Yuanchao 20 Oct '20

20 Oct '20

sometimes the testcase needs a specific version software, add upstream_tag can help find the specific version. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index 1f55701..d96859b 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -34,6 +34,8 @@ class AutoSubmit commit_date = `git -C /srv/git/#{newrefs_info['git_repo']}.git log --format=%ct -1 #{commit_id}` real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") real_argvs.push("upstream_commit=#{commit_id}") + tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id} | cat) + real_argvs.push("upstream_tag=#{tag}") real_argvs.push("commit_date=#{commit_date}") system(real_argvs.join(' ')) -- 2.23.0

2 4

[PATCH v3 compass-ci 4/5] container/send-internet-mail: build
by Luan Shengde 20 Oct '20

20 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/send-internet-mail/build diff --git a/container/send-internet-mail/build b/container/send-internet-mail/build new file mode 100755 index 0000000..078adaf --- /dev/null +++ b/container/send-internet-mail/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t send-internet-mail:latest . -- 2.23.0

1 0

[PATCH v3 compass-ci 3/5] container/send-internet-mail: Dockerfile
by Luan Shengde 20 Oct '20

20 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/Dockerfile | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 container/send-internet-mail/Dockerfile diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile new file mode 100644 index 0000000..dfa0d46 --- /dev/null +++ b/container/send-internet-mail/Dockerfile @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian +MAINTAINER luanshd0525(a)163.com +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y ruby-mail ruby-json ruby-sinatra -- 2.23.0

1 0

[PATCH v3 compass-ci 2/5] container/send-internet-mail: send-internet-mail.rb
by Luan Shengde 20 Oct '20

20 Oct '20

set up smtp config for smtp.qq.com send mail with recieved email data Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- .../send-internet-mail/send-internet-mail.rb | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100755 container/send-internet-mail/send-internet-mail.rb diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb new file mode 100755 index 0000000..d8c8d64 --- /dev/null +++ b/container/send-internet-mail/send-internet-mail.rb @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'mail' + +# setup smtp config +smtp = { + address: 'smtp.qq.com', + port: 25, + domain: 'qq.com', + user_name: ENV['EMAIL_ADDRESS'], + password: ENV['EMAIL_PASSWORD'], + openssl_verify_mode: 'none', + enable_starttls_auto: true +} + +Mail.defaults { delivery_method :smtp, smtp } + +# send mail +def send_mail(mail_info) + mail = Mail.new do + references mail_info['references'] + from ENV['EMAIL_ADDRESS'] + subject mail_info['subject'] + to mail_info['to'] + body mail_info['body'] + end + mail.deliver! +end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: merge overrides in advance
by Wei Jihui 20 Oct '20

20 Oct '20

[why] we have a new job yaml like: override: nr_task: - 100% - 50% << : jobs/unixbench.yaml overrides args:[100%, 50%] shoule be add job at the first time, so it will split to atomic job, nr_task:[nr_cpu*100%, nr_cpu*50%] job has overrides, so del revise_hash(job, deepcopy(@overrides), true), otherwise it will override job with nr_task:[100%, 50%], expand_params can not deal this list directly. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 2fd4567e..bca0a100 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -249,6 +249,7 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end + hash.merge!(@overrides) @jobs << hash end rescue StandardError => e @@ -404,7 +405,6 @@ class Job i = include_files job = deepcopy(@job) revise_hash(job, deepcopy(@job2), true) - revise_hash(job, deepcopy(@overrides), true) job['___'] = nil @jobx = job if @job.include?(:expand_params) -- 2.23.0

2 1