October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 1/5] container/send-internet-mail: mail-post.rb
by Luan Shengde 20 Oct '20

20 Oct '20

get request for send internet mail analysis email data invoke send-internet-mail to send the mail Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- lib/mail-post.rb | 50 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb new file mode 100755 index 0000000..ea8bf56 --- /dev/null +++ b/lib/mail-post.rb @@ -0,0 +1,50 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'yaml' +require 'open3' +require_relative "#{ENV['CCI_SRC']}/container/send-internet-mail/send-internet-mail.rb" + +set :bind, '0.0.0.0' +set :port, 11312 + +post '/send_mail_yaml' do + data = YAML.safe_load request.body.read + raise TypeError, data, 'request data type error' unless data.class.eql? Hash + + mail_info = { + 'references' => data['references'], + 'from' => data['from'], + 'subject' => data['subject'] || nil, + 'to' => data['to'] || nil, + 'body' => data['body'] || nil + } + check_send_mail(mail_info) +end + +post '/send_mail_text' do + data = Mail.read_from_string(request.body.read) + + mail_info = { + 'references' => data.references, + 'from' => data.from, + 'subject' => data.subject || nil, + 'to' => data.to || nil, + 'body' => data.body.decoded || nil + } + check_send_mail(mail_info) +end + +def check_send_mail(mail_info) + raise 'empty subject.' if mail_info['subject'].nil? + raise 'empty email address.' if mail_info['to'].nil? + raise 'empty email content.' if mail_info['body'].nil? + + puts ENV['EMAIL_ADDRESS'] + puts ENV['EMAIL_PASSWORD'] + send_mail(mail_info) +end -- 2.23.0

3 6

[PATCH v3 compass-ci 5/5] container/send-internet-mail: start file
by Luan Shengde 20 Oct '20

20 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/start | 72 ++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100755 container/send-internet-mail/start diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start new file mode 100755 index 0000000..e4d641d --- /dev/null +++ b/container/send-internet-mail/start @@ -0,0 +1,72 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'send-internet-mail' + +names = Set.new %w[ + EMAIL_PASSWORD + EMAIL_ADDRESS +] + +defaults = relevant_defaults(names) + +EMAIL_ADDRESS = defaults['EMAIL_ADDRESS'] +EMAIL_PASSWORD = defaults['EMAIL_PASSWORD'] + +cmd = %W[ + docker run + --restart=always + --name=send-internet-mail + -d + -e EMAIL_ADDRESS=#{EMAIL_ADDRESS} + -e EMAIL_PASSWORD=#{EMAIL_PASSWORD} + -e CCI_SRC=#{ENV['CCI_SRC']} + -p 11312:11312 + -v #{ENV['CCI_SRC']}:#{ENV['CCI_SRC']} + -v /etc/localtime:/etc/localtime:ro + --log-driver json-file + send-internet-mail +] + +cmd += ['bash', '-c', "#{ENV['CCI_SRC']}/lib/mail-post.rb"] + +system(*cmd) + + +puts <<EOF +port: +--- +data='{ +"subject": "email subject", +"to": "email_to_addr", +"body": "email message" +}' + +or + +data=" +subject: email subject +to: email_to_addr +body: email message" + +Usage: + curl -XPOST 'send-mail-server:11312/send_mail_yaml' -d "$data" + +--- +data=" +To: email_to_addr +Subject: email_subject + +mail_msg_line1 +mail_msg_line2 +... +" + +Usage: + curl -XPOST 'send-mail-server:11312/send_mail_text' -d "$data" +EOF -- 2.23.0

2 1

[PATCH compass-ci] git_mirror: feedback_info[:git_repo] => git_repo
by Li Yuanchao 20 Oct '20

20 Oct '20

simplify the code Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index fef4aca..e80b7ca 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -151,12 +151,13 @@ class MirrorMain return if @feedback_queue.empty? feedback_info = @feedback_queue.pop(true) - @fork_stat[feedback_info[:git_repo]][:queued] = false + git_repo = feedback_info[:git_repo] + @fork_stat[git_repo][:queued] = false return unless feedback_info[:possible_new_refs] - return reload_fork_info if feedback_info[:git_repo] == 'upstream-repos/upstream-repos' + return reload_fork_info if git_repo == 'upstream-repos/upstream-repos' - new_refs = check_new_refs(feedback_info[:git_repo]) + new_refs = check_new_refs(git_repo) return if new_refs[:heads].empty? feedback_info[:new_refs] = new_refs -- 2.23.0

1 0

[PATCH compass-ci] container/fluentd: use workers to improve concurrency
by Wu Zhende 20 Oct '20

20 Oct '20

[Why] Currently,only one worker is used to process all logs. As a result,logs can't be processed when there are a large number of concurrent logs. So use 10 workers. But tail plugin don't support multi-process. Use worker 0 for tail plugin. Use worker 1-9 for forward plugin. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/fluentd/docker-fluentd.conf | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/container/fluentd/docker-fluentd.conf b/container/fluentd/docker-fluentd.conf index ac73dfb..23c7585 100644 --- a/container/fluentd/docker-fluentd.conf +++ b/container/fluentd/docker-fluentd.conf @@ -1,8 +1,8 @@ -<source> - @type forward - bind 0.0.0.0 -</source> +<system> + workers 10 +</system> +<worker 0> <source> @type tail path /srv/cci/serial/logs/* @@ -37,6 +37,21 @@ </format> </store> </match> +</worker> + +<worker 1-9> +<source> + @type forward + bind 0.0.0.0 +</source> + +<filter **> + @type record_transformer + enable_ruby + <record> + time ${time.strftime('%Y-%m-%d %H:%M:%S%z')} + </record> +</filter> <filter **> @type parser @@ -74,6 +89,7 @@ </server> </store> </match> +</worker> <label @FLUENT_LOG> <match fluent.*> -- 2.23.0

1 0

[PATCH compass-ci] container/monitoring: fix Missing ENV key: "LKP_SRC"
by Cao Xueliang 20 Oct '20

20 Oct '20

[why] The monitor container require job.cr, and job.cr require jobfile_operate.cr which need to get the "LKP_SRC" from ENV. So, need add env variable "LKP_SRC" and mount lkp src to the monitor container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/monitoring/start | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/monitoring/start b/container/monitoring/start index 199f091..2e2946c 100755 --- a/container/monitoring/start +++ b/container/monitoring/start @@ -16,6 +16,7 @@ defaults = relevant_defaults(names) env = docker_env(defaults) DEFAULT_CCI = '/c/cci' +DEFAULT_LKP = '/c/lkp-tests' MONITOR_PORT = defaults['MONITOR_PORT'] || '11310' docker_rm "monitoring-#{MONITOR_PORT}" @@ -28,7 +29,9 @@ cmd = %W[ -p #{MONITOR_PORT}:#{MONITOR_PORT} ] + env + %W[ -e CCI_SRC=#{DEFAULT_CCI} + -e LKP_SRC=#{DEFAULT_LKP} -v /etc/localtime:/etc/localtime:ro + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v /srv/result:/srv/result monitoring-#{MONITOR_PORT} ] -- 2.23.0

1 0

[PATCH v4 compass-ci] service/scheduler: add sshr port to job content
by Wu Zhende 20 Oct '20

20 Oct '20

[Why] Network users need to use sshr to login testbox. So testbox should set sshr. And testbox get sshr info from job content. Scheduler needs to add sshr info to job content Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/start | 6 ++++++ src/lib/job.cr | 9 +++++++++ 2 files changed, 15 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 81e7139..305916e 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -24,10 +24,16 @@ names = Set.new %w[ GIT_SERVER FLUENTD_SERVER_HOST FLUENTD_SERVER_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN lab ] defaults = relevant_defaults(names) +defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT_LEN'] ||= 2000 env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/lib/job.cr b/src/lib/job.cr index 15dc03e..12d3980 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -139,6 +139,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() + set_sshr_port() end private def append_init_field @@ -168,6 +169,14 @@ class Job end end + private def set_sshr_port + return unless self["sshd"]? + + self["sshr_port"] = ENV["SSHR_PORT"] + self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] + self["sshr_port_len"] = ENV["SSHR_PORT_LEN"] + end + private def set_os_dir self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end -- 2.23.0

1 0

[PATCH v3 compass-ci] auto_submit: add upstream_tag if the commit is tagged
by Li Yuanchao 20 Oct '20

20 Oct '20

sometimes the testcase needs a specific version software, add upstream_tag can help find the specific version. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index 1f55701..21958d3 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -34,6 +34,8 @@ class AutoSubmit commit_date = `git -C /srv/git/#{newrefs_info['git_repo']}.git log --format=%ct -1 #{commit_id}` real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") real_argvs.push("upstream_commit=#{commit_id}") + tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) + real_argvs.push("upstream_tag=#{tag}") unless tag.empty? real_argvs.push("commit_date=#{commit_date}") system(real_argvs.join(' ')) -- 2.23.0

1 0

[PATCH v3 compass-ci] service/scheduler: add sshr port to job content
by Wu Zhende 20 Oct '20

20 Oct '20

[Why] Network users need to use sshr to login testbox. So testbox should set sshr. And testbox get sshr info from job content. Scheduler needs to add sshr info to job content Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/start | 6 ++++++ src/lib/job.cr | 9 +++++++++ 2 files changed, 15 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 81e7139..7587e6c 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -24,10 +24,16 @@ names = Set.new %w[ GIT_SERVER FLUENTD_SERVER_HOST FLUENTD_SERVER_PORT + SSHR_PORT + SSHR_PORT_BASE + SSHR_PORT_LEN lab ] defaults = relevant_defaults(names) +defaults["SSHR_PORT"] = 5050 unless defaults["SSHR_PORT"] +defaults["SSHR_PORT_BASE"] = 50000 unless defaults["SSHR_PORT_BASE"] +defaults["SSHR_PORT_LEN"]= 2000 unless defaults["SSHR_PORT_LEN"] env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..6d52a21 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -139,6 +139,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() + set_sshr_port() end private def append_init_field @@ -168,6 +169,14 @@ class Job end end + private def set_sshr_port + return unless self["sshd"]? + + self["sshr_port"] = ENV["SSHR_PORT"] + self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] + self["sshr_port_len"] = ENV["SSHR_PORT_LEN"] + end + private def set_os_dir self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end -- 2.23.0

2 1

[PATCH v2 compass-ci] auto_submit: add upstream_tag if the commit is tagged
by Li Yuanchao 20 Oct '20

20 Oct '20

sometimes the testcase needs a specific version software, add upstream_tag can help find the specific version. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index 1f55701..d96859b 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -34,6 +34,8 @@ class AutoSubmit commit_date = `git -C /srv/git/#{newrefs_info['git_repo']}.git log --format=%ct -1 #{commit_id}` real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") real_argvs.push("upstream_commit=#{commit_id}") + tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id} | cat) + real_argvs.push("upstream_tag=#{tag}") real_argvs.push("commit_date=#{commit_date}") system(real_argvs.join(' ')) -- 2.23.0

2 4

[PATCH v3 compass-ci 4/5] container/send-internet-mail: build
by Luan Shengde 20 Oct '20

20 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/send-internet-mail/build diff --git a/container/send-internet-mail/build b/container/send-internet-mail/build new file mode 100755 index 0000000..078adaf --- /dev/null +++ b/container/send-internet-mail/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t send-internet-mail:latest . -- 2.23.0

1 0