- Compass-ci - mailweb.openeuler.org

[PATCH v7 compass-ci 08/12] send-internet-mail: optimize
by Luan Shengde 14 Jan '21

14 Jan '21

add init redis host/port Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 3 --- 1 file changed, 3 deletions(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 7b357f8..6003028 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -9,9 +9,6 @@ require 'sinatra' REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp REDIS_PORT = ENV['REDIS_PORT'] -require_relative 'email_init' -require_relative 'email_limit_queue' -require_relative 'email_mapping' require_relative '../../lib/mail-post' smtp = { -- 2.23.0

1 1

[PATCH v7 compass-ci 07/12] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 14 Jan '21

14 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 1

[PATCH v7 compass-ci 06/12] lib/mail-post: add email counts/mapping checks
by Luan Shengde 14 Jan '21

14 Jan '21

add check email count: only email addresses that email count within the limited value can be sent emails add check email mapping: the address will be converted to the mapping value if it has been set an email mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..803a134 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +97,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 1

[PATCH v7 compass-ci 05/12] send-internet-mail: email rate limit
by Luan Shengde 14 Jan '21

14 Jan '21

add email_in_limit to store email address that send count less than limit value. add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..c7e3954 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit. +# email_in_limit is used to store email address email count +# less than the limit value. +# email_out_limit is used to store email address email count +# beyond the limit. +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1. +# case the email in email_in_limit but email count less than the +# limit value, its value will +1. +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit. +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar. +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 1

[PATCH v7 compass-ci 04/12] send-internet-mail: add email mapping
by Luan Shengde 14 Jan '21

14 Jan '21

example: name => email_addr tag => email_addr email_addr_r => email_addr_d an email address can be set multi mappings, you can set a name or email to a specified email address. when you write either a name or an email address in the send address bar, the email address will be converted to the mapped email address. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..02bfc6b --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,50 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# email address translation according to the email_mapping queue. +# email_mapping is used to set email mapping, example: +# email_mapping: +# name => email_address_d +# tag => email_address_d +# email_address_r => email_address_d +# ... +# +# can set multi keys for an email address, the key can be a name, +# an email, or some something else like a tag. +# case add the key is added to email address bar, the key will be +# transferred to the mapped email address. +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 1

[PATCH v7 compass-ci 03/12] send-internet-mail: add timing work
by Luan Shengde 14 Jan '21

14 Jan '21

daily timing work: reset email rate limit queues everyday at 00:00 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..d3f8323 --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday. +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user. +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 1

[PATCH compass-ci 3/3] container/auto-submit: delete unnecessary mount volume
by Liu Yinsi 14 Jan '21

14 Jan '21

[why] $CCI_SRC/sbin/auto-submit will call build_my_info_client, and auto output ~/.config/compass-ci/default/account.yaml in container, so mount /etc/compass-ci/account is unnecessary. delete code in sparrow/3-code/dev-env, reason is same as above. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 1 - sparrow/3-code/dev-env | 7 ------- 2 files changed, 8 deletions(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..4632ab0 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,6 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index ca91da5..4bafa80 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -52,13 +52,6 @@ ES_HOST: $ES_HOST ES_PORT: $ES_PORT EOF -mkdir -p /etc/compass-ci/account -cat > /etc/compass-ci/account/autosubmit.yaml <<EOF -my_name: Auto Submit Robot -my_email: autosubmit@localhost -my_token: $(uuidgen) -EOF - cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci -- 2.23.0

1 0

[PATCH compass-ci 2/3] container/auto-submit: add package
by Liu Yinsi 14 Jan '21

14 Jan '21

[why] util-linux, io-console, elasticsearch will be used in $CCI_SRC/sbin/auto-submit Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/auto-submit/Dockerfile b/container/auto-submit/Dockerfile index 52cd996..5804642 100644 --- a/container/auto-submit/Dockerfile +++ b/container/auto-submit/Dockerfile @@ -9,9 +9,9 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN apk update && \ apk upgrade && \ apk add --no-cache git && \ - apk add ruby-dev make gcc g++ + apk add ruby-dev make gcc g++ util-linux RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc RUN umask 002 && \ - gem install bunny json activesupport git rest-client + gem install bunny json activesupport git rest-client io-console elasticsearch -- 2.23.0

1 0

[PATCH v7 compass-ci 02/12] send-internet-mail: optimize
by Luan Shengde 14 Jan '21

14 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 1

[PATCH v7 compass-ci 01/12] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 14 Jan '21

14 Jan '21

[why] redis: used to read/store email counts from/to redis queue rufus-scheduler: used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 1