- Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci 12/12] mail-robot: start file
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..eab1f12 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +defaults['CCI_SRC'] = '/c/compass-ci' +defaults['MAILDIR'] = '/srv/cci/Maildir/.compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'run-mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v5 compass-ci 11/12] mail-robot: build file
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v5 compass-ci 10/12] mail-robot: Dockerfile
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v5 compass-ci 09/12] mail-robot: documentation for service
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..2eb0dd1 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. assign-account + init my_info + - my_email + - my_name + send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

1 0

[PATCH v5 compass-ci 08/12] mail-robot: build fail message
by Luan Shengde 11 Nov '20

11 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-fail-eamil.rb | 38 ++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 lib/assign-account-fail-eamil.rb diff --git a/lib/assign-account-fail-eamil.rb b/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..1c2133c --- /dev/null +++ b/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following error: + + #{message} + + In order to successfully apply an account, please pay attention to the following points: + + 1. mail subject + the subject should exactly: apply account + + 2. commit url + when you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + ensure you commit url exist and available to access. + + 3. ssh pubkey + you need to add a pubkey as an attachment with name 'id_rsa.pub' to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v5 compass-ci 06/12] mail-robot: apply account for user
by Luan Shengde 11 Nov '20

11 Nov '20

ApplyJumperAccount apply jumper account for user required data: my_info: - my_email - my_name - my_uuid my_ssh_pubkey usage: reference assign-account.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/apply-jumper-account.rb | 40 +++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 lib/apply-jumper-account.rb diff --git a/lib/apply-jumper-account.rb b/lib/apply-jumper-account.rb new file mode 100755 index 0000000..9d07abe --- /dev/null +++ b/lib/apply-jumper-account.rb @@ -0,0 +1,40 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +# apply jumper account for user +# apply_jumper_account +# apply jumper account with my_info +# check account exists +class ApplyJumperAccount + def initialize(my_info, my_ssh_pubkey) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + @my_ssh_pubkey = my_ssh_pubkey + end + + def apply_jumper_account + @my_info['my_ssh_pubkey'] = @my_ssh_pubkey + + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = 'No more available jumper account.' + error_message += 'You may try again later or consulting the manager for a solution.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 03/12] mail-robot: check to assign account
by Luan Shengde 11 Nov '20

11 Nov '20

AssignAccount assign account for user required data: mail_content - send_account parse email_content parse commit url parse pubkey apply_my_account generate uuid apply account store account info to es send email send success email send fail email usage: reference mail-robot.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account.rb | 78 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 lib/assign-account.rb diff --git a/lib/assign-account.rb b/lib/assign-account.rb new file mode 100755 index 0000000..6efeee0 --- /dev/null +++ b/lib/assign-account.rb @@ -0,0 +1,78 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'es_client' + +# assign uuid/account for user +# send_account +# parse my_commit_url and my_ssh_pubkey +# apply account +# store account info +# send mail +class AssignAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def send_account + my_commit_url, my_ssh_pubkey = parse_commit_url_pub_key + acct_info = apply_my_account(my_ssh_pubkey) + + @my_info['my_commit_url'] = my_commit_url + @my_info['my_login_name'] = acct_info['my_login_name'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + my_commit_url = parse_apply_account_email.parse_commit_url + my_ssh_pubkey = parse_apply_account_email.parse_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def apply_my_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info, my_ssh_pubkey) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(error_message) + if error_message.empty? + email_message = build_apply_account_email(@my_info) + else + email_message = build_apply_account_fail_email(@my_info, error_message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 02/12] mail-robot: listen and handle new email
by Luan Shengde 11 Nov '20

11 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..0ae80f3 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.send_account +end -- 2.23.0

1 0

[PATCH v4 compass-ci 08/12] mail-robot: build fail message
by Luan Shengde 11 Nov '20

11 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-fail-eamil.rb | 38 ++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 lib/assign-account-fail-eamil.rb diff --git a/lib/assign-account-fail-eamil.rb b/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..d15cd69 --- /dev/null +++ b/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following errors + + #{message} + + In order to successfully apply a account, pay attention to the following points: + + 1. mail subject + the subject should exactly: apply account + + 2. commit url + when you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + ensure you commit url exist and available to access. + + 3. ssh pubkey + you need to add a pubkey as an attachment with name 'id_rsa.pub' to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

2 2

[PATCH v4 compass-ci 05/12] mail-robot: check commit url for gitee
by Luan Shengde 11 Nov '20

11 Nov '20

GiteeCommitUrlCheck check commit url availability for hub gitee - clone the repo - check commit log for the commit usage: reference parse-apply-account-email.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/gitee-commit-url-check.rb | 44 +++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100755 lib/gitee-commit-url-check.rb diff --git a/lib/gitee-commit-url-check.rb b/lib/gitee-commit-url-check.rb new file mode 100755 index 0000000..11fbb6d --- /dev/null +++ b/lib/gitee-commit-url-check.rb @@ -0,0 +1,44 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check commit url availability for hub gitee.com +# gitee_commit_index +# clone the repo +# check commit available +class GiteeCommitUrlCheck + def initialize(my_info, url, base_url) + @my_info = my_info + @url = url + @base_url = base_url + end + + def gitee_commit_index + repo_dir = @url.split('/')[-3] + repo_url = [@base_url, 'git'].join('.') + commit_id = @url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] + + FileUtils.rm_rf repo_dir + + gitee_commit_exist(email_index) + end + + def gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = "We can not conform the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end +end -- 2.23.0

2 2