- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] os-cifs/*: change base image name
by Wang Yong 13 Nov '20

13 Nov '20

update cifs base image name to maintain the consistency with os-http/os-nfs Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- container/os-cifs/build | 2 +- container/os-cifs/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/os-cifs/build b/container/os-cifs/build index 56ebbed..77be8e9 100755 --- a/container/os-cifs/build +++ b/container/os-cifs/build @@ -2,4 +2,4 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t alpine/samba . +docker build -t os-cifs . diff --git a/container/os-cifs/start b/container/os-cifs/start index 1b2cd23..f42c71c 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -19,7 +19,7 @@ cmd=( -v /srv/result:/srv/result --name os-cifs --restart=always - alpine/samba + os-cifs ) "${cmd[@]}" -- 2.23.0

1 0

[PATCH v7 compass-ci 11/11] mail-robot: documentation for service
by Luan Shengde 13 Nov '20

13 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..385ce96 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. apply-account + init my_info + - my_email + - my_name + check_to_send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

1 0

[PATCH v7 compass-ci 10/11] mail-robot: start file
by Luan Shengde 13 Nov '20

13 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..5827e18 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +defaults['MAILDIR'] = '/srv/cci/Maildir/.compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v7 compass-ci 09/11] mail-robot: build file
by Luan Shengde 13 Nov '20

13 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v7 compass-ci 08/11] mail-robot: Dockerfile
by Luan Shengde 13 Nov '20

13 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v7 compass-ci 07/11] mail-robot: build fail message
by Luan Shengde 13 Nov '20

13 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 38 +++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 container/mail-robot/lib/assign-account-fail-eamil.rb diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..769853f --- /dev/null +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following error: + + #{message} + + In order to successfully apply an account, please pay attention to the following points: + + 1. mail subject + The subject should exactly: apply account + + 2. commit url + When you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + Attention: + Ensure you commit url exist and available to access. + + 3. ssh pubkey + You need to add a pubkey as an attachment to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v7 compass-ci 05/11] mail-robot: apply new account for user
by Luan Shengde 13 Nov '20

13 Nov '20

ApplyJumperAccount used to apply account for user when successfully parsed my_commit_url and my_ssh_pubkey input: my_info, my_ssh_pubkey apply_jumper_account apply account with my_info and my_ssh_pubkey my_info: - my_email - my_name - my_uuid my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/apply-jumper-account.rb | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/mail-robot/lib/apply-jumper-account.rb diff --git a/container/mail-robot/lib/apply-jumper-account.rb b/container/mail-robot/lib/apply-jumper-account.rb new file mode 100755 index 0000000..a2c239c --- /dev/null +++ b/container/mail-robot/lib/apply-jumper-account.rb @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 + +# used to apply account +# be called after AssignAccount successfully parsed my_commit_url and my_ssh_pubkey +# apply_jumper_account +# apply jumper account with my_info and my_ssh_pubkey +# account_info_exist +# check account exists +class ApplyJumperAccount + def initialize(my_info, my_ssh_pubkey) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + @my_ssh_pubkey = my_ssh_pubkey + end + + def apply_jumper_account + @my_info['my_ssh_pubkey'] = @my_ssh_pubkey + + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = 'No more available jumper account.' + error_message += 'You may try again later or consulting the manager for a solution.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v7 compass-ci 04/11] mail-robot: check commit url for gitee
by Luan Shengde 13 Nov '20

13 Nov '20

GiteeCommitUrlCheck used to check commit for hub gitee.com when execute commit_url_availability in ParseApplyAccountEmail input: my_commit_url gitee_commit_check clone the repo and check commit exists and availalbe Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 container/mail-robot/lib/gitee-commit-url-check.rb diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb new file mode 100755 index 0000000..23b91f1 --- /dev/null +++ b/container/mail-robot/lib/gitee-commit-url-check.rb @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# used to check commit available for hub gitee.com +# it is called when: +# - ParseApplyAccountEmail execute commit_url_availability +# - hub is gitee.com +# gitee_commit_check +# clone the repo +# check commit available +class GiteeCommitUrlCheck + def initialize(my_info, url, base_url) + @my_info = my_info + @url = url + @base_url = base_url + end + + def gitee_commit_check + repo_dir = @url.split('/')[-3] + repo_url = [@base_url, 'git'].join('.') + commit_id = @url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] + + FileUtils.rm_rf repo_dir + + gitee_commit_exist(email_index) + end + + def gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = "We can not conform whether the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with your email.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v7 compass-ci 03/11] mail-robot: parse apply account email
by Luan Shengde 13 Nov '20

13 Nov '20

ParseApplyAccountEmail used to parse my_commit_url and my_ssh_pubkey when execute check_to_send_account in ApplyAccount input: mail_content output: my_commit_url, my_ssh_pubkey parse_commit_url extract_commit_url extract my_commit_url from mail_content base_url_in_upstream_repos check whether the repo's url in upstream-repos commit_url_availability check the commit available - gitee.com call GiteeCommitUrlCheck to check commit available - non-gitee.com execute curl and check feedback for the commit parse_pub_key check attachment for my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/parse-apply-account-email.rb | 130 ++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100755 container/mail-robot/lib/parse-apply-account-email.rb diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb new file mode 100755 index 0000000..f6829c6 --- /dev/null +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -0,0 +1,130 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'gitee-commit-url-check' + +# used to parse mail_content for my_commit_url and my_ssh_pubkey +# be called by AssignAccount when it needs to extract required data: +# - my_commit_url +# - my_ssh_pubkey +# +# input: mail_content +# output: my_commit_url, my_ssh_pubkey +# +# parse_commit_url +# parse my_commit_url from the mail_content and return it +# extract my_commit_url +# extract my_coomit_url from the mail_content +# check commit url exists +# base_url_in_upstream_repos +# check whether the base url in upstream-repos +# commit_url_availability +# check commit available +# gitee.com: +# call GiteeCommitUrlCheck to check the commit +# non-gitee.com: +# execute curl to check the commit +# parse_pub_key +# check my_ssh_pubkey exists and return it +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + + # the commit url should be headed with a prefix: my oss commit + # the commit url must be in a standart format, example: + # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = "No matched commit url found.\n" + error_message += 'Ensure that you have add a right commit url, ' + error_message += 'and with prefix "my oss commit:".' + + raise error_message + end + + def parse_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + base_url_in_upstream_repos('/c/upstream-repos', base_url) + commit_url_availability(url, base_url) + + return url + end + + def base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = "The repo url for your commit is not in our upstream-repo list.\n" + error_message += 'Use a new one, or consulting the manager for available repo list.' + + raise error_message + end + + def commit_url_availability(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit information + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + gitee_commit(url, base_url) + else + non_gitee_commit(url) + end + end + + def gitee_commit(url, base_url) + my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) + my_gitee_commit.gitee_commit_check + end + + def non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_info['my_email'] + + return unless email_index.nil? + + error_message = "We can not confirm the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end + + def parse_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + pub_key_exist(pub_key) + + return pub_key + end + + def pub_key_exist(pub_key) + return unless pub_key.nil? + + error_message = "No pub_key found.\n" + error_message += 'Please add a pub_key as an attachment to your email.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] os-cifs/start: change deprecated name
by Wang Yong 13 Nov '20

13 Nov '20

[problem] The original container could not be deleted due to a container name error before: docker_rm samba after: docker_rm os-cifs Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- container/os-cifs/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/os-cifs/start b/container/os-cifs/start index 6cc08c6..1b2cd23 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -8,7 +8,7 @@ lsmod | grep -q "^cifs\s" || { sudo modprobe cifs } -docker_rm samba +docker_rm os-cifs cmd=( docker run -dt -- 2.23.0

1 0

2025

2024

2023

2022

2021

2020

Compass-ci