- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/6] git_mirror: add webhook handler
by Li Yuanchao 09 Nov '20

09 Nov '20

webhook will send a message when user push. try to find the git_repo when get url of the repository. if found, push it to queue. if not found, it maybe an illegal request, so do nothing Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 65d6940..5332c16 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -88,10 +88,16 @@ class MirrorMain @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info + connection_init + handle_webhook + end + + def connection_init connection = Bunny.new('amqp://172.17.0.1:5672') connection.start channel = connection.create_channel @message_queue = channel.queue('new_refs') + @webhook_queue = channel.queue('web_hook') end def fork_stat_init(stat_key) @@ -287,3 +293,40 @@ class MirrorMain es_repo_update(git_repo) end end + +# main thread +class MirrorMain + def check_git_repo(git_repo, webhook_url) + return @git_info.key?(git_repo) && Array(@git_info[git_repo]['url'])[0] == webhook_url + end + # example + # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware + # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc + # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + def get_git_repo(webhook_url) + strings = webhook_url.split('/') + project = strings[-1] + fork_name = strings[-2] + + git_repo = "#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project}/#{fork_name}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project[0]}/#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + puts "webhook: #{webhook_url} is not found!" + end + + def handle_webhook + Thread.new do + @webhook_queue.subscribe(block: true) do |_delivery, _properties, webhook_url| + git_repo = get_git_repo(webhook_url) + do_push(git_repo) if git_repo + sleep(0.1) + end + end + end +end -- 2.23.0

4 5

[PATCH compass-ci] service/monitoring: fix can't delete closed ws
by Wu Zhende 09 Nov '20

09 Nov '20

Use a hash to save the mapping between "query" and "ws". When call function "add_filter_rule" will change the "query". If query={"job_id": "1"}, will change to "query={"job_id": ["1"]}". So need to use the modified query to delete the key-value pair in the hash. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 2 ++ src/monitoring/monitoring.cr | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 8e672e0..5d3bc8b 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -16,6 +16,8 @@ class Filter @hash[query] = Array(HTTP::WebSocket).new unless @hash[query]? @hash[query] << socket + + return query end private def convert_hash_value_to_array(query) diff --git a/src/monitoring/monitoring.cr b/src/monitoring/monitoring.cr index dff1902..d09c363 100644 --- a/src/monitoring/monitoring.cr +++ b/src/monitoring/monitoring.cr @@ -19,7 +19,7 @@ module Monitoring # also can be {"job_id": ["1", "2"]} query = JSON.parse(msg) if query.as_h? - filter.add_filter_rule(query, socket) + query = filter.add_filter_rule(query, socket) end end -- 2.23.0

1 0

[PATCH compass-ci] kernel_params: add kernel custom params
by Wei Jihui 09 Nov '20

09 Nov '20

[Why] user maybe use custom kernel params to setup OS input: (job yaml) kernel_custom_params: "sched_steal_node_limit=8" output: (setup log) ... ro sched_steal_node_limit=8 rdinit=/sbin/init... Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/scheduler/kernel_params.cr | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 54438b5..a558735 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -7,6 +7,10 @@ class Job return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" end + private def kernel_custom_params + return @hash["kernel_custom_params"] if @hash["kernel_custom_params"]? + end + private def set_kernel_append_root os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") @@ -30,7 +34,7 @@ class Job end private def set_kernel_params - self["kernel_params"] = " #{kernel_common_params()} #{kernel_append_root} #{kernel_console()}" + self["kernel_params"] = " #{kernel_common_params()} #{kernel_custom_params()} #{kernel_append_root} #{kernel_console()}" end end -- 2.23.0

1 0

[PATCH v3 compass-ci 10/11] mail-robot: Dockerfile
by Luan Shengde 09 Nov '20

09 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..7fdc8c8 --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v3 compass-ci 09/11] mail-robot: build file
by Luan Shengde 09 Nov '20

09 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v3 compass-ci 08/11] mail-robot: documentation for service
by Luan Shengde 09 Nov '20

09 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 89 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..ec96ca3 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,89 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + my oss commit: commit_url +attahcment + ssh pub_key + - id_rsa.pub + +## steps overview + +1. monitor + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + apply_account + invoke AssignAccount for new account + +2. assign account + init my_info + - my_email + - my_name + send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email + send mail + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account + apply_jumper_account + premise: pubkey exists + complete my_info + - my_uuid + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + - my_email + - my_name + - my_uuid + - my_commit_url + - my_login_name + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

1 0

[PATCH v3 compass-ci 07/11] mail-robot: build fail message
by Luan Shengde 09 Nov '20

09 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-fail-eamil.rb | 38 ++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 lib/assign-account-fail-eamil.rb diff --git a/lib/assign-account-fail-eamil.rb b/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..d15cd69 --- /dev/null +++ b/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following errors + + #{message} + + In order to successfully apply a account, pay attention to the following points: + + 1. mail subject + the subject should exactly: apply account + + 2. commit url + when you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + ensure you commit url exist and available to access. + + 3. ssh pubkey + you need to add a pubkey as an attachment with name 'id_rsa.pub' to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v3 compass-ci 06/11] mail-robot: build success email
by Luan Shengde 09 Nov '20

09 Nov '20

build email message for successfully assigning account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-email.rb | 52 +++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100755 lib/assign-account-email.rb diff --git a/lib/assign-account-email.rb b/lib/assign-account-email.rb new file mode 100755 index 0000000..3fe95eb --- /dev/null +++ b/lib/assign-account-email.rb @@ -0,0 +1,52 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_email(my_info) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{my_info['my_name']}, + + Thank you for joining us. + + ONE-TIME SETUP + + You can use the following info to submit jobs: + + 1) setup default config + run the following command to add the below setup to default config file + + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF + my_email: #{my_info['my_email']} + my_name: #{my_info['my_name']} + my_uuid: #{my_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + + reference to 'how to write job yaml' section to write the job yaml + you can alse reference to files in compass-ci/jobs as example. + + submit jobs, for example: + + submit jobs/iperf.yaml textbox=vm-2p8g + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v3 compass-ci 05/11] mail-robot: apply account for user
by Luan Shengde 09 Nov '20

09 Nov '20

apply account with my_info and my_ssh_pubkey my_info: - my_email - my_name - my_uuid check account exists Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/apply-jumper-account.rb | 40 +++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 lib/apply-jumper-account.rb diff --git a/lib/apply-jumper-account.rb b/lib/apply-jumper-account.rb new file mode 100755 index 0000000..7e82bb0 --- /dev/null +++ b/lib/apply-jumper-account.rb @@ -0,0 +1,40 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +# apply jumper account for user +# the enter point is apply_jumper_account +# the apply_account_info hash need include +# my_email, my_name, my_uuid, my_ssh_pubkey +class ApplyJumperAccount + def initialize(my_info, my_ssh_pubkey) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + @my_ssh_pubkey = my_ssh_pubkey + end + + def apply_jumper_account + @my_info['my_ssh_pubkey'] = @my_ssh_pubkey + + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = ' No more available jumper account.' + error_message += ' You may try again later or consulting the managet for a solution.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v3 compass-ci 01/11] mail-robot: entry point file for mail-robot service
by Luan Shengde 09 Nov '20

09 Nov '20

call mail-robot.rb parse-apply-account-email.rb assign-account.rb apply-jumper-account.rb assign-account-email.rb assign-account-fail-eamil.rb to run the mail-robot service Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/run-mail-robot.rb | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/mail-robot/run-mail-robot.rb diff --git a/container/mail-robot/run-mail-robot.rb b/container/mail-robot/run-mail-robot.rb new file mode 100755 index 0000000..6cca2d2 --- /dev/null +++ b/container/mail-robot/run-mail-robot.rb @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../../lib/assign-account-email' +require_relative '../../lib/assign-account-fail-eamil' +require_relative '../../lib/apply-jumper-account' +require_relative '../../lib/parse-apply-account-email' +require_relative '../../lib/assign-account' +require_relative '../../lib/mail-robot' + +MAILDIR = '/srv/cci/Maildir/.compass-ci' + +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

1 0