- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] 3-code/git: modify the git download path
by Hu Xuejiao 05 Jan '21

05 Jan '21

[why] if git path is not specified, lkp-tests will be downloaded in /c/compass-ci Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 48741ec..12f27c8 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -5,14 +5,10 @@ # fix files permissions cause container service start failed umask 002 -# download git trees - -cd /c || exit - -# git clone https://gitee.com/openeuler/compass-ci.git +# git clone compass-ci,lkp-tests,upstream-repos # modify and manual run for now: [ -d "compass-ci/.git" ] || { - git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci cd /c/compass-ci || exit [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" @@ -21,14 +17,14 @@ cd /c || exit } [ -d "lkp-tests/.git" ] || { - git clone https://gitee.com/wu_fengguang/lkp-tests.git + git clone https://gitee.com/wu_fengguang/lkp-tests.git /c/lkp-tests cd /c/lkp-tests || exit [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" } [ -d "upstream-repos/.git" ] || { - git clone https://gitee.com/wu_fengguang/upstream-repos.git + git clone https://gitee.com/wu_fengguang/upstream-repos.git /c/upstream-repos cd /c/upstream-repos || exit [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" -- 2.23.0

1 0

[PATCH v4 compass-ci] docker-rootfs: add README.md
by Wang Chenglong 05 Jan '21

05 Jan '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..e7b5084 --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,18 @@ +# Use the tool to create a new nfsroot for os + +## Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs. + +## Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + # Please ensure $HOME/.config/compass-ci/rootfs-passwd exists. + +## Some configuration items: + ./packages-to-install + # If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + # Set the password for the image into this file. -- 2.23.0

1 0

[PATCH compass-ci] lib/matrix2.rb: delete useless code
by Lu Weitao 05 Jan '21

05 Jan '21

[Why] We had filtered all useless job which include nil job['suite'] by the compass-ci/lib/params_group.rb def extract_jobs_list(jobs_list) jobs_list.map! do |job| job['_source'] if job_is_useful?(job) end ... end def job_is_useful?(job) ... suite = job['_source']['suite'] return unless suite && stats.keys.any? { |stat| stat.start_with?(suite) } ... end Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index d8326c5..09573ec 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -83,7 +83,7 @@ def create_matrix(job_list) suites = [] job_list.each do |job| stats = job['stats'] - suites << job['suite'] if job['suite'] + suites << job['suite'] stats.each do |key, value| next if key.include?('timestamp') -- 2.23.0

1 0

[PATCH v3 compass-ci] docker-rootfs/common: add comment for set password file.
by Wang Chenglong 05 Jan '21

05 Jan '21

[Why] add password file is easy to ignore. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- .../README.md | 0 .../{docker-rootfs => docker2rootfs}/common | 43 ++++++++++--------- .../packages-to-install | 0 .../{docker-rootfs => docker2rootfs}/run | 0 .../setup-image | 0 5 files changed, 22 insertions(+), 21 deletions(-) rename container/{docker-rootfs => docker2rootfs}/README.md (100%) rename container/{docker-rootfs => docker2rootfs}/common (77%) rename container/{docker-rootfs => docker2rootfs}/packages-to-install (100%) rename container/{docker-rootfs => docker2rootfs}/run (100%) rename container/{docker-rootfs => docker2rootfs}/setup-image (100%) diff --git a/container/docker-rootfs/README.md b/container/docker2rootfs/README.md similarity index 100% rename from container/docker-rootfs/README.md rename to container/docker2rootfs/README.md diff --git a/container/docker-rootfs/common b/container/docker2rootfs/common similarity index 77% rename from container/docker-rootfs/common rename to container/docker2rootfs/common index fe37da9..c7ae8f5 100755 --- a/container/docker-rootfs/common +++ b/container/docker2rootfs/common @@ -33,7 +33,8 @@ usage() dst_rootfs_new_abspath: destination absolute path to create for rootfs Example: - ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + # Please ensure $HOME/.config/compass-ci/rootfs-passwd exists. " } @@ -100,26 +101,26 @@ cp_rootfs() } create_get_initrd() { - echo "Creating initrd.lkp via container/dracut-initrd..." - cd "$CCI_SRC/container/dracut-initrd" || { - echo "Failed to change into $CCI_SRC/container/dracut-initrd" - exit 6 - } - ./run "$ROOTFS_DIR/lib/modules/$ROOTFS_KERNEL" - - echo "Finding initrd.lkp under $ROOTFS_DIR/boot ..." - cd "$ROOTFS_DIR" || { - echo "Failed to change into dir \"$ROOTFS_DIR\"" - exit 7 - } - local initrd_lkp=$(find ./boot -name "initramfs.lkp*") && export ROOTFS_INITRD_LKP=$initrd_lkp - - [ -f "$ROOTFS_INITRD_LKP" ] || { - echo "Failed to generate \"$ROOTFS_INITRD_LKP\"" - exit 8 - } - - echo "initrd_lkp: $ROOTFS_INITRD_LKP" + echo "Creating initrd.lkp via container/dracut-initrd..." + cd "$CCI_SRC/container/dracut-initrd" || { + echo "Failed to change into $CCI_SRC/container/dracut-initrd" + exit 6 + } + ./run "$ROOTFS_DIR/lib/modules/$ROOTFS_KERNEL" + + echo "Finding initrd.lkp under $ROOTFS_DIR/boot ..." + cd "$ROOTFS_DIR" || { + echo "Failed to change into dir \"$ROOTFS_DIR\"" + exit 7 + } + local initrd_lkp=$(find ./boot -name "initramfs.lkp*") && export ROOTFS_INITRD_LKP=$initrd_lkp + + [ -f "$ROOTFS_INITRD_LKP" ] || { + echo "Failed to generate \"$ROOTFS_INITRD_LKP\"" + exit 8 + } + + echo "initrd_lkp: $ROOTFS_INITRD_LKP" } get_rootfs_kernel() diff --git a/container/docker-rootfs/packages-to-install b/container/docker2rootfs/packages-to-install similarity index 100% rename from container/docker-rootfs/packages-to-install rename to container/docker2rootfs/packages-to-install diff --git a/container/docker-rootfs/run b/container/docker2rootfs/run similarity index 100% rename from container/docker-rootfs/run rename to container/docker2rootfs/run diff --git a/container/docker-rootfs/setup-image b/container/docker2rootfs/setup-image similarity index 100% rename from container/docker-rootfs/setup-image rename to container/docker2rootfs/setup-image -- 2.23.0

1 1

[PATCH compass-ci] container/assign-account/answerback-email.rb: fix spell error
by Lu Kaiyi 04 Jan '21

04 Jan '21

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- container/assign-account/answerback-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index a45fef2..a99306b 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -140,7 +140,7 @@ options = OptionParser.new do |opts| init_info(mail_content, email_info, my_info, my_info_es) end - opts.on('-u', '--update', 'updata configurations') do + opts.on('-u', '--update', 'update configurations') do conf_info['is_update_account'] = true end -- 2.23.0

1 0

[PATCH compass-ci 4/7] send-internet-mail: add email mapping
by Luan Shengde 04 Jan '21

04 Jan '21

if user have multi email address, can set mapping to redis queue 'email_mapping' to enable to send mail to a specified address of user with any email address of user in the send address bar example: email_addr_a => email_addr email_addr_b => email_addr when you write either email_addr_a or email_addr_b in the send address bar, the email address will be converted to email_addr Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..60ff3b8 --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,39 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# to check if the emails count has beyond the limit +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_mapping', email + mapped_email = @redis.hget 'email_mapping', 'luanshd(a)crystal.ci' + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + end + return mail_list + end +end -- 2.23.0

1 1

[PATCH compass-ci 7/7] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 04 Jan '21

04 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH compass-ci 6/7] lib/mail-post: add email check
by Luan Shengde 04 Jan '21

04 Jan '21

add check email count only email addresses that email count within the limited value can be sent mail add check email mapping if has set email mapping for an email address. the email address will be converted to the mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..eb21ddb --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,31 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + check_email_limit(mail_info.clone) end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +96,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH compass-ci 5/7] send-internet-mail: add email limit queue
by Luan Shengde 04 Jan '21

04 Jan '21

add email_in_limit to store email address that send count less than limit value add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../email_redis_queue_limit.rb | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100755 container/send-internet-mail/email_redis_queue_limit.rb diff --git a/container/send-internet-mail/email_redis_queue_limit.rb b/container/send-internet-mail/email_redis_queue_limit.rb new file mode 100755 index 0000000..0999983 --- /dev/null +++ b/container/send-internet-mail/email_redis_queue_limit.rb @@ -0,0 +1,48 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# to check if the emails count has beyond the limit +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + end + + if @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + if (@redis.hget 'email_in_limit', email).to_i >= 10 + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end + end + return mail_list + end +end -- 2.23.0

1 0

[PATCH compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 04 Jan '21

04 Jan '21

timing work to reset email queues Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..099d89d --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday +# the email counts queues will be reset everyday +# to enable to send mail in the new day to user +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0