January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] git_mirror.rb: optimize method to pass rubocop
by Li Yuanchao 06 Jan '21

06 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 7f265d9..ce1dc9e 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -349,13 +349,13 @@ class MirrorMain end end - def handle_submodule(submodule, parent_project) + def handle_submodule(submodule) submodule.each_line do |line| next unless line.include?('url = ') url = line.split(' = ')[1].chomp git_repo = url.split('://')[1] if url.include?('://') - return unless git_repo + break unless git_repo @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true } fork_stat_init(git_repo) -- 2.23.0

2 1

[PATCH lab-z9] hosts: add new testbox host info
by Xiao Shenwei 06 Jan '21

06 Jan '21

Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/taishan200-2280-2s64p-256g--a115 | 20 ++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a117 | 20 ++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a119 | 20 ++++++++++++++++++++ hosts/vm-2p32g | 6 ++++++ 4 files changed, 66 insertions(+) create mode 100644 hosts/taishan200-2280-2s64p-256g--a115 create mode 100644 hosts/taishan200-2280-2s64p-256g--a117 create mode 100644 hosts/taishan200-2280-2s64p-256g--a119 create mode 100644 hosts/vm-2p32g diff --git a/hosts/taishan200-2280-2s64p-256g--a115 b/hosts/taishan200-2280-2s64p-256g--a115 new file mode 100644 index 0000000..64e2b1a --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a115 @@ -0,0 +1,20 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399888177f5 +rootfs_partition: +- /dev/disk/by-id/scsi-35000039988817359 +mac_addr: + - 44:a1:91:9d:f3:44 + - 44:a1:91:9d:f3:45 + - 44:a1:91:9d:f3:46 + - 44:a1:91:9d:f3:47 + - 44:a1:91:e5:94:51 + - 44:a1:91:e5:94:52 + - 44:a1:91:e5:94:53 + - 44:a1:91:e5:94:54 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.8 diff --git a/hosts/taishan200-2280-2s64p-256g--a117 b/hosts/taishan200-2280-2s64p-256g--a117 new file mode 100644 index 0000000..31167f9 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a117 @@ -0,0 +1,20 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b57d1 +rootfs_partition: +- /dev/disk/by-id/scsi-350000399886b720d +mac_addr: + - 44:67:47:85:f4:0b + - 44:67:47:85:f4:0c + - 44:67:47:85:f4:0d + - 44:67:47:85:f4:0e + - cc:05:77:ee:b3:ff + - cc:05:77:ee:b4:00 + - cc:05:77:ee:b4:01 + - cc:05:77:ee:b4:02 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.10 diff --git a/hosts/taishan200-2280-2s64p-256g--a119 b/hosts/taishan200-2280-2s64p-256g--a119 new file mode 100644 index 0000000..5775be3 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a119 @@ -0,0 +1,20 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b5dbd +rootfs_partition: +- /dev/disk/by-id/scsi-350000399886b6be9 +mac_addr: + - 44:67:47:e9:79:65 + - 44:67:47:e9:79:66 + - 44:67:47:e9:79:67 + - 44:67:47:e9:79:68 + - 44:a1:91:9f:9b:4a + - 44:a1:91:9f:9b:4b + - 44:a1:91:9f:9b:4c + - 44:a1:91:9f:9b:4d +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.12 diff --git a/hosts/vm-2p32g b/hosts/vm-2p32g new file mode 100644 index 0000000..fcb23ed --- /dev/null +++ b/hosts/vm-2p32g @@ -0,0 +1,6 @@ +arch: aarch64 +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 32G -- 2.23.0

1 0

[PATCH lkp-tests 2/2] stats/crystal/kernbench.cr: kernbench to kernbench.cr
by Lu Kaiyi 06 Jan '21

06 Jan '21

change kernbench into kernbench.cr Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/kernbench.cr | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-) diff --git a/stats/crystal/kernbench.cr b/stats/crystal/kernbench.cr index 0da37fe88..9777a11ca 100644 --- a/stats/crystal/kernbench.cr +++ b/stats/crystal/kernbench.cr @@ -1,28 +1,26 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal -LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) +require "../../lib/statistics" -require "#{LKP_SRC}/lib/statistics" - -$stdin.each_line do |line| +STDIN.each_line do |line| case line when /Elapsed Time/ - puts 'elapsed_time: ' + line.split[2] - puts 'elapsed_time_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "elapsed_time: " + line.split[2] + puts "elapsed_time_stddev: " + line.split[3].gsub(/[()]/, "") when /User Time/ - puts 'user_time: ' + line.split[2] - puts 'user_time_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "user_time: " + line.split[2] + puts "user_time_stddev: " + line.split[3].gsub(/[()]/, "") when /System Time/ - puts 'system_time: ' + line.split[2] - puts 'system_time_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "system_time: " + line.split[2] + puts "system_time_stddev: " + line.split[3].gsub(/[()]/, "") when /Percent CPU/ - puts 'percent_cpu: ' + line.split[2] - puts 'percent_cpu_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "percent_cpu: " + line.split[2] + puts "percent_cpu_stddev: " + line.split[3].gsub(/[()]/, "") when /Context Switches/ - puts 'context_switches: ' + line.split[2] - puts 'context_switches_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "context_switches: " + line.split[2] + puts "context_switches_stddev: " + line.split[3].gsub(/[()]/, "") when /Sleeps/ - puts 'sleeps: ' + line.split[1] - puts 'sleeps_stddev: ' + line.split[2].gsub(/[()]/, "") + puts "sleeps: " + line.split[1] + puts "sleeps_stddev: " + line.split[2].gsub(/[()]/, "") end end -- 2.23.0

1 0

[PATCH v3 compass-ci 7/7] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 06 Jan '21

06 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v3 compass-ci 6/7] lib/mail-post: add email checks
by Luan Shengde 06 Jan '21

06 Jan '21

add check email count only email addresses that email count within the limited value can be sent emails add check email mapping if has set email mapping for the entered address. the address will be converted to the mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..803a134 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +97,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v3 compass-ci 5/7] send-internet-mail: email rate limit
by Luan Shengde 06 Jan '21

06 Jan '21

add email_in_limit to store email address that send count less than limit value add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..a5ca748 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,50 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# to check if the emails count has beyond the limit +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 0

[PATCH v3 compass-ci 4/7] send-internet-mail: add email mapping
by Luan Shengde 06 Jan '21

06 Jan '21

example: name => email_addr email_addr_a => email_addr email_addr_b => email_addr an email address can be set multi mappings, you can set a name or email to a specified email address when you write either a name or an email address in the send address bar, the email address will be converted to the specified address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..f190896 --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,39 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# to check if the emails count has beyond the limit +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 0

[PATCH v3 compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 06 Jan '21

06 Jan '21

timing work: reset email rate limit queues everyday Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..099d89d --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday +# the email counts queues will be reset everyday +# to enable to send mail in the new day to user +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v3 compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 06 Jan '21

06 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH v3 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 06 Jan '21

06 Jan '21

[why] the service need to install redis to store email counts to redis queue the service need to install rufus-scheduler to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 0