January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sbin/result2stats: support merge yaml file
by Lu Weitao 12 Jan '21

12 Jan '21

[Why] user may extract $program result to $program.yaml Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index e603763..8444334 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -39,19 +39,21 @@ def file_check(file) File.basename(file, '.json') when /\.json\.gz$/ File.basename(file, '.json.gz') + when /\.yaml$/ + File.basename(file, '.yaml') end end def create_stats(result_root) stats = {} - monitor_files = Dir["#{result_root}/*.{json,json.gz}"] + monitor_files = Dir["#{result_root}/*.{json,json.gz,yaml}"] monitor_files.each do |file| next unless File.size?(file) monitor = file_check(file) - next if monitor == 'stats' # stats.json already created? + next if %w[stats job].include?(monitor) extract_pre_result(stats, monitor, file) end -- 2.23.0

1 0

[PATCH lkp-tests 3/3] stats/crystal/memory--stat.cr: fix some errors manually
by Lu Kaiyi 12 Jan '21

12 Jan '21

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr index da7ac0d86..f45ab5fb1 100644 --- a/stats/crystal/memory--stat.cr +++ b/stats/crystal/memory--stat.cr @@ -6,14 +6,14 @@ def parse cg = "nocg" - $stdin.each_line do |line| + STDIN.each_line do |line| case line when /^time:/ puts line when /^cgroup: (\S+)/ cg = $1 when /(\S+)\s*(\S+)/ - k, v, = line.split + k, v = line.split puts "#{cg}.#{k}: #{v}" end end -- 2.23.0

1 0

[PATCH lkp-tests 2/3] stats/crystal/memory--stat.cr: do first conversion by ruby2crystal tools
by Lu Kaiyi 12 Jan '21

12 Jan '21

use the ruby2crystal tools do first conversion like below: /c/crystal/ruby_crystal_codemod/exe/ruby_crystal_codemod memory--stat.cr /c/lkp-tests/tools/ruby2crystal.sh memory--stat.cr Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/memory--stat.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/stats/crystal/memory--stat.cr b/stats/crystal/memory--stat.cr index 1a39e4075..da7ac0d86 100644 --- a/stats/crystal/memory--stat.cr +++ b/stats/crystal/memory--stat.cr @@ -1,11 +1,11 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal # time: 123456 # cgroup: pmbench.1 # cache 0 def parse - cg = 'nocg' + cg = "nocg" $stdin.each_line do |line| case line when /^time:/ -- 2.23.0

1 0

[PATCH compass-ci] addsoftware: add to clean hidden files in runtime dir
by Xu Xijian 12 Jan '21

12 Jan '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index 60e5e07..85f038b 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -43,7 +43,7 @@ install_cmds=( bash -c " yum install -y --skip-broken --installroot=/mnt/root $packages cd /mnt/root - rm -rf srv/* opt/* mnt/* media/* home/* lost+found/* dev/* proc/* run/* sys/* + rm -rf srv/{*,.*} opt/{*,.*} mnt/{*,.*} media/{*,.*} home/{*,.*} lost+found/{*,.*} dev/{*,.*} proc/{*,.*} run/{*,.*} sys/{*,.*} " ) -- 2.23.0

2 4

[PATCH v1 compass-ci] lib/job.cr: set the upload path of package based on os attribute of top-level package style
by Cao Xueliang 12 Jan '21

12 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..ea0c259 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -249,11 +249,33 @@ class Job set_upload_dirs() end - def get_package_dir - if @hash["cci-makepkg"]? || @hash["cci-depends"]? || @hash["build-pkg"]? - mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - common_dir = "#{mount_type}/#{os}/#{os_arch}/#{os_version}" + def get_pkg_common_dir + pkg_style = nil + ["cci-makepkg", "cci-depends", "build-pkg"].each do |item| + if @hash.has_key?(item) + pkg_style = item + break + end end + return nil unless pkg_style + + tmp = @hash["#{pkg_style}"] + tmp_os = tmp["os"]? ? tmp["os"].to_s : "#{os}" + tmp_os_arch = tmp["os_arch"]? ? tmp["os_arch"].to_s : "#{os_arch}" + tmp_os_version = tmp["os_version"]? ? tmp["os_version"].to_s : "#{os_version}" + + tmp_os_mount = tmp["os_mount"]? ? tmp["os_mount"].to_s : "#{os_mount}" + mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + + return common_dir + end + + def get_package_dir + package_dir = "" + common_dir = get_pkg_common_dir + return package_dir unless common_dir if @hash["cci-makepkg"]? package_dir = ",/initrd/pkg/#{common_dir}/#{@hash["cci-makepkg"]["benchmark"]}" @@ -265,10 +287,9 @@ class Job else package_name = @hash["pkgbuild_repo"].to_s.split("/")[-1] end + package_dir = ",/initrd/build-pkg/#{common_dir}/#{package_name}" package_dir += ",/cci/build-config" if @hash["config"]? - else - package_dir = "" end return package_dir -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: set default value
by Liu Yinsi 12 Jan '21

12 Jan '21

$lab will be used in $CCI_SRC/sbin/build-my-info.rb, so add it. can see commit 66b58a7be3cf371c74946950ebe9dc73efd9a16e Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 1b311f0..69b31cd 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -43,6 +43,7 @@ LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +lab: nolab EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH v6 compass-ci 7/7] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 12 Jan '21

12 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v6 compass-ci 6/7] lib/mail-post: add email counts/mapping checks
by Luan Shengde 12 Jan '21

12 Jan '21

add check email count: only email addresses that email count within the limited value can be sent emails add check email mapping: the address will be converted to the mapping value if it has been set an email mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..803a134 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +97,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v6 compass-ci 5/7] send-internet-mail: email rate limit
by Luan Shengde 12 Jan '21

12 Jan '21

add email_in_limit to store email address that send count less than limit value. add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..c7e3954 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit. +# email_in_limit is used to store email address email count +# less than the limit value. +# email_out_limit is used to store email address email count +# beyond the limit. +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1. +# case the email in email_in_limit but email count less than the +# limit value, its value will +1. +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit. +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar. +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 0

[PATCH v6 compass-ci 4/7] send-internet-mail: add email mapping
by Luan Shengde 12 Jan '21

12 Jan '21

example: name => email_addr tag => email_addr email_addr_r => email_addr_d an email address can be set multi mappings, you can set a name or email to a specified email address. when you write either a name or an email address in the send address bar, the email address will be converted to the mapped email address. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..02bfc6b --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,50 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# email address translation according to the email_mapping queue. +# email_mapping is used to set email mapping, example: +# email_mapping: +# name => email_address_d +# tag => email_address_d +# email_address_r => email_address_d +# ... +# +# can set multi keys for an email address, the key can be a name, +# an email, or some something else like a tag. +# case add the key is added to email address bar, the key will be +# transferred to the mapped email address. +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 0