January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/2] sbin: add email-mapping
by Luan Shengde 07 Jan '21

07 Jan '21

for user to: - add - delete - re-map - search email mapping from stdin or file Usage: email_mapping -n name -e email_address email_mapping -n name -e email_address -r email_mapping -n name -d email_mapping -s name email_mapping -f mapping_file email_mapping -f mapping_file -r|-d options: -n, --name name|email|logo appoint a name|email|logo to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings -s, --search search email mapping -h, --help show this message the name can be either a name, an email or something like logo the file content consists of names and emails, like: name1: email_address1 name2: email_address2 you can map multi name to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 90 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100755 sbin/email-mapping.rb diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..09338c8 --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,90 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' +require 'yaml' +require 'optparse' + +require "#{ENV['CCI_SRC']}/lib/email_mapping_client" + +r_name = nil +d_email = nil +email_mapping = nil +re_mapping = false +del_map = false +search_map = false + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name -e email_address\n" + opts.banner += " email_mapping -n name -e email_address -r\n" + opts.banner += " email_mapping -n name -d\n" + opts.banner += " email_mapping -s name\n" + opts.banner += " email_mapping -f mapping_file\n" + opts.banner += " email_mapping -f mapping_file -r|-d\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|logo', '--name name|email|logo', 'appoint a name|email|logo to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || nil + email_mapping = mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + re_mapping = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + del_map = true + end + + opts.on('-s', '--search', 'search email mapping') do + search_map = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.empty? + ARGV << '-h' +elsif (['-n', '-f'] - ARGV).eql? ['-n', '-f'] + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, re_mapping, del_map, search_map) + email_mapping = AddEmailMapping.new(r_name, d_email) + if re_mapping + email_mapping.re_map_email + elsif del_map + email_mapping.delete_email_mapping + elsif search_map + puts email_mapping.search_map + else + email_mapping.add_email_mapping + end +end + +if email_mapping.nil? + email_mapping(r_name, d_email, re_mapping, del_map, search_map) +else + email_mapping.each do |name, email| + email_mapping(name, email, re_mapping, del_map, search_map) + end +end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: add email_mapping_client
by Luan Shengde 07 Jan '21

07 Jan '21

used to - add - delete - re-map - search email mappings Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/email_mapping_client.rb | 45 +++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 lib/email_mapping_client.rb diff --git a/lib/email_mapping_client.rb b/lib/email_mapping_client.rb new file mode 100755 index 0000000..ec82d93 --- /dev/null +++ b/lib/email_mapping_client.rb @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to add/delete/re-map/search email mappings +class AddEmailMapping + REDIS_HOST = ENV['REDIS_HOST'] || 'localhost' + REDIS_PORT = ENV['REDIS_PORT'] || '6379' + + def initialize(r_name, d_email) + @r_name = r_name + @d_email = d_email + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def add_email_mapping + if @redis.hexists 'email_mapping', @r_name + puts "#{@r_name} has already mapped an email." + else + @redis.hset 'email_mapping', @r_name, @d_email + puts "Successfully mapped email for #{@r_name}." + end + end + + def delete_email_mapping + @redis.hdel 'email_mapping', @r_name + end + + def re_map_email + @redis.hset 'email_mapping', @r_name, @d_email + puts "Successfully re-mapped email for #{@r_name}." + end + + def search_map + d_email = @redis.hget 'email_mapping', @r_name + + return d_email if d_email + + return "#{@r_name} has not mapped an email yet." + end +end -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] tests: add email
by Wei Jihui 07 Jan '21

07 Jan '21

send email to user when test job failed Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/email | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 tests/email diff --git a/tests/email b/tests/email new file mode 100755 index 00000000..40b48fd9 --- /dev/null +++ b/tests/email @@ -0,0 +1,40 @@ +#!/bin/sh +# - my_email +# - info + +: ${info:=failure} + +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/http.sh +. $PROJECT_DIR/lib/env.sh + +data_failure() +{ + data="To: $my_email +Subject: [NOTIFY Compass-ci] Test job failed + +Dear $my_name: + + Sorry to inform you that your test job is failed. you can click the following link to view details. + http://api.compass-ci.openeuler.org:$SRV_HTTP_PORT/$result_root + +Regards +Compass-Ci +" +} + +compose_email() +{ + [ "$info" == "failure" ] && data_failure +} + +run_email() +{ + [ -n "$my_email" ] || return 0 + compose_email + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" +} + +run_email -- 2.23.0

2 1

[PATCH v2 lkp-tests] tests/email: get username from my_email
by Wei Jihui 07 Jan '21

07 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/email | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/email b/tests/email index 40b48fd9..a5f87d42 100755 --- a/tests/email +++ b/tests/email @@ -15,7 +15,7 @@ data_failure() data="To: $my_email Subject: [NOTIFY Compass-ci] Test job failed -Dear $my_name: +Dear $my_username: Sorry to inform you that your test job is failed. you can click the following link to view details. http://api.compass-ci.openeuler.org:$SRV_HTTP_PORT/$result_root @@ -33,6 +33,7 @@ compose_email() run_email() { [ -n "$my_email" ] || return 0 + my_username=$(echo "$my_email" | awk -F '@' '{print $1}') compose_email curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" } -- 2.23.0

3 2

[PATCH v2 lkp-tests 2/2] stats/crystal/kernbench.cr: kernbench to kernbench.cr
by Lu Kaiyi 07 Jan '21

07 Jan '21

change kernbench into kernbench.cr Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/kernbench.cr | 32 +++++++++++++++----------------- 1 file changed, 15 insertions(+), 17 deletions(-) diff --git a/stats/crystal/kernbench.cr b/stats/crystal/kernbench.cr index 0da37fe88..9777a11ca 100644 --- a/stats/crystal/kernbench.cr +++ b/stats/crystal/kernbench.cr @@ -1,28 +1,26 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal -LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) +require "../../lib/statistics" -require "#{LKP_SRC}/lib/statistics" - -$stdin.each_line do |line| +STDIN.each_line do |line| case line when /Elapsed Time/ - puts 'elapsed_time: ' + line.split[2] - puts 'elapsed_time_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "elapsed_time: " + line.split[2] + puts "elapsed_time_stddev: " + line.split[3].gsub(/[()]/, "") when /User Time/ - puts 'user_time: ' + line.split[2] - puts 'user_time_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "user_time: " + line.split[2] + puts "user_time_stddev: " + line.split[3].gsub(/[()]/, "") when /System Time/ - puts 'system_time: ' + line.split[2] - puts 'system_time_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "system_time: " + line.split[2] + puts "system_time_stddev: " + line.split[3].gsub(/[()]/, "") when /Percent CPU/ - puts 'percent_cpu: ' + line.split[2] - puts 'percent_cpu_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "percent_cpu: " + line.split[2] + puts "percent_cpu_stddev: " + line.split[3].gsub(/[()]/, "") when /Context Switches/ - puts 'context_switches: ' + line.split[2] - puts 'context_switches_stddev: ' + line.split[3].gsub(/[()]/, "") + puts "context_switches: " + line.split[2] + puts "context_switches_stddev: " + line.split[3].gsub(/[()]/, "") when /Sleeps/ - puts 'sleeps: ' + line.split[1] - puts 'sleeps_stddev: ' + line.split[2].gsub(/[()]/, "") + puts "sleeps: " + line.split[1] + puts "sleeps_stddev: " + line.split[2].gsub(/[()]/, "") end end -- 2.23.0

2 1

[PATCH v2 lkp-tests] jobs/ebizzy.yaml: modify the value of iterations
by Hu Xuejiao 07 Jan '21

07 Jan '21

[why] seq: invalid floating point argument: 100x Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/ebizzy.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/ebizzy.yaml b/jobs/ebizzy.yaml index 9a3e9a16..cc9994c8 100644 --- a/jobs/ebizzy.yaml +++ b/jobs/ebizzy.yaml @@ -3,7 +3,7 @@ testcase: ebizzy category: benchmark nr_threads: 200% -iterations: 100x +iterations: 100 ebizzy: duration: 10s -- 2.23.0

2 2

[PATCH v3 compass-ci] 3-code/git: modify the git download path
by Hu Xuejiao 07 Jan '21

07 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 48741ec..d72b531 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -5,14 +5,10 @@ # fix files permissions cause container service start failed umask 002 -# download git trees - -cd /c || exit - -# git clone https://gitee.com/openeuler/compass-ci.git +# git clone compass-ci, lkp-tests, upstream-repos # modify and manual run for now: -[ -d "compass-ci/.git" ] || { - git clone https://gitee.com/wu_fengguang/compass-ci.git +[ -d "/c/compass-ci/.git" ] || { + git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci cd /c/compass-ci || exit [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" @@ -20,15 +16,15 @@ cd /c || exit ln -s compass-ci /c/cci } -[ -d "lkp-tests/.git" ] || { - git clone https://gitee.com/wu_fengguang/lkp-tests.git +[ -d "/c/lkp-tests/.git" ] || { + git clone https://gitee.com/wu_fengguang/lkp-tests.git /c/lkp-tests cd /c/lkp-tests || exit [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" } -[ -d "upstream-repos/.git" ] || { - git clone https://gitee.com/wu_fengguang/upstream-repos.git +[ -d "/c/upstream-repos/.git" ] || { + git clone https://gitee.com/wu_fengguang/upstream-repos.git /c/upstream-repos cd /c/upstream-repos || exit [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" -- 2.23.0

2 1

[PATCH v4 compass-ci 5/7] send-internet-mail: email rate limit
by Luan Shengde 07 Jan '21

07 Jan '21

add email_in_limit to store email address that send count less than limit value add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..9891ae9 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit +# email_in_limit is used to store email address email count +# less than the limit value +# email_out_limit is used to store email address email count +# beyond the limit +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1 +# case the email in email_in_limit but email count less than the +# limit value, its value will +1 +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

3 3

[PATCH lkp-tests] jobs/ebizzy.yaml: modify the value of iterations
by Hu Xuejiao 07 Jan '21

07 Jan '21

[why] seq: invalid floating point argument: 100 Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/ebizzy.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/ebizzy.yaml b/jobs/ebizzy.yaml index 9a3e9a16..cc9994c8 100644 --- a/jobs/ebizzy.yaml +++ b/jobs/ebizzy.yaml @@ -3,7 +3,7 @@ testcase: ebizzy category: benchmark nr_threads: 200% -iterations: 100x +iterations: 100 ebizzy: duration: 10s -- 2.23.0

2 1

[PATCH v2 lkp-tests 1/2] stats/crystal/kernbench.cr: create kernbench.cr as same content with stats/kernbench
by Lu Kaiyi 07 Jan '21

07 Jan '21

create kernbench.cr in crystal directory from stats/kernbench first, and then, the second patch will know what has changed from original file. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/kernbench.cr | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 stats/crystal/kernbench.cr diff --git a/stats/crystal/kernbench.cr b/stats/crystal/kernbench.cr new file mode 100644 index 000000000..0da37fe88 --- /dev/null +++ b/stats/crystal/kernbench.cr @@ -0,0 +1,28 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +$stdin.each_line do |line| + case line + when /Elapsed Time/ + puts 'elapsed_time: ' + line.split[2] + puts 'elapsed_time_stddev: ' + line.split[3].gsub(/[()]/, "") + when /User Time/ + puts 'user_time: ' + line.split[2] + puts 'user_time_stddev: ' + line.split[3].gsub(/[()]/, "") + when /System Time/ + puts 'system_time: ' + line.split[2] + puts 'system_time_stddev: ' + line.split[3].gsub(/[()]/, "") + when /Percent CPU/ + puts 'percent_cpu: ' + line.split[2] + puts 'percent_cpu_stddev: ' + line.split[3].gsub(/[()]/, "") + when /Context Switches/ + puts 'context_switches: ' + line.split[2] + puts 'context_switches_stddev: ' + line.split[3].gsub(/[()]/, "") + when /Sleeps/ + puts 'sleeps: ' + line.split[1] + puts 'sleeps_stddev: ' + line.split[2].gsub(/[()]/, "") + end +end -- 2.23.0

1 0