January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v6 compass-ci 1/2] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 07 Jan '21

07 Jan '21

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 70 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..91aa6ab 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,49 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = build_aggs_from_fields(fields) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +189,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields) + aggs_hash = {} + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] ||= {} + aggs_hash['aggs']["all_#{field}"] = { + 'terms' => { field: field, size: 1000 } + } + sub_aggs = build_aggs_from_fields(fields) + aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs + aggs_hash +end -- 2.23.0

1 0

[PATCH v4 compass-ci 7/7] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 07 Jan '21

07 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v4 compass-ci 6/7] lib/mail-post: add email checks
by Luan Shengde 07 Jan '21

07 Jan '21

add check email count only email addresses that email count within the limited value can be sent emails add check email mapping if has set email mapping for the entered address. the address will be converted to the mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..803a134 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +97,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v4 compass-ci 4/7] send-internet-mail: add email mapping
by Luan Shengde 07 Jan '21

07 Jan '21

example: name => email_addr logo => email_addr email_addr_r => email_addr_d an email address can be set multi mappings, you can set a name or email to a specified email address when you write either a name or an email address in the send address bar, the email address will be converted to the specified address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..4c068b0 --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,49 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# email address translation according to the email_mapping queue +# email_mapping is used to set email mapping, example: +# email_mapping: +# name => email_address_d +# email_address_r1 => email_address_d +# email_address_r2 => email_address_d +# ... +# can set multi keys for an email address, the key can be a name, +# an email, or some logo else +# case add the key is added to email address bar, the key will be +# transferred to the destination email address +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 0

[PATCH v4 compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 07 Jan '21

07 Jan '21

timing work: reset email rate limit queues everyday Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..62f27ea --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v4 compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 07 Jan '21

07 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: delete unnecessary mkdir
by Li Yuanchao 07 Jan '21

07 Jan '21

Git clone will mkdir itself, so no need to mkdir before. And if FileUtils.mkdir_p fail, it will cause worker thread die, but 'git clone' will not. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 - 1 file changed, 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index b153978..c7dd019 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -72,7 +72,6 @@ class GitMirror if File.directory?(mirror_dir) possible_new_refs = git_fetch(mirror_dir) else - FileUtils.mkdir_p(mirror_dir) possible_new_refs = git_clone(fork_info['url'], mirror_dir) end feedback(fork_info['git_repo'], possible_new_refs) -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: libvirt: a new boot type of libvirt
by Ren Wen 07 Jan '21

07 Jan '21

Requested by: curl -X GET 'http://<host>:<port>/boot.libvirt/mac/<mac_value>' Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b2f0802..02cced6 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -7,7 +7,7 @@ class Sched boot_type = @env.params.url["boot_type"] case boot_type - when "ipxe" + when "ipxe", "libvirt" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) when "grub" host = @redis.hash_get("sched/mac2host", normalize_mac(value)) @@ -135,6 +135,28 @@ class Sched return response.to_json end + private def get_boot_libvirt(job : Job) + response = {} of String => String | Nil | Array(String) | JSON::Any + + response["job_id"] = job.id + response["kernel_uri"] = job.kernel_uri.lchop("kernel ") + + initrds_uri = job["initrds_uri"]? + initrds_uri = initrds_uri.as_a.map(&.as_s.lchop("initrd ")) if initrds_uri + response["initrds_uri"] = initrds_uri + + kernel_params = job["kernel_params"]? + kernel_params = kernel_params.as_a.map(&.to_s).reject(&.starts_with?("initrd=")).join(" ") if kernel_params + response["kernel_params"] = kernel_params + + response["result_root"] = job.result_root + response["LKP_SERVER"] = job["LKP_SERVER"]? + # *templates* is used by libvirt + response["templates"] = job["templates"]? + + return response.to_json + end + private def get_boot_grub(job : Job) initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" @@ -176,6 +198,8 @@ class Sched case boot_type when "ipxe" return job ? get_boot_ipxe(job) : ipxe_msg("No job now") + when "libvirt" + return job ? get_boot_libvirt(job) : {"info" => "No job now"}.to_json when "grub" return job ? get_boot_grub(job) : grub_msg("No job now") when "container" -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] compare: remove not latest failed metrics
by Lu Weitao 07 Jan '21

07 Jan '21

[Why] User just need the failed metrics belong latest job in compare result [How] 1. query job_list and order by start_time desc 2. auto group job_list - the first job of each dimension is the latest job { group_key => { dim1 => [job1, ...] ... } } 3. get latest job_list for each group 4. in compare result, ignore the failed metric that not belongs the latest job_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 22 ++++++++++++++-------- lib/matrix2.rb | 24 +++++++++++++++++------- 3 files changed, 33 insertions(+), 17 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 6a2f515..21e24df 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -62,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices, suites_hash = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, suites_hash, options) + groups_matrices, suites_hash, latest_jobs_hash = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_hash, latest_jobs_hash, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 21bd5c9..fa578f6 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -159,7 +159,7 @@ def get_matrixes_fields(matrixes_list) matrixes_fields end -def get_matrixes_values(matrixes_list, options) +def get_matrixes_values(matrixes_list, latest_jobs, options) # get all matrixes all field values # matrixes_values = { false => {}, true => {} } @@ -168,6 +168,8 @@ def get_matrixes_values(matrixes_list, options) next if field == 'stats_source' success = successful?(field) + next unless success || latest_failure?(field, latest_jobs) + matrixes_values[success][field] = get_values_by_field( matrixes_list, field, matrixes_size, success, options @@ -176,6 +178,10 @@ def get_matrixes_values(matrixes_list, options) matrixes_values end +def latest_failure?(field, latest_jobs) + latest_jobs.any? { |job| job['stats'][field] } +end + def remove_unchanged_field(matrixes_values, suite_list) # remove unchanged field from matrixes values and remove :changed key # @@ -197,7 +203,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -206,7 +212,7 @@ def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key return warn('Matrix cannot be empty!') || '' if matrixes_empty?(matrixes_list) options = { 'perf-profile': 5, theme: :none, no_print: false }.merge(options) - matrixes_values = get_matrixes_values(matrixes_list, options) + matrixes_values = get_matrixes_values(matrixes_list, latest_jobs, options) remove_unchanged_field(matrixes_values, suite_list) if matrixes_list.length > 1 no_print = options[:no_print] result_str = group_key ? "\n\n\n\n\n" + group_key : '' @@ -387,28 +393,28 @@ end # input: group matrices # output: pre compare result of each group # the result with more comparison objects first -def compare_group_matrices(group_matrices, suites_hash, options) +def compare_group_matrices(group_matrices, suites_hash, latest_jobs_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) have_multi_member = group_matrices_array[0][1].size > 1 group_matrices_array.each do |matrice_kv| next if have_multi_member && matrice_kv[1].size < 2 - result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], latest_jobs_hash[matrice_kv[0]], options) end result_str end -def get_matrix_str(matrice_key, matrice_value, suite_list, options) +def get_matrix_str(matrice_key, matrice_value, suite_list, latest_jobs, options) m_list = [] m_titles = [] matrice_value.each do |dim, matrix| m_titles << dim m_list << matrix end - return compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) if options[:no_print] + return compare_matrixes(m_list, suite_list, latest_jobs, m_titles, matrice_key, options: options) if options[:no_print] - print compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) + print compare_matrixes(m_list, suite_list, latest_jobs, m_titles, matrice_key, options: options) return '' end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 09573ec..e603763 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -115,27 +115,37 @@ end # } def combine_group_query_data(job_list, dims) suites_hash = {} + latest_jobs_hash = {} groups = auto_group(job_list, dims) have_multi_member = multi_member?(groups) one_size_count = 0 groups.each do |group_key, value| if value.size < 2 one_size_count += 1 - if have_multi_member || one_size_count > 3 - groups.delete(group_key) - next - end + next if group_deleted?(groups, group_key, have_multi_member, one_size_count) end - get_groups_matrix(groups, group_key, value, suites_hash) + get_groups_matrix(groups, group_key, value, suites_hash, latest_jobs_hash) end - return groups, suites_hash + return groups, suites_hash, latest_jobs_hash end -def get_groups_matrix(groups, group_key, value, suites_hash) +def group_deleted?(groups, group_key, have_multi_member, one_size_count) + was_deleted = false + if have_multi_member || one_size_count > 3 + groups.delete(group_key) + was_deleted = true + end + + was_deleted +end + +def get_groups_matrix(groups, group_key, value, suites_hash, latest_jobs_hash) suite_list = [] + latest_jobs_hash[group_key] = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) + latest_jobs_hash[group_key] << jobs[0] end suites_hash[group_key] = suite_list end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/compare.rb: order query result by start_time with desc
by Lu Weitao 07 Jan '21

07 Jan '21

[Why] 1. When query result include more than 10,000 jobs, we still need the latest jobs use to compare 2. Support we recognize the latest job that have failure messages from each group-dimension-job_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 9090c3b..6a2f515 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -40,7 +40,7 @@ def create_matrices_list(conditions) suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| - query_results = es.multi_field_query(condition) + query_results = es.multi_field_query(condition, desc_keyword: 'start_time') matrix, suites = combine_query_data(query_results) matrices_list << matrix suite_list.concat(suites) @@ -68,7 +68,7 @@ end def create_groups_matrices_list(conditions, dims) es = ESQuery.new(ES_HOST, ES_PORT) - query_results = es.multi_field_query(conditions) + query_results = es.multi_field_query(conditions, desc_keyword: 'start_time') combine_group_query_data(query_results['hits']['hits'], dims) end -- 2.23.0

1 0