December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] lib: delete the space on the left and right sides of the vertical bar
by Bai Jing 04 Dec '20

04 Dec '20

[why] The multi-args is like this "test|nr_threads", if user add it like this "test |nr_threads" or the value like "- shell1 | 1 ". So we need to delete the space Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 2869dcb3..47fd56d3 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -290,11 +290,11 @@ class Job hash.each { |key, value| next unless key =~ /^\w.*\|.*\w$/ - key_array = key.split('|') + key_array = key.split('|').map!{ |ele| ele.strip } [value].flatten.each do |v| next unless v =~ /^\w.*\|.*\w$/ - v_array = v.split('|') + v_array = v.split('|').map!{ |v_ele| v_ele.strip } next unless key_array.size == v_array.size hash_job = hash.clone -- 2.23.0

2 1

[PATCH compass-ci 3/3] scheduler/Sched: use JSONLogger in Sched class
by Wu Zhende 04 Dec '20

04 Dec '20

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 3 +-- src/scheduler/find_job_boot.cr | 6 +++--- src/scheduler/submit_job.cr | 2 +- 3 files changed, 5 insertions(+), 6 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 22e0e4e..abb8a33 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -34,8 +34,7 @@ class Sched @block_helper = BlockHelper.new @rgc = RemoteGitClient.new @env = env - @log = JSONLogger.new - @log.set_env(env) + @log = env.get("log").as(JSONLogger) end def normalize_mac(mac : String) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 6f51640..3e33604 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -192,15 +192,15 @@ class Sched begin job = @es.get_job(job_id.to_s) rescue ex - puts "Invalid job (id=#{job_id}) in es. Info: #{ex}" - puts ex.inspect_with_backtrace + @log.warn("Invalid job (id=#{job_id}) in es. Info: #{ex}") + @log.warn(ex.inspect_with_backtrace) end end if job job.update({"testbox" => testbox}) job.set_result_root - puts %({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state": "set result root"}) + @log.info(%({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state": "set result root"})) @redis.set_job(job) end return job diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index a9eb3a7..f93ad2b 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -19,7 +19,7 @@ class Sched return submit_cluster_job(job, cluster_config) end rescue ex - puts ex.inspect_with_backtrace + @log.warn(ex.inspect_with_backtrace) return [{ "job_id" => "0", "message" => ex.to_s, -- 2.23.0

1 0

[PATCH compass-ci 1/3] lib/json_logger.cr: support initialize env
by Wu Zhende 04 Dec '20

04 Dec '20

Can use JSONLogger.new(env: env) Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index 2b5ef6a..d136bdd 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -6,8 +6,7 @@ require "json" require "any_merge" class JSONLogger < Logger - def initialize(logdev = STDOUT, formatter = my_formatter) - @env = nil + def initialize(logdev = STDOUT, formatter = my_formatter, @env = nil) @env_info = Hash(String, String | Int32).new super(logdev, formatter: formatter) end -- 2.23.0

1 0

[PATCH compass-ci] answerback-email: add check account exists
by Luan Shengde 04 Dec '20

04 Dec '20

add check account exists for assigning/updating account avoid re-assign accounts for single email address [why] the tool may assign a new account for the email even if the email already has one. It will cause a big waste of account resource for wrong operation or test. [how] add check account exists for the email before assigning a new account. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 38 ++++++++++++++------ 1 file changed, 27 insertions(+), 11 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 53c321d..836ad86 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -61,21 +61,19 @@ conf_info = { 'is_update_account' => false } -def init_info(mail_content, email_info, my_info) +def init_info(mail_content, email_info, my_info, my_info_es) my_info['my_email'] = mail_content.from[0] + + read_my_login_name(my_info['my_email'], my_info_es) email_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') - return if mail_content.attachments.empty? - email_info['new_email_pubkey'] = mail_content.attachments[0].body.decoded.strip + email_info['new_email_pubkey'] = mail_content.attachments[0].body.decoded.strip || nil end def read_my_login_name(my_email, my_info_es) - my_account_info_str = %x(curl -XGET localhost:9200/accounts/_doc/#{my_email}) - my_account_info = YAML.safe_load my_account_info_str - message = "No such email found from the ES: #{my_email}" - raise message unless my_account_info['found'] + my_account_info = ESQuery.new(index: 'accounts').query_by_id(my_email) - my_info_es.update my_account_info['_source'] + my_info_es.update my_account_info || {} end options = OptionParser.new do |opts| @@ -102,6 +100,7 @@ options = OptionParser.new do |opts| end my_info['my_email'] = email_address + read_my_login_name(email_address, my_info_es) end opts.on('-n name', '--name name', 'appoint name') do |name| @@ -134,7 +133,7 @@ options = OptionParser.new do |opts| return false end mail_content = Mail.read(email_file) - init_info(mail_content, email_info, my_info) + init_info(mail_content, email_info, my_info, my_info_es) end opts.on('-g', '--gen-sshkey', 'generate jumper rsa public/private key and return pubkey') do @@ -142,7 +141,6 @@ options = OptionParser.new do |opts| end opts.on('-u', '--update', 'updata configurations') do - read_my_login_name(my_info['my_email'], my_info_es) conf_info['is_update_account'] = true end @@ -223,7 +221,7 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) end def check_server - return true if ENV['HOSTNAME'] == 'z9' + return true if ENV['HOSTNAME'] == 'crystal.ci' message = 'please run the tool on z9 server' puts message @@ -240,9 +238,27 @@ def check_my_name_exist(my_info) return false end +def check_email_assigned_account(conf_info, my_info_es) + if conf_info['is_update_account'] + return true unless my_info_es.empty? + + message = "The email has not assigned an account yet.\n" + message += "Delete the '-u' option to assign a new account." + else + return true if my_info_es.empty? + + message = "The email has already assigned an account.\n" + message += "You can use option '-u' to update the account." + end + puts message + + return false +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) return unless check_server return unless check_my_email(my_info) + return unless check_email_assigned_account(conf_info, my_info_es) my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 0

[PATCH v4 compass-ci] mail-robot: add check account exists for email
by Luan Shengde 04 Dec '20

04 Dec '20

avoid re-assign accounts for single email address [why] the mail-robot may assign a new account if user send 'apply account' email with the same email address. and this is a big waste of account resource. [how] when receive request for applying an account, check whether the email already has already been assigned an account. assign a new one if there is none return the old one if there is already one and update info mail robot will update infos if the mail has been set new: - my_name - my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 34 +++++++++++++++++------ 1 file changed, 25 insertions(+), 9 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..99da510 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -51,6 +51,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +63,8 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + my_account_es = check_account_es + apply_my_account(my_account_es) store_account_info send_mail('') @@ -82,15 +83,30 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp - - @my_info['my_uuid'] = my_uuid + def apply_my_account(my_account_es) + apply_info = {} + if my_account_es + apply_info.update my_account_es + apply_info.update @my_info + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + end + + def check_account_es + ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) end def store_account_info -- 2.23.0

1 0

[PATCH v3 compass-ci] mail-robot: add check account exists for email
by Luan Shengde 04 Dec '20

04 Dec '20

avoid re-assigned accounts for single email address [why] the mail-robot may assign a new account if user send 'apply account' email with the same email address. and this is a big waste of account resource. [how] when receive request for applying an account, check whether the email already has already been assigned an account. assign a new one if there is none return the old one if there is already one and update info mail robot will update infos if the mail has been set new: - my_name - my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 36 +++++++++++++++++------ 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..0b94ca5 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -51,6 +51,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +63,8 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + my_account_es = check_account_es + apply_my_account(my_account_es) store_account_info send_mail('') @@ -82,15 +83,32 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp - - @my_info['my_uuid'] = my_uuid + def apply_my_account(my_account_es) + apply_info = {} + if my_account_es + apply_info.update my_account_es + apply_info.update @my_info + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es + end + + def check_account_es + ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) + # my_account_info_str = %x(curl -XGET #{@es_host}:#{@es_port}/accounts/_doc/#{@my_info['my_email']}) + # YAML.safe_load my_account_info_str end def store_account_info -- 2.23.0

1 1

[PATCH v2 compass-ci] mail-robot: add check account exists for email
by Luan Shengde 04 Dec '20

04 Dec '20

avoid multi accounts assignment for single email address [why] the mail-robot may assign a new account if user send 'apply account' email with the same email address. and this is a big waste of account resource. [how] when get the request for applying account, check whether the email already has already been assigned an account. assign a new one if there is none return the old one if there is already one and update info mail robot will update infos if the mail has been set new: - my_name - my_ssh_pubkey Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 36 +++++++++++++++++------ 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index bf40429..d499840 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -12,6 +12,7 @@ require_relative 'apply-jumper-account' require_relative 'parse-apply-account-email' SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 +ES_PORT = ENV['ES_PORT'] || 9200 # assign uuid/account for user # when mail-robot listened new email, and the email's subject @@ -51,6 +52,8 @@ class ApplyAccount @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp @send_mail_port = SEND_MAIL_PORT @mail_content = mail_content + @es_host = @send_mail_host + @es_port = ES_PORT @my_info = {} end @@ -61,9 +64,8 @@ class ApplyAccount # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] parse_mail_content - acct_info = apply_my_account - - @my_info['my_login_name'] = acct_info['my_login_name'] + my_account_es = check_account_es + apply_my_account(my_account_es) store_account_info send_mail('') @@ -82,15 +84,31 @@ class ApplyAccount @my_info.update parsed_email_info end - def apply_my_account - my_uuid = %x(uuidgen).chomp - - @my_info['my_uuid'] = my_uuid + def apply_my_account(my_account_es) + apply_info = {} + if my_account_es['found'] + apply_info.update my_account_es['_source'] + apply_info.update @my_info + @my_info.update apply_info + apply_info['is_update_account'] = true + else + my_uuid = %x(uuidgen).chomp + @my_info['my_uuid'] = my_uuid + apply_info.update @my_info + end + apply_new_account(apply_info, my_account_es) + end - apply_account = ApplyJumperAccount.new(@my_info) + def apply_new_account(apply_info, my_account_es) + apply_account = ApplyJumperAccount.new(apply_info) acct_info = apply_account.apply_jumper_account - return acct_info + @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es['found'] + end + + def check_account_es + my_account_info_str = %x(curl -XGET #{@es_host}:#{@es_port}/accounts/_doc/#{@my_info['my_email']}) + YAML.safe_load my_account_info_str end def store_account_info -- 2.23.0

2 2

[PATCH v3 lkp-tests] stats/crystal/iozone.cr: keep the changes consistent with iozone
by Lu Kaiyi 04 Dec '20

04 Dec '20

[why] original iozone tool can match the line like: 64 4 1599680 3057153 4274062 4897948 3588436 2892445 2772930 3057153 3958892 3057153 3022727 3363612 4564786 but cannot match the line as below(there are extra whitespace after 2892445): 64 4 1599680 3057153 4274062 4897948 3588436 2892445 Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/iozone.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/stats/crystal/iozone.cr b/stats/crystal/iozone.cr index 7ea6e4b7..2a3ffbe2 100644 --- a/stats/crystal/iozone.cr +++ b/stats/crystal/iozone.cr @@ -8,7 +8,8 @@ per_record = Hash(String | String, Int32).new all_sum = 0 nr_records = 0 while (line = STDIN.gets) - next unless line =~ /^\s*\d+.*\d+$/ + line = line.strip + next unless line =~ /^\d+.*\d+$/ data = line.split data.each_index do |i| -- 2.23.0

1 0

[PATCH lkp-tests] Automatic compilation and configure the variables
by Hu Xuejiao 04 Dec '20

04 Dec '20

[why] Source shell_file based on different shells. [how] First, make and make install is written to the make-env. Second, invoking shell_profile(env.sh) that will distinguish different shell_files. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- make-env | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 make-env diff --git a/make-env b/make-env new file mode 100644 index 00000000..cab39288 --- /dev/null +++ b/make-env @@ -0,0 +1,6 @@ +#!/bin/bash +. lib/env.sh +make +make install + +source $(shell_profile) -- 2.23.0

3 2

[PATCH compass-ci] git_mirror.rb: disable some unnecessary messages
by Li Yuanchao 04 Dec '20

04 Dec '20

Disable some unnecessary messages, so important logs will not be covered. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index edee331..9f254f5 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -235,7 +235,7 @@ class MirrorMain return if @git_info[git_repo]['is_submodule'] mirror_dir = "/srv/git/#{git_repo}.git" - show_ref_out = %x(git -C #{mirror_dir} show-ref --heads) + show_ref_out = %x(git -C #{mirror_dir} show-ref --heads 2>/dev/null) cur_refs = { heads: {} } show_ref_out.each_line do |line| next if line.start_with? '#' -- 2.23.0

1 0