- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci 1/6] container/send-internet-mail: mail-post.rb
by Luan Shengde 21 Oct '20

21 Oct '20

get request for send mail analysis email data invoke send-internet-mail to send the mail Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- lib/mail-post.rb | 66 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb new file mode 100755 index 0000000..889fb23 --- /dev/null +++ b/lib/mail-post.rb @@ -0,0 +1,66 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'yaml' +require 'open3' +# require_relative "#{ENV['SMTP_FILE']}" + +set :bind, '0.0.0.0' +set :port, ENV['SEND_MAIL_PORT'] + +post '/send_mail_yaml' do + data = YAML.safe_load request.body.read + raise TypeError, data, 'request data type error' unless data.class.eql? Hash + + mail_info = { + 'references' => data['references'], + 'from' => data['from'], + 'subject' => data['subject'], + 'to' => data['to'], + 'body' => data['body'] + } + check_send_mail(mail_info) +end + +post '/send_mail_text' do + data = Mail.read_from_string(request.body.read) + + mail_info = { + 'references' => data.references, + 'from' => data.from, + 'subject' => data.subject, + 'to' => data.to, + 'body' => data.body.decoded + } + check_send_mail(mail_info) +end + +def check_send_mail(mail_info) + if ENV['SMTP_REGION'] == 'internet-smtp' + mail_info['from'] = ENV['ROBOT_EMAIL_ADDRESS'] + else + mail_info['from'] = ENV['ROBOT_EMAIL_ADDRESS'] if mail_info['from'].nil? + end + + raise 'no subject.' if mail_info['subject'].nil? + raise 'no email address.' if mail_info['to'].nil? + raise 'no email content.' if mail_info['body'].nil? + + send_mail(mail_info) +end + +def send_mail(mail_info) + setup_smtp + mail = Mail.new do + references mail_info['references'] + from mail_info['from'] + subject mail_info['subject'] + to mail_info['to'] + body mail_info['body'] + end + mail.deliver! +end -- 2.23.0

2 7

[PATCH v4 compass-ci 3/6] container/send-internet-mail: run.rb
by Luan Shengde 21 Oct '20

21 Oct '20

invoke mail-post/internet-smtp to run the send mail service Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/run.rb | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 container/send-internet-mail/run.rb diff --git a/container/send-internet-mail/run.rb b/container/send-internet-mail/run.rb new file mode 100644 index 0000000..efda5b9 --- /dev/null +++ b/container/send-internet-mail/run.rb @@ -0,0 +1,6 @@ +#!/usr/bin/env ruby + +require 'sinatra' + +require "#{ENV['CCI_SRC']}/lib/internet-smtp" +require "#{ENV['CCI_SRC']}/lib/mail-post" -- 2.23.0

2 1

[PATCH v5 compass-ci] container/assign-account: answerback-email.rb
by Luan Shengde 21 Oct '20

21 Oct '20

add parameters for answerback-email -e,--email email_address: add email address -s,--ssh-pubkey pub_key_file: add a ssh login pub_key -f,--raw-email email_file: add email address[, pub_key] via email file Usage: ruby answerback-email.rb -e email_address [-s pub_key_file] ruby answerback-email.rb -f email_file why: easier for administrator to manully assign jumper account for user with the parameters Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 179 ++++++++----------- 1 file changed, 77 insertions(+), 102 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index b5283ba..a3108cb 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -3,74 +3,90 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -=begin - -repo_list: - all repos url list from upstream-repos.git - -API: -call graph: -read_mail_content -send_account_request - email_addr - get email address - email_message_id - get email message_id - check_email_available - check email available - pub_key_value - get pub key - account - send apply account request and return account info - build_message - build email message - send_mail - call send_mail to send mail whit build message - -the returned data for account_info like: -{ - "account" => "guest", - "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", - "jumper_port" => "10000" -} -=end +# xx --email xxx --login --ssh-pubkey xxx --raw-email email-file +# samba mount +# ssh logshn (huawei, ) (install pubkey / send password) require 'json' require 'mail' require 'set' -require_relative '../defconfig.rb' +require 'optparse' +require_relative '../defconfig' names = Set.new %w[ - JUMPER_IP + JUMPER_HOST JUMPER_PORT - CRYSTAL_INTRANET - SEND_MAIL_PORT + SEND_MAIL_HOST + SEND_INTERNET_MAIL_PORT ] defaults = relevant_defaults(names) -JUMPER_IP = defaults['JUMPER_IP'] -JUMPER_PORT = defaults['JUMPER_PORT'] -CRYSTAL_INTRANET = defaults['CRYSTAL_INTRANET'] -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] +JUMPER_HOST = defaults['JUMPER_HOST'] || '183.134.196.212' +JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || localhost +SEND_MAIL_PORT = defaults['SEND_INTERNET_MAIL_PORT'] || 11312 + +apply_info = { + 'my_email' => nil, + 'my_pub_key' => nil +} + +def init_info(email_file, apply_info) + mail_content = Mail.read(email_file) + + my_email = mail_content.from[0] + my_pub_key = mail_content.part[1].body.decoded.gsub(/\r|\n/, '') if mail_content.part[1].filename == 'id_rsa.pub' + + apply_info['my_email'] = my_email + apply_info['my_pub_key'] = my_pub_key + + apply_info +end + +options = OptionParser.new do |opts| + opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key_file] [--raw-email email_file]\n" + opts.banner += " -e or -f is required\n" + opts.banner += ' -s is optional when use -e' + + opts.separator '' + opts.separator 'options:' + + opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + apply_info['my_email'] = email_address + end + + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', 'ssh pub_key file, enable keyless login') do |pub_key_file| + apply_info['my_pub_key'] = File.read(pub_key_file) + end + + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + init_info(email_file, apply_info) + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end -def build_message(email, message_id, infos) +options.parse!(ARGV) + +def build_message(email, acct_infos) message = <<~EMAIL_MESSAGE To: #{email} - Message-ID: #{message_id} Subject: jumper account is ready - Dear #{email} + Dear user: Thank you for joining us. You can use the following command to login the jumper server: login command: - ssh -p #{infos['jumper_port']} #{infos['account']}@#{infos['jumper_ip']} + ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} - account passwd: - account_password: #{infos['passwd']} + account password: + #{acct_infos['passwd']} regards compass-ci @@ -79,67 +95,26 @@ def build_message(email, message_id, infos) return message end -def email_addr(mail_content) - msg = 'not an applying account email' - - raise msg unless mail_content.subject =~ /apply ssh account/i - - email = mail_content.from.join(',') - - return email -end - -# def check_email_available(mail_content, email) -# oos_list = File.read('/c/upstream-repos/repo_list').split(/\n/) -# url = mail_content.body.decoded.split(/\n/).find { |line| line =~ /https?:\/\// } -# base_url = url.split('/')[0,5].join('/') -# message = 'The url is not in upstream repo_list' -# -# raise message unless oos_list.include? base_url -# -# url_fdback = %x(curl #{url}) -# email_index = url_fdback.index email -# -# message = 'No commit info found from the url for the email' -# raise message unless email_index -# end - -def email_message_id(mail_content) - message_id = mail_content.message_id - return message_id -end - -def pub_key_value(mail_content) - pub_key = mail_content.body.decoded.split(/\n/).find { |line| line =~ /ssh-rsa/ } - return pub_key -end - def account_info(pub_key) - account_info_str = %x(curl -XGET '#{JUMPER_IP}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - account_info = JSON.parse account_info_str - - return account_info + account_info_str = if pub_key.nil? + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') + else + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + end + JSON.parse account_info_str end -def send_account(mail_content) - email = email_addr(mail_content) - message_id = email_message_id(mail_content) - # check_email_available(mail_content, email) +def send_account(apply_info) + message = "No email address specified\n" + message += "use -e email_address add a email address\n" + message += 'or use -f to add a email file' + raise message if apply_info['my_email'].nil? - pub_key = pub_key_value(mail_content) - acct_info = account_info(pub_key) - - message = build_message(email, message_id, acct_info) - - %x(curl -XPOST '#{CRYSTAL_INTRANET}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") -end + acct_info = account_info(apply_info['my_pub_key']) -def read_mail_content(mail_file) - mail_content = Mail.read(mail_file) + message = build_message(apply_info['my_email'], acct_info) - return mail_content + %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -mail_file = ARGV[0] -mail_content = read_mail_content(mail_file) -send_account(mail_content) +send_account(apply_info) -- 2.23.0

3 9

[PATCH lkp-tests] lib/monitor: add stop_query as rule to stop monitor
by Wu Zhende 21 Oct '20

21 Oct '20

[Why] In some cases,the query for exiting the monitor may not be the same as the query for monitoring. To meet this requirement, an additional zone stop_query is added for processing. Example: query = {"job_id" => 1}, means I want to monitoring the job 1. stop_query = {"job_state" => "extract_finished"}, means I want to stop monitoring when I get a log, "job_state" equals to "extract_finished". Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index baf172d3..fcf7e220 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -11,7 +11,7 @@ require 'eventmachine' require 'json' class Monitor - attr_accessor :monitor_url, :query, :overrides, :action, :job, :result + attr_accessor :monitor_url, :query, :overrides, :action, :job, :result, :stop_query def initialize(monitor_url = '', query = {}, action = {}) @monitor_url = monitor_url @@ -26,6 +26,7 @@ class Monitor @defaults = {} load_default @result = [] + @stop_query = {} end def load_default @@ -100,6 +101,13 @@ class Monitor EM.stop end + def stop(data, web_socket) + @stop_query.each do |key, value| + return false unless data[key] == value + end + stop_em(web_socket) + end + def run(timeout = nil) merge_overrides field_check @@ -128,7 +136,7 @@ class Monitor output(data) connect(data, ws) - stop_em(ws) if @action['stop'] + stop(data, ws) if @action['stop'] end ws.on :close do |event| -- 2.23.0

1 0

[PATCH lkp-tests 2/2] spec/submit: add monitors to spec results
by Wei Jihui 21 Oct '20

21 Oct '20

according to spec results, add monitors args. spec cmd: rspec spec/submit_spec.rb Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- spec/submit/ftq/ftq-1000-2-100000ss-add.yaml | 3 +- spec/submit/ftq/ftq-10000-2-100000ss-add.yaml | 3 +- spec/submit/ftq/ftq-20-2-6000ss-cache.yaml | 3 +- ...-68000000-1-65535-set_get-never-never.yaml | 29 ++++++++++++++++++- 4 files changed, 34 insertions(+), 4 deletions(-) diff --git a/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml index 3849bb30..3f26d1ae 100644 --- a/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml +++ b/spec/submit/ftq/ftq-1000-2-100000ss-add.yaml @@ -7,9 +7,9 @@ samples: 100000ss ftq: test: add freq: 1000 +lab: spec_lab testbox: vm-2p8g--spec_submit tbox_group: vm-2p8g--spec_submit -lab: spec_lab provider: qemu template: kvm nr_node: 1 @@ -26,3 +26,4 @@ pp: freq: 1000 test: add cpufreq_governor: +monitors: {} diff --git a/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml index 2d9785e4..87945a00 100644 --- a/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml +++ b/spec/submit/ftq/ftq-10000-2-100000ss-add.yaml @@ -7,9 +7,9 @@ samples: 100000ss ftq: test: add freq: 10000 +lab: spec_lab testbox: vm-2p8g--spec_submit tbox_group: vm-2p8g--spec_submit -lab: spec_lab provider: qemu template: kvm nr_node: 1 @@ -26,3 +26,4 @@ pp: freq: 10000 test: add cpufreq_governor: +monitors: {} diff --git a/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml index 6c915e93..247cdc37 100644 --- a/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml +++ b/spec/submit/ftq/ftq-20-2-6000ss-cache.yaml @@ -7,9 +7,9 @@ samples: 6000ss ftq: test: cache freq: 20 +lab: spec_lab testbox: vm-2p8g--spec_submit tbox_group: vm-2p8g--spec_submit -lab: spec_lab provider: qemu template: kvm nr_node: 1 @@ -26,3 +26,4 @@ pp: freq: 20 test: cache cpufreq_governor: +monitors: {} diff --git a/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 43132b8f..466a2054 100644 --- a/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -25,10 +25,10 @@ if role client: requests: 68000000 n_pipeline: 3 key_len: 68000000 +lab: spec_lab testbox: vm-2p8g--spec_submit tbox_group: vm-2p8g--spec_submit node_roles: server client -lab: spec_lab provider: qemu template: kvm nr_node: 1 @@ -86,3 +86,30 @@ pp: nr_processes: 4 cpu_node_bind: even cpufreq_governor: +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: -- 2.23.0

1 0

[PATCH lkp-tests 1/2] job: add monitors to job
by Wei Jihui 21 Oct '20

21 Oct '20

find all monitors and add to job scheduler will use it to get monitors program's cgz. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 7 +++++++ sbin/submit | 1 + 2 files changed, 8 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index caa76c4b..e2416eb8 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -564,6 +564,13 @@ class Job return option_value end + def add_monitors + @job['monitors'] = {} + for_each_in(@job, monitor_params) do |_pk, _h, m_n, _m_args| + @job['monitors'][m_n] = nil + end + end + def add_pp() @job["pp"] = Hash.new() init_program_options() diff --git a/sbin/submit b/sbin/submit index a749ea20..8b46db32 100755 --- a/sbin/submit +++ b/sbin/submit @@ -108,6 +108,7 @@ ARGV.each do |jobfile| raise 'Please configure SCHED_PORT' unless job['SCHED_PORT'] job.add_pp + job.add_monitors job.add_define_files if opt_auto_define_files # get job shell function -- 2.23.0

1 0

[PATCH v6 compass-ci] container/assign-account: answerback-email.rb
by Luan Shengde 21 Oct '20

21 Oct '20

add parameters for answerback-email easier for administrator to manully assign jumper account for user with the parameters -e,--email email_address: add email address -s,--ssh-pubkey pub_key_file: add a ssh login pub_key -f,--raw-email email_file: add email address[, pub_key] via email file Usage: ruby answerback-email.rb -e email_address [-s pub_key_file] ruby answerback-email.rb -f email_file Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 178 ++++++++----------- 1 file changed, 76 insertions(+), 102 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index b5283ba..bb8e809 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -3,74 +3,89 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -=begin - -repo_list: - all repos url list from upstream-repos.git - -API: -call graph: -read_mail_content -send_account_request - email_addr - get email address - email_message_id - get email message_id - check_email_available - check email available - pub_key_value - get pub key - account - send apply account request and return account info - build_message - build email message - send_mail - call send_mail to send mail whit build message - -the returned data for account_info like: -{ - "account" => "guest", - "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", - "jumper_port" => "10000" -} -=end +# xx --email xxx --login --ssh-pubkey xxx --raw-email email-file +# samba mount +# ssh logshn (huawei, ) (install pubkey / send password) require 'json' require 'mail' require 'set' -require_relative '../defconfig.rb' +require 'optparse' +require_relative '../defconfig' names = Set.new %w[ - JUMPER_IP + JUMPER_HOST JUMPER_PORT - CRYSTAL_INTRANET - SEND_MAIL_PORT + SEND_MAIL_HOST_INTERNET + SEND_MAIL_PORT_INTERNET ] defaults = relevant_defaults(names) -JUMPER_IP = defaults['JUMPER_IP'] -JUMPER_PORT = defaults['JUMPER_PORT'] -CRYSTAL_INTRANET = defaults['CRYSTAL_INTRANET'] -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] +JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 + +$apply_info = { + 'my_email' => nil, + 'my_ssh_pubkey' => nil +} + +def init_info(email_file) + mail_content = Mail.read(email_file) + + $apply_info['my_email'] = mail_content.from[0] + $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded.gsub(/\r|\n/, '') + end + + $apply_info +end + +options = OptionParser.new do |opts| + opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key_file] [--raw-email email_file]\n" + opts.banner += " -e or -f is required\n" + opts.banner += ' -s is optional when use -e' + + opts.separator '' + opts.separator 'options:' + + opts.on('-e|--email email_address', 'appoint email address') do |email_address| + $apply_info['my_email'] = email_address + end + + opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| + $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + end + + opts.on('-f|--raw-email email_file', 'email file') do |email_file| + init_info(email_file) + end + + opts.on_tail('-h|--help', 'show this message') do + puts opts + exit + end +end + +options.parse!(ARGV) -def build_message(email, message_id, infos) +def build_message(email, acct_infos) message = <<~EMAIL_MESSAGE To: #{email} - Message-ID: #{message_id} Subject: jumper account is ready - Dear #{email} + Dear user: Thank you for joining us. You can use the following command to login the jumper server: login command: - ssh -p #{infos['jumper_port']} #{infos['account']}@#{infos['jumper_ip']} + ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} - account passwd: - account_password: #{infos['passwd']} + account password: + #{acct_infos['passwd']} regards compass-ci @@ -79,67 +94,26 @@ def build_message(email, message_id, infos) return message end -def email_addr(mail_content) - msg = 'not an applying account email' - - raise msg unless mail_content.subject =~ /apply ssh account/i - - email = mail_content.from.join(',') - - return email -end - -# def check_email_available(mail_content, email) -# oos_list = File.read('/c/upstream-repos/repo_list').split(/\n/) -# url = mail_content.body.decoded.split(/\n/).find { |line| line =~ /https?:\/\// } -# base_url = url.split('/')[0,5].join('/') -# message = 'The url is not in upstream repo_list' -# -# raise message unless oos_list.include? base_url -# -# url_fdback = %x(curl #{url}) -# email_index = url_fdback.index email -# -# message = 'No commit info found from the url for the email' -# raise message unless email_index -# end - -def email_message_id(mail_content) - message_id = mail_content.message_id - return message_id -end - -def pub_key_value(mail_content) - pub_key = mail_content.body.decoded.split(/\n/).find { |line| line =~ /ssh-rsa/ } - return pub_key -end - def account_info(pub_key) - account_info_str = %x(curl -XGET '#{JUMPER_IP}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - account_info = JSON.parse account_info_str - - return account_info + account_info_str = if pub_key.nil? + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') + else + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + end + JSON.parse account_info_str end -def send_account(mail_content) - email = email_addr(mail_content) - message_id = email_message_id(mail_content) - # check_email_available(mail_content, email) +def send_account + message = "No email address specified\n" + message += "use -e email_address add a email address\n" + message += 'or use -f to add a email file' + raise message if $apply_info['my_email'].nil? - pub_key = pub_key_value(mail_content) - acct_info = account_info(pub_key) - - message = build_message(email, message_id, acct_info) - - %x(curl -XPOST '#{CRYSTAL_INTRANET}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") -end + acct_info = account_info($apply_info['my_ssh_pubkey']) -def read_mail_content(mail_file) - mail_content = Mail.read(mail_file) + message = build_message($apply_info['my_email'], acct_info) - return mail_content + %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -mail_file = ARGV[0] -mail_content = read_mail_content(mail_file) -send_account(mail_content) +send_account -- 2.23.0

1 0

[PATCH compass-ci] sched:add monitors depends to job
by Cao Xueliang 21 Oct '20

21 Oct '20

[why] It may add monitors field in job yaml, sched will parse the monitors field and add depends to the initrd string. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 12d3980..00b5e68 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -396,12 +396,42 @@ class Job initrd_pkg_arr = Array(String).new initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup + + get_monitors_initrd(mount_type, initrd_http_prefix, initrd_deps_arr, initrd_pkg_arr) + get_pp_initrd(mount_type, initrd_http_prefix, initrd_deps_arr, initrd_pkg_arr) + + self["initrd_deps"] = initrd_deps_arr.uniq.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq.join(" ") + end + + private def get_monitors_initrd(mount_type, initrd_http_prefix, initrd_deps_arr, initrd_pkg_arr) + if @hash["monitors"]? + program_params = @hash["monitors"].as_h + program_params.keys.each do |program| + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + + if File.exists?("#{user_lkp_src}/monitors/#{program}") + if File.exists?(deps_dest_file) + initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) + end + + if File.exists?(pkg_dest_file) + initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) + end + end + end + end + end + + private def get_pp_initrd(mount_type, initrd_http_prefix, initrd_deps_arr, initrd_pkg_arr) if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| if program =~ /^(.*)-\d+$/ program = $1 end + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -414,9 +444,6 @@ class Job end end end - - self["initrd_deps"] = initrd_deps_arr.uniq.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.uniq.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

2 3

[PATCH lkp-tests 2/2] feat(iso2rootfs): add step: submit rootfs test job after rootfs is made
by Yu Chuan 21 Oct '20

21 Oct '20

[Why] The produced dailybuild rootfs needs further testing by submit a job. [How] 1. Configure lkp env by {LKP_SRC}/sbin/install-dependencies.sh 2. Generate global compass-ci yaml. - The path of yaml: - /etc/compass-ci/defaults/sparrow.yaml - The yaml have two necessary fields: - SCHED_HOST - SCHED_PORT - The value of these two fields are consistent with the value in job.yaml of iso2rootfs. 3. Modify job.yaml of rootfs test job. - Add the required fields of job.yaml of rootfs test job to job.yaml of iso2rootfs; - os - os_arch - os_version - os_mount - testbox - Replace/insert them before submit the rootfs test job. 4. Submit the job.yaml of rootfs test job. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/iso2rootfs.yaml | 6 ++++ tests/iso2rootfs | 72 +++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 77 insertions(+), 1 deletion(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index f413c9cee1f0..644c64be06b4 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -11,3 +11,9 @@ iso2rootfs: rootfs_path: initrd_http_host: initrd_http_port: + test_yaml: + test_os: + test_os_arch: + test_os_version: + test_os_mount: + test_testbox: diff --git a/tests/iso2rootfs b/tests/iso2rootfs index bf1f995e78df..5f56f5c21db9 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -53,6 +53,12 @@ check_yaml_vars() "rootfs_path" "initrd_http_host" "initrd_http_port" + "test_yaml" + "test_os" + "test_os_arch" + "test_os_version" + "test_os_mount" + "test_testbox" ) local yaml_t @@ -404,6 +410,69 @@ test_rootfs() EOF } +############ submit test job yaml ############ +generate_global_yaml() +{ + local config_yaml="/etc/compass-ci/defaults/sparrow.yaml" + + mkdir -p "$(dirname "${config_yaml}")" + echo "SCHED_HOST: ${SCHED_HOST}" >> "${config_yaml}" + echo "SCHED_PORT: ${SCHED_PORT}" >> "${config_yaml}" +} + +assign_var() +{ + local var_name=$1 + local var_value=$2 + + if grep -qw "${var_name}:" "${test_yaml}" + then + sed -i "s/^$ *${var_name}:$.*/\1 ${var_value}/g" "${test_yaml}" + else + echo "${var_name}: ${var_value}" >> "${test_yaml}" + fi +} + +modify_test_job_yaml() +{ + log_info "starting prepare ${test_yaml}" + + [ -f "${test_yaml}" ] || die "test yaml$${test_yaml}$ exist in {LKP_SRC}/jobs !!!" + + local wait_add_vars=( + "os" + "os_arch" + "os_version" + "os_mount" + "testbox" + ) + for vt in "${wait_add_vars[@]}" + do + assign_var "$vt" "$(eval echo "\$test_${vt}")" + done +} + +submit_test_job_yaml() +{ + log_info "starting submit ${test_yaml} ..." + + [ -d "${LKP_SRC}" ] || die "no LKP_SRC !!!" + + cd "${LKP_SRC}" && { + chmod +x ./sbin/install-dependencies.sh + ./sbin/install-dependencies.sh + + generate_global_yaml + } + + cd "${LKP_SRC}/jobs" && { + modify_test_job_yaml + "${LKP_SRC}/sbin/submit" "${test_yaml}" + } + + log_info "submit ${test_yaml} finished" +} + ############ post works ############ post_works() { @@ -414,7 +483,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "$(basename ${ROOTFS_DES_DIR})" "${soft_link}" + ln -s "$(basename "${ROOTFS_DES_DIR}")" "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" @@ -433,6 +502,7 @@ main() run_qcow2rootfs test_rootfs + submit_test_job_yaml post_works } -- 2.23.0

4 5

[PATCH compass-ci 1/2] src/lib: find depends_initrd for multitask
by Li Ping 21 Oct '20

21 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the code need multi-qemu, not multi-qemu-x, and scheduler can not find the depends_initrd multi-qemu-x.cgz. Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 12d3980..f0a9007 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,7 +399,7 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| - if program =~ /^(.*)-\d+$/ + if program =~ /(.*)-\d+$/ program = $1 end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" -- 2.23.0

1 1