- Compass-ci - mailweb.openeuler.org

[PATCH v7 lkp-tests 1/2] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 07 Nov '20

07 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..53f1ac46 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write, read, rand_rw block_size: - 64k -- 2.23.0

2 1

[PATCH v7 lkp-tests 2/2] tests/iozone: change the way of parsing parameter
by Lu Kaiyi 07 Nov '20

07 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] change the way of parsing parameter. small example for parsing parameter: test=" write, read, rand_rw" OLD_IFS="$IFS" IFS="," array=($test) IFS="$OLD_IFS" for ele in ${array[@]} do case $ele in "write") echo $ele ;; "read") echo $ele ;; "rand_rw") echo $ele ;; esac done output: ~/workspace/shell_learning% ./test.sh write read rand_rw if use "${array[@]}" in for loop, there will be no output. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 66 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..b6a1e206 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,52 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + # reset IFS to split $test by "," and then restore default + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write") args+=" -i 0" + ;; + "read") args+=" -i 1" + ;; + "rand_rw") args+=" -i 2" + ;; + "backwards") args+=" -i 3" + ;; + "record") args+=" -i 4" + ;; + "stride") args+=" -i 5" + ;; + "fwrite") args+=" -i 6" + ;; + "fread") args+=" -i 7" + ;; + "rand_mix") args+=" -i 8" + ;; + "pwrite") args+=" -i 9" + ;; + "pread") args+=" -i 10" + ;; + "pwritev") args+=" -i 11" + ;; + "preadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

2 1

[PATCH v2 compass-ci 07/10] container/mail-robot: Dockerfile
by Luan Shengde 07 Nov '20

07 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..4b660c1 --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER shdluan(a)163.com + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

2 4

[PATCH v2 compass-ci 10/10] doc/mail-robot.md
by Luan Shengde 07 Nov '20

07 Nov '20

documentation for mail-robot service Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 76 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..1e65d39 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,76 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + my oss commit: commit_url +attahcment + ssh pub_key + - id_rsa.pub + +## steps overview + +1. monitor + - the monitor will listen to the mailbox for new email files + handle new email file: + check to send account + email subject: apply account + +2. assign account + - check email + check commit url + check whether there is a standard commit url + check whether the repo's url in upstream-repos + check whether the commit url valid + if the hub name is gitee.com + # authentication is needed qwhen use curl to check the url + clone the repo and check the commit + if the hub name is not gitee.com + check the commit with curl + + check pub_key + check whether there is a ssh pub_key in the email file + + it will return failure email if any check end with failure + + - generate uuid + - apply account + build user_info + - my_email + - my_name + - my_uuid + - lab + - pub_key + execute command to apply account on jumper and return the account info + curl -XGET "${JUMPER_HOST}:${JUMPER_PORT}/assign_account" -d '${user_info}' + write my_email, my_name, my_uuid, lab to the defaule config file: + + - store apply infos + apply_infos: + - my_email + - my_name + - my_commit_url + - my_login_name + - my_uuid + invoke es to store the apply infos + + - send uuid + build send uuid email message with the apply_infos + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

2 2

[PATCH v7 compass-ci 3/3] assign-account: optimize answerback-email.rb
by Luan Shengde 07 Nov '20

07 Nov '20

1. disable use golbal variable apply_info [why] rubocop: Style/GlobalVars: Do not introduce global variables. [how] use local variables instead of global variables 2. fix OptionParser argument [why] when write this: opts.on('-e|--email email_address', 'appoint email address') will lead to could not identify the options ./answerback-email.rb --email xxx(a)163.com ./answerback-email.rb:72:in `<main>': invalid option: --email (OptionParser::InvalidOption) [how] write seprately like: opts.on('-e email_address', '--email email_address', 'appoint email address') 3. add my info when execute apply account command [why] when applying account, the assign-account service need to write the my info to user's default config file ~/.config/compass-ci/defaults/account.yaml my_info: - my_email - my_name - my_uuid [how] transfer the user info along with the pub_key when sending apply account request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 94 +++++++++++--------- 1 file changed, 54 insertions(+), 40 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..686a327 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,58 +12,63 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +my_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, my_info) mail_content = Mail.read(email_file) - - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + my_info['my_email'] = mail_content.from[0] + my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end end options = OptionParser.new do |opts| - opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key_file] [--raw-email email_file]\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" opts.banner += " -e or -f is required\n" opts.banner += ' -s is optional when use -e' opts.separator '' opts.separator 'options:' - opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + my_info['my_email'] = email_address + # when apply account with email address, will get no user name + my_info['my_name'] = '' end - opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ + 'ssh pub_key file, enable password-less login') do |pub_key_file| + my_info['my_ssh_pubkey'] = File.read(pub_key_file) end - opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + init_info(email_file, my_info) end - opts.on_tail('-h|--help', 'show this message') do + opts.on_tail('-h', '--help', 'show this message') do puts opts exit end @@ -71,21 +76,24 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: Thank you for joining us. You can use the following command to login the jumper server: - login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} - account password: - #{acct_infos['passwd']} + Suggest: + If you use the password to login, change it in time. regards compass-ci @@ -94,26 +102,32 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(my_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(my_info) message = "No email address specified\n" - message += "use -e email_address add a email address\n" + message += "use -e to add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? + raise message if my_info['my_email'].nil? - acct_info = account_info($apply_info['my_ssh_pubkey']) - - message = build_message($apply_info['my_email'], acct_info) + account_info = apply_account(my_info) + # for manually assign account, there will be no my_commit_url + # but the key my_commit_url is required for es + my_info['my_commit_url'] = '' + my_info['my_login_name'] = account_info['my_login_name'] + my_info.delete 'my_ssh_pubkey' + store_account_info(my_info) + message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(my_info) -- 2.23.0

1 0

[PATCH v7 compass-ci 2/3] assign-account: add config default yaml for user
by Luan Shengde 07 Nov '20

07 Nov '20

when assign account, config the default yaml for user [why] easier for user to config the default yaml file [how] write my info to the default yaml file ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 90 ++++++++++++-------- 1 file changed, 53 insertions(+), 37 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..239f709 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -7,17 +7,17 @@ ACCOUNT_DIR dir layout: tree -├── assigned-users -│ ├── user1 -│ ├── user2 -│ ├── user3 -│ ├── ... -├── available-users -│ ├── user11 -│ ├── user12 -│ ├── user13 -│ ├── ... -└── jumper-info +|-- assigned-users +| |-- user1 +| |-- user2 +| |-- user3 +| |-- ... +|-- available-users +| |-- user11 +| |-- user12 +| |-- user13 +| |-- ... +|-- jumper-info assigned-users: store assigned user files available-users: store available user files @@ -28,11 +28,12 @@ jumper-info: store jumper server ip:port for the service API: call graph: -setup_jumper_account_info +assign_new_account read_account_info - build_account_name + build_account_info read_jumper_info - setup_authorized_key + config_default_yaml + config_authorized_key the returned data for setup_jumper_account_info like: { @@ -44,6 +45,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -61,12 +64,12 @@ class AccountStorage message = 'no more available users' raise message if files.empty? - account_info = build_account_name(available_dir, files) + account_info = build_account_info(available_dir, files) return account_info end - def build_account_name(available_dir, files) + def build_account_info(available_dir, files) files.sort account_info = [] account_info.push files[0] @@ -90,38 +93,51 @@ class AccountStorage return jumper_info end - def setup_jumper_account_info + def assign_new_account account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end - jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + + new_account_info = { + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) - return jumper_account_info + config_authorized_key(login_name, pub_key) unless pub_key.nil? + config_default_yaml(login_name) + + return new_account_info + end + + def config_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def config_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

1 0

[PATCH v7 compass-ci 1/3] assign-account: add check my info
by Luan Shengde 07 Nov '20

07 Nov '20

check required my info when assigning account [why] my info is required when assigning account used for config the default yaml file [how] add check my info before assign new account my info: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..560f7cc 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,10 +17,19 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + assign_jumper_account(data) +end + +def assign_jumper_account(data) + lacked_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lacked_info.join(', ')}." + raise error_message unless lacked_info.empty? + ref_account_info = AccountStorage.new(data) - account_info = ref_account_info.setup_jumper_account_info + account_info = ref_account_info.assign_new_account return account_info.to_json end -- 2.23.0

1 0

[PATCH v6 compass-ci 1/3] assign-account: add check for assign account
by Luan Shengde 07 Nov '20

07 Nov '20

check required my info when assigning account [why] my info is required when assigning the account and config the default yaml file [how] add check for my info before assign account my info: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..834f666 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,17 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + check_to_assign_account(data) +end + +def check_to_assign_account(data) + lacked_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lacked_info.join(', ')}." + raise error_message unless lacked_info.empty? + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

2 2

[PATCH compass-ci] add-software: change vmlinuz judge path
by Xu Xijian 06 Nov '20

06 Nov '20

[why] vmlinuz file is under $os_dir/boot/ now, so change the judge path Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index f4c6833..089efd2 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -[ -L "$1/vmlinuz" -a -n "$2" ] || { +[ -L "$1/boot/vmlinuz" -a -n "$2" ] || { cat << 'EOF' Example usage: 1> ./run /os/centos/aarch64/7.6.1810 pack1 pack2 ... -- 2.23.0

1 0

[PATCH v2 compass-ci] jobfile_operate.cr: simplify if block
by Xu Xijian 06 Nov '20

06 Nov '20

[why] Somewhere there is only one line of code in if block, simply them with format "$cmd if $condition" Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/jobfile_operate.cr | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 68818ed..4ec9f96 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -56,11 +56,9 @@ module Jobfile::Operate end def self.parse_one(script_lines, key, val) - if valid_shell_variable?(key) - if val.as_h? - return false - end + return false if val.as_h? + if valid_shell_variable?(key) value = if val.as_a? shell_escape(val.as_a) else @@ -133,9 +131,7 @@ module Jobfile::Operate end # if the create job cpio failed, what to do? - if idd.match(/ERROR/) - puts idd - end + puts idd if idd.match(/ERROR/) # create result dir and copy job.sh, job.yaml and job.cgz to result dir src_dir = File.dirname(temp_yaml) dst_dir = File.join("/srv", job_content["result_root"].to_s) @@ -204,9 +200,7 @@ module Jobfile::Operate def self.auto_submit_job(job_file, override_parameter, other_parameters = nil) cmd = "#{ENV["LKP_SRC"]}/sbin/submit " - if other_parameters - cmd += other_parameters.join(" ") - end + cmd += other_parameters.join(" ") if other_parameters cmd += " -s '#{override_parameter}' #{job_file}" puts `#{cmd}` end -- 2.23.0

1 0