November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci 3/3] container/assign-account: answerback-email.rb
by Luan Shengde 04 Nov '20

04 Nov '20

1. disable use golbal variable apply_info [why]: rubocop: Style/GlobalVars: Do not introduce global variables. [how]: use local variables instead of global variables 2. add transfer user info when execute apply account command [why]: when applying account, the assign-account service need to write the my info to user's default config file ~/.config/compass-ci/defaults/account.yaml my info: - my_email - my_name - my_uuid [how]: transfer the user info along with the pub_key when sending apply account request Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 80 ++++++++++++-------- 1 file changed, 48 insertions(+), 32 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..5e6cc58 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,35 +12,38 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +apply_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, apply_info) mail_content = Mail.read(email_file) + apply_info['my_email'] = mail_content.from[0] + apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + apply_info end options = OptionParser.new do |opts| @@ -52,15 +55,17 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + apply_info['my_email'] = email_address + # when apply account with email address, will get no user name + apply_info['my_name'] = '' end opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + apply_info['my_ssh_pubkey'] = File.read(pub_key_file) end opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + init_info(email_file, apply_info) end opts.on_tail('-h|--help', 'show this message') do @@ -71,10 +76,10 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: @@ -82,10 +87,10 @@ def build_message(email, acct_infos) You can use the following command to login the jumper server: login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} account password: - #{acct_infos['passwd']} + #{account_info['my_password']} regards compass-ci @@ -94,26 +99,37 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(apply_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(apply_info) message = "No email address specified\n" message += "use -e email_address add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? - - acct_info = account_info($apply_info['my_ssh_pubkey']) - - message = build_message($apply_info['my_email'], acct_info) + raise message if apply_info['my_email'].nil? + + account_info = apply_account(apply_info) + my_info = { + 'my_email' => apply_info['my_email'], + 'my_name' => apply_info['my_name'], + # there is no need to add a commit url for this tool + # the es has the key: my_commit_url + 'my_commit_url' => '', + 'my_login_name' => account_info['my_login_name'], + 'my_uuid' => apply_info['my_uuid'] + } + + store_account_info(my_info) + message = build_message(apply_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(apply_info) -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] container/assign-account: get_account_info
by Luan Shengde 04 Nov '20

04 Nov '20

add function: config default yaml file [why]: easier for user to config the default yaml file [how]: write received user infos to the defaults yaml file: ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/get_account_info.rb | 51 +++++++++++++------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..b2d2474 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,35 +95,48 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) + setup_authorized_key(login_name, pub_key) unless pub_key.nil? + setup_default_yaml(login_name) + return jumper_account_info end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def setup_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + end + + def setup_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] container/assign-account: assign-account.rb
by Luan Shengde 04 Nov '20

04 Nov '20

disable assigning account for user if there is no: - my_email - my_name - my_uuid [why]: my_email, my_name, my_uuid is required when initialize the default config file [how]: check if the parsed data has keys: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/assign-account.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..af10524 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,17 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + check_to_assign_account(data) +end + +def check_to_assign_account(data) + lack_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lack_info.join(', ')}." + raise error_message unless lack_info.empty? + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

1 0

[PATCH v3 lkp-tests 3/3] tests/iozone: modify the way of parsing parameter
by Lu Kaiyi 04 Nov '20

04 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] modify the way of parsing parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 65 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 37 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..b85bc4a9 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,51 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write_rewrite") args+=" -i 0" + ;; + "read_reread") args+=" -i 1" + ;; + "random_read_write") args+=" -i 2" + ;; + "read_backwards") args+=" -i 3" + ;; + "rewrite_record") args+=" -i 4" + ;; + "stride_read") args+=" -i 5" + ;; + "fwrite_refwrite") args+=" -i 6" + ;; + "fread_refread") args+=" -i 7" + ;; + "random_mix") args+=" -i 8" + ;; + "pwrite_repwrite") args+=" -i 9" + ;; + "pread_repread") args+=" -i 10" + ;; + "pwritev_repwritev") args+=" -i 11" + ;; + "preadv_repreadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

2 1

[PATCH v2 lab-z9] lab-z9/host: fix taishan200-2280-2s48p-256g--a95 to taishan200-2280-2s48p-512g--a95
by Zhang Yale 04 Nov '20

04 Nov '20

[Why] The memory of taishan200-2280-2s48p-256g--a95 is 512g, so need to fix 256g to 512g. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- ...an200-2280-2s48p-256g--a95 => taishan200-2280-2s48p-512g--a95} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename hosts/{taishan200-2280-2s48p-256g--a95 => taishan200-2280-2s48p-512g--a95} (100%) diff --git a/hosts/taishan200-2280-2s48p-256g--a95 b/hosts/taishan200-2280-2s48p-512g--a95 similarity index 100% rename from hosts/taishan200-2280-2s48p-256g--a95 rename to hosts/taishan200-2280-2s48p-512g--a95 -- 2.23.0

1 0

Re: [PATCH v3 compass-ci 3/3] container/assign-account: answerback-email.rb
by Zhang Yuhang 04 Nov '20

04 Nov '20

Comment message not clear to specify patch info. Reply-To: In-Reply-To: <20201103075716.1022070-3-shdluan(a)163.com> On Tue, Nov 03, 2020 at 03:57:16PM +0800, Luan Shengde wrote: >1. disable use golbal variable apply_info spell mistake: global > [why] > Style/GlobalVars: Do not introduce global variables. > [how] > use local variables instead of global variables > >2. enable transfer user info when execute apply account command > [why] > when applying account, the assign-account service will write the > my info to user's default config file > ~/.config/compass-ci/defaults/account.yaml > my info: > - my_email > - my_name > - my_uuid > [how] > transfer the user info along with the pub_key when sending apply > account request. Suggest to split it into two patch. >Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> >--- > container/assign-account/answerback-email.rb | 80 ++++++++++++-------- > 1 file changed, 48 insertions(+), 32 deletions(-) > >diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb >index bb8e809..5e6cc58 100755 >--- a/container/assign-account/answerback-email.rb >+++ b/container/assign-account/answerback-email.rb >@@ -12,35 +12,38 @@ require 'mail' > require 'set' > require 'optparse' > require_relative '../defconfig' >+require_relative '../../lib/es_client' > > names = Set.new %w[ > JUMPER_HOST > JUMPER_PORT >- SEND_MAIL_HOST_INTERNET >- SEND_MAIL_PORT_INTERNET >+ SEND_MAIL_HOST >+ SEND_MAIL_PORT > ] > > defaults = relevant_defaults(names) > > JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' > JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 >-SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' >-SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 >+SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' >+SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 > >-$apply_info = { >+apply_info = { > 'my_email' => nil, >+ 'my_name' => nil, >+ 'my_uuid' => %x(uuidgen).chomp, > 'my_ssh_pubkey' => nil > } > >-def init_info(email_file) >+def init_info(email_file, apply_info) > mail_content = Mail.read(email_file) >+ apply_info['my_email'] = mail_content.from[0] >+ apply_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') >+ apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' >+ mail_content.part[1].body.decoded >+ end > >- $apply_info['my_email'] = mail_content.from[0] >- $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' >- mail_content.part[1].body.decoded.gsub(/\r|\n/, '') >- end >- >- $apply_info >+ apply_info > end > > options = OptionParser.new do |opts| >@@ -52,15 +55,17 @@ options = OptionParser.new do |opts| > opts.separator 'options:' > > opts.on('-e|--email email_address', 'appoint email address') do |email_address| >- $apply_info['my_email'] = email_address >+ apply_info['my_email'] = email_address >+ # when apply account with email address, will get no user name >+ apply_info['my_name'] = '' > end > > opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| >- $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) >+ apply_info['my_ssh_pubkey'] = File.read(pub_key_file) > end > > opts.on('-f|--raw-email email_file', 'email file') do |email_file| >- init_info(email_file) >+ init_info(email_file, apply_info) > end > > opts.on_tail('-h|--help', 'show this message') do >@@ -71,10 +76,10 @@ end > > options.parse!(ARGV) > >-def build_message(email, acct_infos) >+def build_message(email, account_info) > message = <<~EMAIL_MESSAGE > To: #{email} >- Subject: jumper account is ready >+ Subject: [compass-ci] jumper account is ready > > Dear user: > >@@ -82,10 +87,10 @@ def build_message(email, acct_infos) > You can use the following command to login the jumper server: > > login command: >- ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} >+ ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} > > account password: >- #{acct_infos['passwd']} >+ #{account_info['my_password']} > > regards > compass-ci >@@ -94,26 +99,37 @@ def build_message(email, acct_infos) > return message > end > >-def account_info(pub_key) >- account_info_str = if pub_key.nil? >- %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') >- else >- %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") >- end >+def apply_account(apply_info) >+ account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') > JSON.parse account_info_str > end > >-def send_account >+def send_account(apply_info) The function name so wired. > message = "No email address specified\n" > message += "use -e email_address add a email address\n" > message += 'or use -f to add a email file' >- raise message if $apply_info['my_email'].nil? >- >- acct_info = account_info($apply_info['my_ssh_pubkey']) >- >- message = build_message($apply_info['my_email'], acct_info) >+ raise message if apply_info['my_email'].nil? Could we do like this? raise message unless apply_info['my_email'] Thanks, Zhang Yuhang >+ >+ account_info = apply_account(apply_info) >+ my_info = { >+ 'my_email' => apply_info['my_email'], >+ 'my_name' => apply_info['my_name'], >+ # there is no need to add a commit url for this tool >+ # the es has the key: my_commit_url >+ 'my_commit_url' => '', >+ 'my_login_name' => account_info['my_login_name'], >+ 'my_uuid' => apply_info['my_uuid'] >+ } >+ >+ store_account_info(my_info) >+ message = build_message(apply_info['my_email'], account_info) > > %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") > end > >-send_account >+def store_account_info(my_info) >+ es = ESClient.new(index: 'accounts') >+ es.put_source_by_id(my_info['my_email'], my_info) >+end >+ >+send_account(apply_info) >-- >2.23.0 >

1 0

[PATCH lab-z9] lab-z9/hosts: rename taishan200-2280-2s48p-256g--a95 to taishan200-2280-2s48p-512g--a95
by Zhang Yale 04 Nov '20

04 Nov '20

[Why] The memory of taishan200-2280-2s48p-256g--a95 is 512g, so need to fix the host name to 512g. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- hosts/taishan200-2280-2s48p-512g--a95 | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-512g--a95 diff --git a/hosts/taishan200-2280-2s48p-512g--a95 b/hosts/taishan200-2280-2s48p-512g--a95 new file mode 100644 index 0000000..213d10f --- /dev/null +++ b/hosts/taishan200-2280-2s48p-512g--a95 @@ -0,0 +1,20 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 512G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039988792e31 + - /dev/disk/by-id/scsi-35000039988793705 + - /dev/disk/by-id/scsi-35000039988792aa5 + - /dev/disk/by-id/scsi-350000399887932fd +mac_addr: + - cc:05:77:fd:fb:61 + - cc:05:77:fd:fb:62 + - cc:05:77:fd:fb:63 + - cc:05:77:fd:fb:64 + - cc:05:77:fd:fb:65 + - cc:05:77:fd:fb:66 + - cc:05:77:fd:fb:67 + - cc:05:77:fd:fb:68 +model_name: Kunpeng-920 -- 2.23.0

1 1

[PATCH compass-ci 1/2] compare values by each metrics
by Lu Weitao 04 Nov '20

04 Nov '20

compare values by each metrics based on groups matrices, and format compare result as echart data_set background: For support compare with user-defined template feature, the work-flow of user-defined template feature: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: compare_values by each metrics ---> format/show results Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 107 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 078028a..7e6dfb1 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -399,8 +399,115 @@ def compare_group_matrices(group_matrices, options) result_str end +# compare group_matrices with each dimension(matrix) of each group, +# and reorganize the result as group-metric-compare-values +# +# Input: groups_matrices +# { +# group_key_1 => { +# dimension_1 => matrix_1, (openeuler 20.03) +# dimension_2 => matrix_2, (openeuler 20.09) +# dimension_3 => matrix_3, (centos 7.6) +# }, +# group_key_2 => {...} +# } +# +# output: compare_metrics_values +# { +# group_key_1 => { +# metric_1 => { +# 'average' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'standard_deviation' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'change' => { +# 'dimension_2 vs dimension_1' => xxx +# 'dimension_3 vs dimension_1' => xxx +# 'dimension_3 vs dimension_2' => xxx +# } +# }, +# metric_2 => {...} +# } +# } +def compare_metrics_values(groups_matrices) + metrics_compare_values = {} + groups_matrices.each do |group_key, dimensions| + metrics_compare_values[group_key] = get_metric_values(dimensions) + end + metrics_compare_values +end + +def get_metric_values(dimensions) + metrics_values = {} + dimensions.each do |dim, matrix| + matrix.each do |metric, values| + metrics_values[metric] ||= {} + metrics_values[metric]['average'] ||= {} + metrics_values[metric]['standard_deviation'] ||= {} + metric_value = get_values(values, true) + metrics_values[metric]['average'][dim] = metric_value[:average] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] + end + end + get_metric_change(metrics_values) + metrics_values +end + +def get_metric_change(metrics_values) + metrics_values.each do |metric, values| + metrics_values[metric]['change'] = {} + next if values['average'].size < 2 + + dimension_list = values['average'].keys + change_titles = get_change_titles(dimension_list) + change_titles.each do |pk| + change = get_compare_value(values['average'][pk[0]], values['average'][pk[1]], true) + values['change'] = { "#{pk[1]} vs #{pk[0]}" => change } + end + end +end + +# create all dimension combinations as change_title +# input: dimension_list +# eg: ['openeuler 20.03', 'debian sid', 'centos 7.6'] +# return: Array(base_dimension: String, pk_dimension: String) +# [ +# ['openeuler 20.03', 'debian sid'], +# ['openeuler 20.03', 'centos 7.6'], +# ['debian sid', 'centos 7.6'] +# ] +def get_change_titles(dimension_list) + dims = [] + dimension_list_size = dimension_list.size + (1..dimension_list_size - 1).each do |i| + (i..dimension_list_size - 1).each do |j| + dims << [dimension_list[i - 1], dimension_list[j]] + end + end + dims +end + +def show_chart_compare_result(chart_result) + print JSON.pretty_generate(chart_result) +end + # Format Fields +def format_for_chart(metrics_compare_values, template_params) + chart_result = {} + chart_result['title'] = template_params['title'] || nil + chart_result['unit'] = template_params['unit'] || nil + chart_result['x_name'] = template_params['x_params'].join('|') || nil + chart_result['tables'] = metrics_compare_values + chart_result +end + def format_fails_runs(fails, runs) fails_width = (SUB_LONG_COLUMN_WIDTH * FAILS_PROPORTION).to_i runs_width = SUB_LONG_COLUMN_WIDTH - fails_width - 1 -- 2.23.0

2 7

[PATCH v3 lkp-tests 2/3] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 04 Nov '20

04 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..f9ab305f 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write_rewrite,read_reread,random_read_write block_size: - 64k -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/3] jobs/iozone-bs.yaml: fix mapper_parsing_exception
by Lu Kaiyi 04 Nov '20

04 Nov '20

[why] when key and value of fs: xfs in same line for iozone-bs.yaml, submit job will cause some error occurred like below: submit /home/lukaiyi/lkp-tests/jobs/iozone-bs.yaml failed, got job_id=z9.144590, error: [{"type" => "mapper_parsing_exception", "reason" => "failed to parse [pp.fs.fs]"}] [how] change key and value of fs: xfs to different line, same to iosched: kyber Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index 868979fc..e2cd9f48 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -14,6 +14,11 @@ block_size: - 16m disk: 1HDD -fs: xfs -iosched: kyber + +fs: +- xfs + +iosched: +- kyber + iozone: -- 2.23.0

1 0