- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 3/3] answerback-email: rename key my_uuid with my_token
by Luan Shengde 14 Dec '20

14 Dec '20

key my_uuid is renamed to my_token. when applying/updating account, will use my_token instead to do this change with no sense for old user(update account) keep key my_uuid and add key my_token for new user just add key my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 21 ++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c08f10e..eb906b3 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -21,6 +21,7 @@ names = Set.new %w[ JUMPER_PORT SEND_MAIL_HOST SEND_MAIL_PORT + lab ] defaults = relevant_defaults(names) @@ -29,12 +30,14 @@ JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = defaults['lab'] +HOST_SERVER = 'crystal.ci' my_info = { 'my_email' => nil, 'my_name' => nil, 'my_commit_url' => nil, - 'my_uuid' => nil, + 'my_token' => nil, 'my_login_name' => nil, 'my_ssh_pubkey' => [] } @@ -179,6 +182,7 @@ def apply_account(my_info, conf_info) apply_info = {} apply_info.update my_info apply_info.update conf_info + apply_info['lab'] = LAB account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str @@ -200,9 +204,7 @@ def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) new_stdin_pubkey = stdin_info.delete 'new_ssh_pubkey' new_pubkey = new_stdin_pubkey || new_email_pubkey - my_info.update my_info_es unless my_info_es.empty? - my_info.update email_info unless email_info.empty? - my_info.update stdin_info unless stdin_info.empty? + update_my_info(my_info, my_info_es, email_info, stdin_info) return if new_pubkey.nil? return if my_info['my_ssh_pubkey'].include? new_pubkey @@ -210,6 +212,13 @@ def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) my_info['my_ssh_pubkey'].insert(0, new_pubkey) end +def update_my_info(my_info, my_info_es, email_info, stdin_info) + my_info['my_token'] = my_info_es['my_uuid'] if my_info['my_token'].nil? + my_info.update my_info_es unless my_info_es.empty? + my_info.update email_info unless email_info.empty? + my_info.update stdin_info unless stdin_info.empty? +end + def build_my_info_from_account_info(my_info, account_info, conf_info) unless account_info['my_jumper_pubkey'].nil? return if my_info['my_ssh_pubkey'][-1] == account_info['my_jumper_pubkey'] @@ -221,7 +230,7 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) end def check_server - return true if ENV['HOSTNAME'] == 'z9' + return true if ENV['HOSTNAME'] == HOST_SERVER message = 'please run the tool on z9 server' puts message @@ -260,7 +269,7 @@ def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) return unless check_my_email(my_info) return unless check_email_assigned_account(conf_info, my_info_es) - my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] + my_info['my_token'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) return unless check_my_name_exist(my_info) -- 2.23.0

2 2

[PATCH v2 lkp-tests] tests/build-pkg: fix git clone error
by Wang Yong 12 Dec '20

12 Dec '20

[before] '"${var}"' [after] '${var}' " will be treated as a character, it's redundant --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index aeaf17787..518298228 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -107,7 +107,7 @@ replace_source() do echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { - url="\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" } } url=$(echo "$url" | sed 's|https://|http://|g') -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix git clone error
by Wang Yong 12 Dec '20

12 Dec '20

[before] '${var}' [after] '"${var}"' character " is redundant --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index aeaf17787..518298228 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -107,7 +107,7 @@ replace_source() do echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { - url="\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" } } url=$(echo "$url" | sed 's|https://|http://|g') -- 2.23.0

2 2

[PATCH v3 compass-ci 3/3] lib/compare_data_format.rb: limit the number of table columns in a single row
by Lu Kaiyi 12 Dec '20

12 Dec '20

[example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - meminfo.Active - meminfo.Memused filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 4k|1G | 4k|10G | 4k|80G | 4k|100G | 16k|1G | 16k|10G | 16k|100G | 32k|1G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1051645.34 | 704958.00 | 706798.00 | 657669.27 | 1030484.85 | 979289.00 | 948576.00 | 1262591.17 | | standard_deviation openeuler 20.03 | 74.00 | 0.00 | 0.00 | 8.00 | 46.00 | 0.00 | 2.00 | 42.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 64k|1G | 64k|100G | 128k|1G | 128k|100G | 256k|1G | 256k|100G | 512k|1G | 512k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1418731.53 | 1854231.33 | 2004321.76 | 2960171.75 | 2979326.50 | 4775340.00 | 5003555.53 | 5630845.75 | | standard_deviation openeuler 20.03 | 47.00 | 10.00 | 65.00 | 18.00 | 88.00 | 8.00 | 108.00 | 15.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 1024k|1G | 1024k|100G | 2048k|100G | 4096k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 8974190.91 | 9641039.00 | 5480763.67 | 7302509.67 | | standard_deviation openeuler 20.03 | 118.00 | 36.00 | 86.00 | 67.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 42 ++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 9 deletions(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index c1f7023..0226a65 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -72,19 +72,20 @@ end # format compare template results into a table format # class FormatTableData - def initialize(result_hash) + def initialize(result_hash, row_size=8) @title = result_hash['title'] @tables = result_hash['tables'] @unit = result_hash['unit'] @x_name = result_hash['x_name'] + @row_size = row_size end def show_table @tables.each do |table_title, table| @tb = Terminal::Table.new set_table_title - set_field_names(table_title, table) - add_rows(table) + row_num = get_row_num(table) + split_data_column(table_title, table, row_num) set_align_column print_table end @@ -94,14 +95,37 @@ class FormatTableData @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" end - def set_field_names(table_title, table) + def get_row_num(table) + data_column_size = table['average']['source'][0].size + row_num = data_column_size / @row_size + row_rem = data_column_size % @row_size + if row_rem > 0 + row_num += 1 + end + row_num + end + + def split_data_column(table_title, table, row_num) + row_num.times do |row| + starts = 1 + row * @row_size + ends = starts + @row_size + set_field_names(table_title, table, starts, ends) + add_rows(table, starts, ends) + break if row == row_num - 1 + + @tb.add_separator + @tb.add_separator + end + end + + def set_field_names(table_title, table, starts, ends) field_names = [table_title] - field_names.concat(table['average']['source'][0]) + field_names.concat(table['average']['source'][0][starts-1...ends-1]) @tb.add_row(field_names) @tb.add_separator end - def add_rows(table) + def add_rows(table, starts, ends) row_names = %w[average standard_deviation change] max_size = row_names.map(&:size).max row_names.each do |row_name| @@ -109,15 +133,15 @@ class FormatTableData dimensions_size = table[row_name]['dimensions'].size (1...dimensions_size).each do |index| - add_row(table, row_name, index, max_size) + add_row(table, row_name, index, max_size, starts, ends) end end end - def add_row(table, row_name, index, max_size) + def add_row(table, row_name, index, max_size, starts, ends) row = table[row_name]['source'][index] row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') - format_data_row = row[1..-1] + format_data_row = row[starts...ends] if row_name == 'change' format_data_row.map! { |data| format('%.1f%%', data) } else -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] add options for compare_by_template
by Lu Kaiyi 12 Dec '20

12 Dec '20

if want json format results, use command like below: compare -t compare_template.yaml --theme json otherwise, print table format results by default: compare -t compare_template.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 11 ++++++++--- sbin/compare | 2 +- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index f34e239..db264f8 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -96,11 +96,11 @@ end # unit: KB/s # -def compare_by_template(template) +def compare_by_template(template, options) template_params = load_template(template) groups_matrices = create_groups_matrices(template_params) compare_results = compare_metrics_values(groups_matrices) - show_compare_result(compare_results, template_params) + show_compare_result(compare_results, template_params, options) end def load_template(template) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 6a60f2b..0e08ec7 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -507,11 +507,16 @@ def get_dimensions_combination(dimension_list) dims end -def show_compare_result(metrics_compare_results, template_params) +def show_compare_result(metrics_compare_results, template_params, options) + theme = options[:theme] formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - table_results = FormatTableData.new(echart_results) - table_results.show_table + if theme == 'json' + print JSON.pretty_generate(echart_results) + else + table_results = FormatTableData.new(echart_results) + table_results.show_table + end end # Format Fields diff --git a/sbin/compare b/sbin/compare index c13ec1a..e8d1ac4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -67,7 +67,7 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful if template - compare_by_template(template) + compare_by_template(template, options) elsif is_group compare_group(argv, dimensions, options) else -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] lib: friendly show compare template result in command line
by Lu Kaiyi 12 Dec '20

12 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the results of "compare -t <compare_template.yaml>" with terminal-table in Ruby. [example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | average openeuler 20.03 | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | standard_deviation openeuler 20.03 | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | average centos 7.6 | 345243.03 | 142698.79 | 62108.35 | 34747.73 | 26330.19 | 10317.85 | 7471.71 | 3558.30 | | average openeuler 20.03 | 122003.54 | 33528.53 | 31469.06 | 13870.14 | 8249.71 | 4329.45 | 1976.54 | 1141.00 | | standard_deviation centos 7.6 | 97.00 | 95.00 | 122.00 | 125.00 | 100.00 | 130.00 | 101.00 | 103.00 | | standard_deviation openeuler 20.03 | 174.00 | 188.00 | 171.00 | 197.00 | 181.00 | 175.00 | 170.00 | 176.00 | | change centos 7.6 vs openeuler 20.03 | 183.0% | 325.6% | 97.4% | 150.5% | 219.2% | 138.3% | 278.0% | 211.9% | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 73 ++++++++++++++++++++++++++++++++++++++ lib/compare_matrixes.rb | 3 +- 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index bf4013b..c1f7023 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -2,6 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'terminal-table' + # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # @@ -66,6 +68,77 @@ class FormatEchartData end end +# ---------------------------------------------------------------------------------------------------- +# format compare template results into a table format +# +class FormatTableData + def initialize(result_hash) + @title = result_hash['title'] + @tables = result_hash['tables'] + @unit = result_hash['unit'] + @x_name = result_hash['x_name'] + end + + def show_table + @tables.each do |table_title, table| + @tb = Terminal::Table.new + set_table_title + set_field_names(table_title, table) + add_rows(table) + set_align_column + print_table + end + end + + def set_table_title + @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" + end + + def set_field_names(table_title, table) + field_names = [table_title] + field_names.concat(table['average']['source'][0]) + @tb.add_row(field_names) + @tb.add_separator + end + + def add_rows(table) + row_names = %w[average standard_deviation change] + max_size = row_names.map(&:size).max + row_names.each do |row_name| + next unless table[row_name] + + dimensions_size = table[row_name]['dimensions'].size + (1...dimensions_size).each do |index| + add_row(table, row_name, index, max_size) + end + end + end + + def add_row(table, row_name, index, max_size) + row = table[row_name]['source'][index] + row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') + format_data_row = row[1..-1] + if row_name == 'change' + format_data_row.map! { |data| format('%.1f%%', data) } + else + format_data_row.map! { |data| format('%.2f', data) } + end + @tb.add_row([row_title, *format_data_row]) + end + + def set_align_column + @tb.number_of_columns.times do |index| + @tb.align_column(index + 1, :right) + end + @tb.align_column(0, :left) + end + + def print_table + puts @tb + puts + end +end + # input: x_params_list # eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] # output: diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 5293989..6a60f2b 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -510,7 +510,8 @@ end def show_compare_result(metrics_compare_results, template_params) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - print JSON.pretty_generate(echart_results) + table_results = FormatTableData.new(echart_results) + table_results.show_table end # Format Fields -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] assign-account: rename my_uuid to my_token
by Luan Shengde 12 Dec '20

12 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index c0753b0..4ed95c0 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -24,7 +24,7 @@ get '/assign_account' do end def assign_jumper_account(data) - lacked_info = %w[my_email my_name my_uuid] - data.keys + lacked_info = %w[my_email my_name my_token] - data.keys error_message = "lack of my infos: #{lacked_info.join(', ')}." raise error_message unless lacked_info.empty? -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] assign-account: add config lab yaml file
by Luan Shengde 12 Dec '20

12 Dec '20

[why] key my_uuid is renamed to my_token. key my_token will be stored in file: ~/.config/compass-ci/include/lab/${lab}.yaml for this change with no sense: for old user(update account): keep my_uuid in the default yaml add my_token to ~/.config/compass-ci/include/lab/${lab}.yaml for new user: add my_email, my_name to the default yaml add my_token to ~/.config/compass-ci/include/lab/${lab}.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 44 ++++++++++++++++---- 1 file changed, 35 insertions(+), 9 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 95bec14..7bc5377 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -110,7 +110,7 @@ class AccountStorage jumper_host, jumper_port = read_jumper_info config_authorized_key(login_name) - config_default_yaml(login_name) + config_yaml(login_name) permit_login_config(login_name) if @data.key?('gen_sshkey') && @data['gen_sshkey'] my_jumper_pubkey = generate_ssh_key(login_name) @@ -127,6 +127,14 @@ class AccountStorage return jumper_account_info end + def config_yaml(login_name) + default_yaml = touch_default_yaml(login_name) + config_default_yaml(login_name, default_yaml) + + lab_yaml = touch_lab_yaml(login_name) + config_lab_yaml(login_name, lab_yaml) + end + def permit_login_config(login_name) if @data.key?('enable_login') && @data['enable_login'] %x(usermod -s /usr/bin/zsh #{login_name}) @@ -158,19 +166,37 @@ class AccountStorage default_yaml end - def config_default_yaml(login_name) - default_yaml = touch_default_yaml(login_name) - + def config_default_yaml(login_name, default_yaml) account_yaml = YAML.load_file(default_yaml) || {} - # my_email, my_name, my_uuid is required to config default yaml file + # my_email, my_name is required to config default yaml file # they are added along with 'my_ssh_pubkey' when sending assign account request account_yaml['my_email'] = @data['my_email'] account_yaml['my_name'] = @data['my_name'] - account_yaml['my_uuid'] = @data['my_uuid'] - f = File.new(default_yaml, 'w') - f.puts account_yaml.to_yaml - f.close + File.open(default_yaml, 'w') do |f| + f.puts account_yaml.to_yaml + end + + FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") + end + + def touch_lab_yaml(login_name) + lab_yaml_dir = File.join('/home', login_name, '.config/compass-ci/include/lab') + FileUtils.mkdir_p lab_yaml_dir unless File.exist? lab_yaml_dir + + lab_yaml = File.join(lab_yaml_dir, "#{@data['lab']}.yaml") + FileUtils.touch lab_yaml unless File.exist? lab_yaml + lab_yaml + end + + def config_lab_yaml(login_name, lab_yaml) + lab_yaml_info = YAML.load_file(lab_yaml) || {} + lab_yaml_info['my_token'] = @data['my_token'] + + File.open(lab_yaml, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") end -- 2.23.0

1 0

[PATCH compass-ci 2/3] assign-account: rename key
by Luan Shengde 12 Dec '20

12 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index c0753b0..4ed95c0 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -24,7 +24,7 @@ get '/assign_account' do end def assign_jumper_account(data) - lacked_info = %w[my_email my_name my_uuid] - data.keys + lacked_info = %w[my_email my_name my_token] - data.keys error_message = "lack of my infos: #{lacked_info.join(', ')}." raise error_message unless lacked_info.empty? -- 2.23.0

2 2

[PATCH compass-ci 1/3] assign-account: add config lab yaml file
by Luan Shengde 12 Dec '20

12 Dec '20

[why] key my_uuid is renamed to my_token. key my_token will be stored in file: ~/.config/compass-ci/include/lab/${lab}.yaml for this change with no sense: for old user(update account): keep my_uuid in the default yaml add my_token to ~/.config/compass-ci/include/lab/${lab}.yaml for new user: add my_email, my_name to the default yaml add my_token to ~/.config/compass-ci/include/lab/${lab}.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 25 ++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 95bec14..7471c61 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -111,6 +111,7 @@ class AccountStorage config_authorized_key(login_name) config_default_yaml(login_name) + config_lab_yaml(login_name) permit_login_config(login_name) if @data.key?('gen_sshkey') && @data['gen_sshkey'] my_jumper_pubkey = generate_ssh_key(login_name) @@ -162,11 +163,10 @@ class AccountStorage default_yaml = touch_default_yaml(login_name) account_yaml = YAML.load_file(default_yaml) || {} - # my_email, my_name, my_uuid is required to config default yaml file + # my_email, my_name is required to config default yaml file # they are added along with 'my_ssh_pubkey' when sending assign account request account_yaml['my_email'] = @data['my_email'] account_yaml['my_name'] = @data['my_name'] - account_yaml['my_uuid'] = @data['my_uuid'] f = File.new(default_yaml, 'w') f.puts account_yaml.to_yaml @@ -174,6 +174,27 @@ class AccountStorage FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") end + def touch_lab_yaml(login_name) + lab_yaml_dir = File.join('/home', login_name, '.config/compass-ci/include/lab') + FileUtils.mkdir_p lab_yaml_dir unless File.exist? lab_yaml_dir + + lab_yaml = File.join(lab_yaml_dir, "#{@data['lab']}.yaml") + FileUtils.touch lab_yaml unless File.exist? lab_yaml + lab_yaml + end + + def config_lab_yaml(login_name) + lab_yaml = touch_lab_yaml(login_name) + + lab_yaml_info = YAML.load_file(lab_yaml) || {} + lab_yaml_info['my_token'] = @data['my_token'] + + f = File.new(lab_yaml, 'w') + f.puts lab_yaml_info.to_yaml + f.close + FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") + end + def config_authorized_key(login_name) pub_key = @data['my_ssh_pubkey'][0] -- 2.23.0

2 2

[PATCH v2 compass-ci] lib: friendly show compare template result in command line
by Lu Kaiyi 12 Dec '20

12 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the results of "compare -t <compare_template.yaml>" with terminal-table in Ruby. [example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | average openeuler 20.03 | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | standard_deviation openeuler 20.03 | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | average centos 7.6 | 345243.03 | 142698.79 | 62108.35 | 34747.73 | 26330.19 | 10317.85 | 7471.71 | 3558.30 | | average openeuler 20.03 | 122003.54 | 33528.53 | 31469.06 | 13870.14 | 8249.71 | 4329.45 | 1976.54 | 1141.00 | | standard_deviation centos 7.6 | 97.00 | 95.00 | 122.00 | 125.00 | 100.00 | 130.00 | 101.00 | 103.00 | | standard_deviation openeuler 20.03 | 174.00 | 188.00 | 171.00 | 197.00 | 181.00 | 175.00 | 170.00 | 176.00 | | change centos 7.6 vs openeuler 20.03 | 183.0% | 325.6% | 97.4% | 150.5% | 219.2% | 138.3% | 278.0% | 211.9% | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 73 ++++++++++++++++++++++++++++++++++++++ lib/compare_matrixes.rb | 3 +- 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index bf4013b..c1f7023 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -2,6 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'terminal-table' + # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # @@ -66,6 +68,77 @@ class FormatEchartData end end +# ---------------------------------------------------------------------------------------------------- +# format compare template results into a table format +# +class FormatTableData + def initialize(result_hash) + @title = result_hash['title'] + @tables = result_hash['tables'] + @unit = result_hash['unit'] + @x_name = result_hash['x_name'] + end + + def show_table + @tables.each do |table_title, table| + @tb = Terminal::Table.new + set_table_title + set_field_names(table_title, table) + add_rows(table) + set_align_column + print_table + end + end + + def set_table_title + @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" + end + + def set_field_names(table_title, table) + field_names = [table_title] + field_names.concat(table['average']['source'][0]) + @tb.add_row(field_names) + @tb.add_separator + end + + def add_rows(table) + row_names = %w[average standard_deviation change] + max_size = row_names.map(&:size).max + row_names.each do |row_name| + next unless table[row_name] + + dimensions_size = table[row_name]['dimensions'].size + (1...dimensions_size).each do |index| + add_row(table, row_name, index, max_size) + end + end + end + + def add_row(table, row_name, index, max_size) + row = table[row_name]['source'][index] + row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') + format_data_row = row[1..-1] + if row_name == 'change' + format_data_row.map! { |data| format('%.1f%%', data) } + else + format_data_row.map! { |data| format('%.2f', data) } + end + @tb.add_row([row_title, *format_data_row]) + end + + def set_align_column + @tb.number_of_columns.times do |index| + @tb.align_column(index + 1, :right) + end + @tb.align_column(0, :left) + end + + def print_table + puts @tb + puts + end +end + # input: x_params_list # eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] # output: diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 5293989..6a60f2b 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -510,7 +510,8 @@ end def show_compare_result(metrics_compare_results, template_params) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - print JSON.pretty_generate(echart_results) + table_results = FormatTableData.new(echart_results) + table_results.show_table end # Format Fields -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] mail-robot: update email message
by Luan Shengde 12 Dec '20

12 Dec '20

update email message with new key: my_uuid => my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index a88626c..834e96f 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -27,7 +27,7 @@ def build_apply_account_email(my_info) EOF mkdir -p ~/.config/compass-ci/include/lab cat >> ~/.config/compass-ci/include/lab/#{ENV['lab']}.yaml <<-EOF - my_uuid: #{my_info['my_uuid']} + my_token: #{my_info['my_token']} EOF 2) download lkp-tests and dependencies -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] mail-robot: use key my_token instead of my_uuid
by Luan Shengde 12 Dec '20

12 Dec '20

[why] key my_uuid is renamed to my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 40 +++++++++++++++-------- 1 file changed, 26 insertions(+), 14 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 0fff3b5..1fd3574 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -30,14 +30,14 @@ SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 # my_info: # - my_email # - my_name -# - my_uuid +# - my_token # my_ssh_pubkey # store_account_info # call ESClient to store my_info # my_info: # - my_email # - my_name -# - my_uuid +# - my_token # - my_commit_url # - my_login_name # - my_ssh_pubkey @@ -87,21 +87,37 @@ class ApplyAccount accountx_es.query_by_id(@my_info['my_email']) end + def build_apply_info(apply_info, my_account_es) + my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') + apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? + apply_info.update my_account_es + apply_info.update @my_info + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + @my_info.update apply_info + apply_info['is_update_account'] = true + apply_info + end + def apply_my_account my_account_es = read_my_account_es apply_info = {} if my_account_es - my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') - apply_info.update my_account_es - apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq - @my_info.update apply_info - apply_info['is_update_account'] = true + build_apply_info(apply_info, my_account_es) + # my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') + # my_token will take the place of my_uuid. + # to make the change with no sense, will keep the my_uuid for now + # apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? + # apply_info.update my_account_es + # apply_info.update @my_info + # apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + # @my_info.update apply_info + # apply_info['is_update_account'] = true else - my_uuid = %x(uuidgen).chomp - @my_info['my_uuid'] = my_uuid + my_token = %x(uuidgen).chomp + @my_info['my_token'] = my_token apply_info.update @my_info end + apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end @@ -112,10 +128,6 @@ class ApplyAccount @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es end - def check_account_es - ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) - end - def store_account_info es = ESClient.new(index: 'accounts') es.put_source_by_id(@my_info['my_email'], @my_info) -- 2.23.0

1 0

[PATCH compass-ci] delimiter: rename job_id to bad_job_id
by Cao Xueliang 12 Dec '20

12 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 4 ++-- sbin/git-bisect | 6 +++--- src/delimiter/delimiter.rb | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index b54be4d..189edb8 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -34,7 +34,7 @@ class GitBisect def set_ids puts "task content: #{@task}" @error_id = @task['error_id'] - @bad_job_id = @task['job_id'] + @bad_job_id = @task['bad_job_id'] end def set_bad_job @@ -95,7 +95,7 @@ class GitBisect error_messages = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, - 'job_id' => @bad_job_id, 'error_messages' => error_messages.join("\n")] + 'error_messages' => error_messages.join("\n")] end # first search the good commit in db diff --git a/sbin/git-bisect b/sbin/git-bisect index 86f79d4..f9da6de 100755 --- a/sbin/git-bisect +++ b/sbin/git-bisect @@ -21,10 +21,10 @@ def parse_argv end op = OptionParser.new do |opts| - opts.banner = 'Usage: git-bisect job_id=$job_id, error_id=$error_id' + opts.banner = 'Usage: git-bisect bad_job_id=$bad_job_id, error_id=$error_id' opts.separator '' - opts.separator 'job_id: you submit a job to compass-ci and the job has some errors' + opts.separator 'bad_job_id: you submit a job to compass-ci and the job has some errors' opts.separator 'error_id: you want git bisect a error from the errors that you got' opts.separator '' @@ -36,7 +36,7 @@ end op.parse!(ARGV) items = parse_argv -unless items['job_id'] || items['error_id'] +unless items['bad_job_id'] || items['error_id'] system 'git-bisect -h' exit end diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index 3dcb3de..c1922e2 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -25,7 +25,7 @@ class Delimiter next end - %x(#{LKP_SRC}/sbin/submit job_id=#{task['job_id']} error_id=#{task['error_id']}) + %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id']} bisect.yaml) rescue StandardError => e puts e sleep(60) -- 2.23.0

1 1

[PATCH v1 compass-ci] delimiter: rename job_id to bad_job_id
by Cao Xueliang 12 Dec '20

12 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 4 ++-- sbin/git-bisect | 6 +++--- src/delimiter/delimiter.rb | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index b54be4d..189edb8 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -34,7 +34,7 @@ class GitBisect def set_ids puts "task content: #{@task}" @error_id = @task['error_id'] - @bad_job_id = @task['job_id'] + @bad_job_id = @task['bad_job_id'] end def set_bad_job @@ -95,7 +95,7 @@ class GitBisect error_messages = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, - 'job_id' => @bad_job_id, 'error_messages' => error_messages.join("\n")] + 'error_messages' => error_messages.join("\n")] end # first search the good commit in db diff --git a/sbin/git-bisect b/sbin/git-bisect index 86f79d4..f9da6de 100755 --- a/sbin/git-bisect +++ b/sbin/git-bisect @@ -21,10 +21,10 @@ def parse_argv end op = OptionParser.new do |opts| - opts.banner = 'Usage: git-bisect job_id=$job_id, error_id=$error_id' + opts.banner = 'Usage: git-bisect bad_job_id=$bad_job_id, error_id=$error_id' opts.separator '' - opts.separator 'job_id: you submit a job to compass-ci and the job has some errors' + opts.separator 'bad_job_id: you submit a job to compass-ci and the job has some errors' opts.separator 'error_id: you want git bisect a error from the errors that you got' opts.separator '' @@ -36,7 +36,7 @@ end op.parse!(ARGV) items = parse_argv -unless items['job_id'] || items['error_id'] +unless items['bad_job_id'] || items['error_id'] system 'git-bisect -h' exit end diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index 3dcb3de..c1922e2 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -25,7 +25,7 @@ class Delimiter next end - %x(#{LKP_SRC}/sbin/submit job_id=#{task['job_id']} error_id=#{task['error_id']}) + %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id']} bisect.yaml) rescue StandardError => e puts e sleep(60) -- 2.23.0

1 0

[PATCH compass-ci] scheduler: prints logs in Sched class
by Wu Zhende 12 Dec '20

12 Dec '20

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 24 ++++++++-- src/scheduler/close_job.cr | 7 ++- src/scheduler/download_file.cr | 14 ++++++ src/scheduler/find_job_boot.cr | 6 ++- src/scheduler/find_next_job_boot.cr | 6 ++- src/scheduler/scheduler.cr | 64 +++++---------------------- src/scheduler/submit_job.cr | 10 +++-- src/scheduler/update_job_parameter.cr | 2 +- 8 files changed, 68 insertions(+), 65 deletions(-) create mode 100644 src/scheduler/download_file.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 6225ac2..10a51ab 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -21,6 +21,7 @@ require "../scheduler/close_job" require "../scheduler/request_cluster_state" require "../scheduler/update_job_parameter" require "../scheduler/create_job_cpio" +require "../scheduler/download_file" class Sched property es @@ -37,6 +38,15 @@ class Sched @log = env.log.as(JSONLogger) end + def debug_message(response) + @log.info(%({"from": "#{(a)env.request.remote_address}", "response": #{response.to_json}})) + end + + def alive(version) + debug_message("Env= {\n#{`export`}}") + "LKP Alive! The time is #{Time.local}, version = #{version}" + end + def normalize_mac(mac : String) mac.gsub(":", "-") end @@ -97,11 +107,19 @@ class Sched # json log hash["testbox"] = testbox - return hash.to_json + @log.info(hash.to_json) end - def report_ssh_port(testbox : String, ssh_port : String) - @redis.hash_set("sched/tbox2ssh_port", testbox, ssh_port) + def report_ssh_port + testbox = @env.params.query["tbox_name"] + ssh_port = @env.params.query["ssh_port"].to_s + job_id = @env.params.query["job_id"].to_s + + if testbox && ssh_port + @redis.hash_set("sched/tbox2ssh_port", testbox, ssh_port) + end + + @log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) end private def query_consumable_keys(shortest_queue_name) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 2c33ffd..579c2c0 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -2,7 +2,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def close_job(job_id : String) + def close_job + job_id = @env.params.query["job_id"]? + return unless job_id + job = @redis.get_job(job_id) delete_access_key_file(job) if job @@ -22,7 +25,7 @@ class Sched @redis.remove_finished_job(job_id) - return %({"job_id": "#{job_id}", "job_state": "complete"}) + @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) end def delete_access_key_file(job : Job) diff --git a/src/scheduler/download_file.cr b/src/scheduler/download_file.cr new file mode 100644 index 0000000..e1f0c9a --- /dev/null +++ b/src/scheduler/download_file.cr @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def download_file + job_id = @env.params.url["job_id"] + job_package = @env.params.url["job_package"] + file_path = ::File.join [Kemal.config.public_folder, job_id, job_package] + + @log.info(%({"job_id": "#{job_id}", "job_state": "download"})) + + send_file @env, file_path + end +end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index fb94e91..5a11523 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -16,7 +16,11 @@ class Sched host = value end - get_job_boot(host, boot_type) + response = get_job_boot(host, boot_type) + job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? + @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id + + response end # auto submit a job to collect the host information diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 09abbd5..0bdfe5b 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -9,6 +9,10 @@ class Sched hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) end - get_job_boot(hostname, "ipxe") + response = get_job_boot(hostname, "ipxe") + job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? + @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id + + response end end diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 4fde6e3..7ec149c 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -41,15 +41,9 @@ module Scheduler env.create_sched end - # for debug (maybe kemal debug|logger does better) - def self.debug_message(env, response) - env.log.info(%({"from": "#{env.request.remote_address}", "response": #{response.to_json}})) - end - # echo alive get "/" do |env| - debug_message(env, "Env= {\n#{`export`}}") - "LKP Alive! The time is #{Time.local}, version = #{VERSION}" + env.sched.alive(VERSION) end # for XXX_runner get job @@ -58,46 +52,24 @@ module Scheduler # /boot.xxx/host/${hostname} # /boot.yyy/mac/${mac} get "/boot.:boot_type/:parameter/:value" do |env| - response = env.sched.find_job_boot - - job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? - env.log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id - - response + env.sched.find_job_boot end # /~lkp/cgi-bin/gpxelinux.cgi?hostname=:hostname&mac=:mac&last_kernel=:last_kernel get "/~lkp/cgi-bin/gpxelinux.cgi" do |env| - response = env.sched.find_next_job_boot - - job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? - env.log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id - - response + env.sched.find_next_job_boot end # enqueue # - echo job_id to caller # -- job_id = "0" ? means failed post "/submit_job" do |env| - job_messages = env.sched.submit_job - - job_messages.each do |job_message| - env.log.info(job_message.to_json) - end - - job_messages.to_json + env.sched.submit_job.to_json end # file download server get "/job_initrd_tmpfs/:job_id/:job_package" do |env| - job_id = env.params.url["job_id"] - job_package = env.params.url["job_package"] - file_path = ::File.join [Kemal.config.public_folder, job_id, job_package] - - env.log.info(%({"job_id": "#{job_id}", "job_state": "download"})) - - send_file env, file_path + env.sched.download_file end # client(runner) report its hostname and mac @@ -155,7 +127,7 @@ module Scheduler # ?job_file=/lkp/scheduled/job.yaml&job_state=post_run&job_id=10 # ?job_file=/lkp/scheduled/job.yaml&loadavg=0.28 0.82 0.49 1/105 3389&start_time=1587725398&end_time=1587725698&job_id=10 get "/~lkp/cgi-bin/lkp-jobfile-append-var" do |env| - env.log.info(env.sched.update_job_parameter) + env.sched.update_job_parameter "Done" end @@ -178,42 +150,26 @@ module Scheduler # response: get "server ip" from cluster state, # return "server=<server ip>". get "/~lkp/cgi-bin/lkp-cluster-sync" do |env| - response = env.sched.request_cluster_state - - debug_message(env, response) - - response + env.sched.request_cluster_state end # client(runner) report job post_run finished # /~lkp/cgi-bin/lkp-post-run?job_file=/lkp/scheduled/job.yaml&job_id=40 # curl "http://localhost:3000/~lkp/cgi-bin/lkp-post-run?job_file=/lkp/scheduled/job…" get "/~lkp/cgi-bin/lkp-post-run" do |env| - # get job_id from request - job_id = env.params.query["job_id"]? - if job_id - env.log.info(env.sched.close_job(job_id)) - end + env.sched.close_job "Done" end get "/~lkp/cgi-bin/lkp-wtmp" do |env| - env.log.info(env.sched.update_tbox_wtmp) + env.sched.update_tbox_wtmp "Done" end get "/~lkp/cgi-bin/report_ssh_port" do |env| - testbox = env.params.query["tbox_name"] - ssh_port = env.params.query["ssh_port"].to_s - job_id = env.params.query["job_id"].to_s - - if testbox && ssh_port - env.sched.report_ssh_port(testbox, ssh_port) - end - - env.log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) + env.sched.report_ssh_port "Done" end diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 3909cb7..4303b1d 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -11,20 +11,24 @@ class Sched cluster_file = job["cluster"] if cluster_file.empty? || cluster_file == "cs-localhost" - return submit_single_job(job) + response = submit_single_job(job) else cluster_config = get_cluster_config(cluster_file, job.lkp_initrd_user, job.os_arch).not_nil! - return submit_cluster_job(job, cluster_config) + response = submit_cluster_job(job, cluster_config) end rescue ex @log.warn(ex.inspect_with_backtrace) - return [{ + response = [{ "job_id" => "0", "message" => ex.to_s, "job_state" => "submit", }] + ensure + response.each do |job_message| + @log.info(job_message.to_json) + end end # return: diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index cee8b9b..da528c1 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -29,6 +29,6 @@ class Sched # json log log = job_content.dup log["job_id"] = log.delete("id").not_nil! - return log.to_json + @log.info(log.to_json) end end -- 2.23.0

1 0

[PATCH compass-ci 2/2] mail-robot: update email message
by Luan Shengde 12 Dec '20

12 Dec '20

update email message with new key: my_uuid => my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index a88626c..834e96f 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -27,7 +27,7 @@ def build_apply_account_email(my_info) EOF mkdir -p ~/.config/compass-ci/include/lab cat >> ~/.config/compass-ci/include/lab/#{ENV['lab']}.yaml <<-EOF - my_uuid: #{my_info['my_uuid']} + my_token: #{my_info['my_token']} EOF 2) download lkp-tests and dependencies -- 2.23.0

1 0

[PATCH compass-ci 1/2] mail-robot: use new key my_token for apply account
by Luan Shengde 12 Dec '20

12 Dec '20

[why] key my_uuid is renamed to my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 0fff3b5..93920f4 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -31,6 +31,7 @@ SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 # - my_email # - my_name # - my_uuid +# - my_token # my_ssh_pubkey # store_account_info # call ESClient to store my_info @@ -38,6 +39,7 @@ SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 # - my_email # - my_name # - my_uuid +# - my_token # - my_commit_url # - my_login_name # - my_ssh_pubkey @@ -62,6 +64,7 @@ class ApplyAccount # in order to successfully send email for failed parsing # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] + @my_info['lab'] = ENV['lab'] parse_mail_content apply_my_account @@ -92,14 +95,17 @@ class ApplyAccount apply_info = {} if my_account_es my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') + # my_token will take the place of my_uuid. + # to make the change with no sense, will keep the my_uuid for now + apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? apply_info.update my_account_es apply_info.update @my_info apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq @my_info.update apply_info apply_info['is_update_account'] = true else - my_uuid = %x(uuidgen).chomp - @my_info['my_uuid'] = my_uuid + my_token = %x(uuidgen).chomp + @my_info['my_token'] = my_token apply_info.update @my_info end apply_new_account(apply_info, my_account_es) @@ -112,10 +118,6 @@ class ApplyAccount @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es end - def check_account_es - ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) - end - def store_account_info es = ESClient.new(index: 'accounts') es.put_source_by_id(@my_info['my_email'], @my_info) -- 2.23.0

1 0

[PATCH compass-ci 3/3] answerback-email: rename key my_uuid with my_token
by Luan Shengde 12 Dec '20

12 Dec '20

key my_uuid is renamed to my_token. when applying/updating account, will use my_token instead for this change with no sense for old user(update account) keep key my_uuid and add key my_token for new user delete my_uuid and just add key my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c08f10e..401bd8c 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -29,12 +29,13 @@ JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = 'z9' my_info = { 'my_email' => nil, 'my_name' => nil, 'my_commit_url' => nil, - 'my_uuid' => nil, + 'my_token' => nil, 'my_login_name' => nil, 'my_ssh_pubkey' => [] } @@ -200,6 +201,7 @@ def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) new_stdin_pubkey = stdin_info.delete 'new_ssh_pubkey' new_pubkey = new_stdin_pubkey || new_email_pubkey + my_info['my_token'] = my_info_es['my_uuid'] if my_info['my_token'].nil? my_info.update my_info_es unless my_info_es.empty? my_info.update email_info unless email_info.empty? my_info.update stdin_info unless stdin_info.empty? @@ -221,7 +223,7 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) end def check_server - return true if ENV['HOSTNAME'] == 'z9' + return true if ENV['HOSTNAME'] == LAB message = 'please run the tool on z9 server' puts message @@ -260,7 +262,7 @@ def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) return unless check_my_email(my_info) return unless check_email_assigned_account(conf_info, my_info_es) - my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] + my_info['my_token'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) return unless check_my_name_exist(my_info) -- 2.23.0

1 0

[PATCH v2 compass-ci] logging: fix time difference of eight hours
by Wu Zhende 12 Dec '20

12 Dec '20

If the time zone information is not contained, the ES saves time as the UTC time by default. As a result, the kibana display time is eight hours later. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 2 +- src/lib/json_logger.cr | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index b3daeb4..7bf2229 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -11,7 +11,7 @@ @type record_transformer enable_ruby <record> - time ${time.strftime('%Y-%m-%dT%H:%M:%S')} + time ${time.strftime('%Y-%m-%dT%H:%M:%S.%3N+0800')} </record> </filter> diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index dd87020..6d61197 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -18,7 +18,7 @@ class JSONLogger < Logger Logger::Formatter.new do | severity, datetime, progname, msg, io| get_env_info(@env.as(HTTP::Server::Context)) if @env level_num = severity.to_i32 - datetime = datetime.to_s("%Y-%m-%dT%H:%M:%S") + datetime = datetime.to_s("%Y-%m-%dT%H:%M:%S.%3N+0800") logger_hash = JSON.parse(%({"level_num": #{level_num}, "level": "#{severity}", "time": "#{datetime}" -- 2.23.0

1 0

[PATCH compass-ci] README.md: modify the link address
by Li Ping 11 Dec '20

11 Dec '20

1 0

[PATCH compass-ci] container/defconfig.rb: split functions
by Liu Yinsi 11 Dec '20

11 Dec '20

split into 2 functions. function 1 -> meminfo_hash: create a more generic function by returning a hash, function 2 -> get_available_memory: create a dedicated function for use by es and logging-es container Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index d353bb3..482dbc8 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -34,10 +34,14 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end +def meminfo_hash + File.readlines('/proc/meminfo').map { |s| k, v, kb = s.split; { k.chomp(':') => v } } +end + def get_available_memory - memory = File.readlines('/proc/meminfo')[0].chomp.split[1].to_f / 1048576 + memtotal = meminfo_hash[0]['MemTotal'].to_f / 1048576 # set container available memory size, minimum size is 1024m, maximum size is 30720m, # take the middle value according to the system memory size. - [1024, 30720, Math.sqrt(memory) * 1024].sort[1].to_i + [1024, 30720, Math.sqrt(memtotal) * 1024].sort[1].to_i end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container/defconfig.rb: limit container memory usage
by Liu Yinsi 11 Dec '20

11 Dec '20

use Math.sqrt(memory) to set container occupied memory according to system memory size, instead of setting fixed size, more reasonable. In addition, '-Xms' not support floating number, so use to_i. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..026a183 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + +def set_available_memory + memory = `awk '($1 == "MemTotal:"){print $2/1048576}' /proc/meminfo`.to_f + + # set container available memory size, minimum size is 1g, maximum size is 30g, + # take the middle value according to the system memory size. + [1, 30, Math.sqrt(memory)].sort[1].to_i +end -- 2.23.0

2 2

[PATCH compass-ci] README.zh.md: modify the link address
by Li Ping 11 Dec '20

11 Dec '20

1 0

[PATCH lkp-tests] bin/lkp-setup-rootfs: modify test-machine is not restarted
by Hu Xuejiao 11 Dec '20

11 Dec '20

[why] When force_reboot is not added to the job.yaml, the test-machine is "no KERNEL found". Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- bin/lkp-setup-rootfs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 0eabea75..cca1b53b 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -61,7 +61,10 @@ job_does_not_need_reboot() job_force_reboot() { - grep -q "^force_reboot: \+1$" $job + grep -q "^force_reboot: 1$" $job && return 0 + grep -q "^do_not_reboot: 1$" $job || return 0 + + return 1 } reboot_for_next_job() -- 2.23.0

1 0

2025

2024

2023

2022

2021

2020

Compass-ci