- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/3] assign-account: add config lab yaml file
by Luan Shengde 12 Dec '20

12 Dec '20

[why] key my_uuid is renamed to my_token. key my_token will be stored in file: ~/.config/compass-ci/include/lab/${lab}.yaml for this change with no sense: for old user(update account): keep my_uuid in the default yaml add my_token to ~/.config/compass-ci/include/lab/${lab}.yaml for new user: add my_email, my_name to the default yaml add my_token to ~/.config/compass-ci/include/lab/${lab}.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 25 ++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 95bec14..7471c61 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -111,6 +111,7 @@ class AccountStorage config_authorized_key(login_name) config_default_yaml(login_name) + config_lab_yaml(login_name) permit_login_config(login_name) if @data.key?('gen_sshkey') && @data['gen_sshkey'] my_jumper_pubkey = generate_ssh_key(login_name) @@ -162,11 +163,10 @@ class AccountStorage default_yaml = touch_default_yaml(login_name) account_yaml = YAML.load_file(default_yaml) || {} - # my_email, my_name, my_uuid is required to config default yaml file + # my_email, my_name is required to config default yaml file # they are added along with 'my_ssh_pubkey' when sending assign account request account_yaml['my_email'] = @data['my_email'] account_yaml['my_name'] = @data['my_name'] - account_yaml['my_uuid'] = @data['my_uuid'] f = File.new(default_yaml, 'w') f.puts account_yaml.to_yaml @@ -174,6 +174,27 @@ class AccountStorage FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") end + def touch_lab_yaml(login_name) + lab_yaml_dir = File.join('/home', login_name, '.config/compass-ci/include/lab') + FileUtils.mkdir_p lab_yaml_dir unless File.exist? lab_yaml_dir + + lab_yaml = File.join(lab_yaml_dir, "#{@data['lab']}.yaml") + FileUtils.touch lab_yaml unless File.exist? lab_yaml + lab_yaml + end + + def config_lab_yaml(login_name) + lab_yaml = touch_lab_yaml(login_name) + + lab_yaml_info = YAML.load_file(lab_yaml) || {} + lab_yaml_info['my_token'] = @data['my_token'] + + f = File.new(lab_yaml, 'w') + f.puts lab_yaml_info.to_yaml + f.close + FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") + end + def config_authorized_key(login_name) pub_key = @data['my_ssh_pubkey'][0] -- 2.23.0

2 2

[PATCH v2 compass-ci] lib: friendly show compare template result in command line
by Lu Kaiyi 12 Dec '20

12 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the results of "compare -t <compare_template.yaml>" with terminal-table in Ruby. [example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | average openeuler 20.03 | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | standard_deviation openeuler 20.03 | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | average centos 7.6 | 345243.03 | 142698.79 | 62108.35 | 34747.73 | 26330.19 | 10317.85 | 7471.71 | 3558.30 | | average openeuler 20.03 | 122003.54 | 33528.53 | 31469.06 | 13870.14 | 8249.71 | 4329.45 | 1976.54 | 1141.00 | | standard_deviation centos 7.6 | 97.00 | 95.00 | 122.00 | 125.00 | 100.00 | 130.00 | 101.00 | 103.00 | | standard_deviation openeuler 20.03 | 174.00 | 188.00 | 171.00 | 197.00 | 181.00 | 175.00 | 170.00 | 176.00 | | change centos 7.6 vs openeuler 20.03 | 183.0% | 325.6% | 97.4% | 150.5% | 219.2% | 138.3% | 278.0% | 211.9% | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 73 ++++++++++++++++++++++++++++++++++++++ lib/compare_matrixes.rb | 3 +- 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index bf4013b..c1f7023 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -2,6 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'terminal-table' + # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # @@ -66,6 +68,77 @@ class FormatEchartData end end +# ---------------------------------------------------------------------------------------------------- +# format compare template results into a table format +# +class FormatTableData + def initialize(result_hash) + @title = result_hash['title'] + @tables = result_hash['tables'] + @unit = result_hash['unit'] + @x_name = result_hash['x_name'] + end + + def show_table + @tables.each do |table_title, table| + @tb = Terminal::Table.new + set_table_title + set_field_names(table_title, table) + add_rows(table) + set_align_column + print_table + end + end + + def set_table_title + @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" + end + + def set_field_names(table_title, table) + field_names = [table_title] + field_names.concat(table['average']['source'][0]) + @tb.add_row(field_names) + @tb.add_separator + end + + def add_rows(table) + row_names = %w[average standard_deviation change] + max_size = row_names.map(&:size).max + row_names.each do |row_name| + next unless table[row_name] + + dimensions_size = table[row_name]['dimensions'].size + (1...dimensions_size).each do |index| + add_row(table, row_name, index, max_size) + end + end + end + + def add_row(table, row_name, index, max_size) + row = table[row_name]['source'][index] + row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') + format_data_row = row[1..-1] + if row_name == 'change' + format_data_row.map! { |data| format('%.1f%%', data) } + else + format_data_row.map! { |data| format('%.2f', data) } + end + @tb.add_row([row_title, *format_data_row]) + end + + def set_align_column + @tb.number_of_columns.times do |index| + @tb.align_column(index + 1, :right) + end + @tb.align_column(0, :left) + end + + def print_table + puts @tb + puts + end +end + # input: x_params_list # eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] # output: diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 5293989..6a60f2b 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -510,7 +510,8 @@ end def show_compare_result(metrics_compare_results, template_params) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - print JSON.pretty_generate(echart_results) + table_results = FormatTableData.new(echart_results) + table_results.show_table end # Format Fields -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] mail-robot: update email message
by Luan Shengde 12 Dec '20

12 Dec '20

update email message with new key: my_uuid => my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index a88626c..834e96f 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -27,7 +27,7 @@ def build_apply_account_email(my_info) EOF mkdir -p ~/.config/compass-ci/include/lab cat >> ~/.config/compass-ci/include/lab/#{ENV['lab']}.yaml <<-EOF - my_uuid: #{my_info['my_uuid']} + my_token: #{my_info['my_token']} EOF 2) download lkp-tests and dependencies -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] mail-robot: use key my_token instead of my_uuid
by Luan Shengde 12 Dec '20

12 Dec '20

[why] key my_uuid is renamed to my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 40 +++++++++++++++-------- 1 file changed, 26 insertions(+), 14 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 0fff3b5..1fd3574 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -30,14 +30,14 @@ SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 # my_info: # - my_email # - my_name -# - my_uuid +# - my_token # my_ssh_pubkey # store_account_info # call ESClient to store my_info # my_info: # - my_email # - my_name -# - my_uuid +# - my_token # - my_commit_url # - my_login_name # - my_ssh_pubkey @@ -87,21 +87,37 @@ class ApplyAccount accountx_es.query_by_id(@my_info['my_email']) end + def build_apply_info(apply_info, my_account_es) + my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') + apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? + apply_info.update my_account_es + apply_info.update @my_info + apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + @my_info.update apply_info + apply_info['is_update_account'] = true + apply_info + end + def apply_my_account my_account_es = read_my_account_es apply_info = {} if my_account_es - my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') - apply_info.update my_account_es - apply_info.update @my_info - apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq - @my_info.update apply_info - apply_info['is_update_account'] = true + build_apply_info(apply_info, my_account_es) + # my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') + # my_token will take the place of my_uuid. + # to make the change with no sense, will keep the my_uuid for now + # apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? + # apply_info.update my_account_es + # apply_info.update @my_info + # apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq + # @my_info.update apply_info + # apply_info['is_update_account'] = true else - my_uuid = %x(uuidgen).chomp - @my_info['my_uuid'] = my_uuid + my_token = %x(uuidgen).chomp + @my_info['my_token'] = my_token apply_info.update @my_info end + apply_info['lab'] = ENV['lab'] apply_new_account(apply_info, my_account_es) end @@ -112,10 +128,6 @@ class ApplyAccount @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es end - def check_account_es - ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) - end - def store_account_info es = ESClient.new(index: 'accounts') es.put_source_by_id(@my_info['my_email'], @my_info) -- 2.23.0

1 0

[PATCH compass-ci] delimiter: rename job_id to bad_job_id
by Cao Xueliang 12 Dec '20

12 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 4 ++-- sbin/git-bisect | 6 +++--- src/delimiter/delimiter.rb | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index b54be4d..189edb8 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -34,7 +34,7 @@ class GitBisect def set_ids puts "task content: #{@task}" @error_id = @task['error_id'] - @bad_job_id = @task['job_id'] + @bad_job_id = @task['bad_job_id'] end def set_bad_job @@ -95,7 +95,7 @@ class GitBisect error_messages = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, - 'job_id' => @bad_job_id, 'error_messages' => error_messages.join("\n")] + 'error_messages' => error_messages.join("\n")] end # first search the good commit in db diff --git a/sbin/git-bisect b/sbin/git-bisect index 86f79d4..f9da6de 100755 --- a/sbin/git-bisect +++ b/sbin/git-bisect @@ -21,10 +21,10 @@ def parse_argv end op = OptionParser.new do |opts| - opts.banner = 'Usage: git-bisect job_id=$job_id, error_id=$error_id' + opts.banner = 'Usage: git-bisect bad_job_id=$bad_job_id, error_id=$error_id' opts.separator '' - opts.separator 'job_id: you submit a job to compass-ci and the job has some errors' + opts.separator 'bad_job_id: you submit a job to compass-ci and the job has some errors' opts.separator 'error_id: you want git bisect a error from the errors that you got' opts.separator '' @@ -36,7 +36,7 @@ end op.parse!(ARGV) items = parse_argv -unless items['job_id'] || items['error_id'] +unless items['bad_job_id'] || items['error_id'] system 'git-bisect -h' exit end diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index 3dcb3de..c1922e2 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -25,7 +25,7 @@ class Delimiter next end - %x(#{LKP_SRC}/sbin/submit job_id=#{task['job_id']} error_id=#{task['error_id']}) + %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id']} bisect.yaml) rescue StandardError => e puts e sleep(60) -- 2.23.0

1 1

[PATCH v1 compass-ci] delimiter: rename job_id to bad_job_id
by Cao Xueliang 12 Dec '20

12 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 4 ++-- sbin/git-bisect | 6 +++--- src/delimiter/delimiter.rb | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index b54be4d..189edb8 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -34,7 +34,7 @@ class GitBisect def set_ids puts "task content: #{@task}" @error_id = @task['error_id'] - @bad_job_id = @task['job_id'] + @bad_job_id = @task['bad_job_id'] end def set_bad_job @@ -95,7 +95,7 @@ class GitBisect error_messages = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, - 'job_id' => @bad_job_id, 'error_messages' => error_messages.join("\n")] + 'error_messages' => error_messages.join("\n")] end # first search the good commit in db diff --git a/sbin/git-bisect b/sbin/git-bisect index 86f79d4..f9da6de 100755 --- a/sbin/git-bisect +++ b/sbin/git-bisect @@ -21,10 +21,10 @@ def parse_argv end op = OptionParser.new do |opts| - opts.banner = 'Usage: git-bisect job_id=$job_id, error_id=$error_id' + opts.banner = 'Usage: git-bisect bad_job_id=$bad_job_id, error_id=$error_id' opts.separator '' - opts.separator 'job_id: you submit a job to compass-ci and the job has some errors' + opts.separator 'bad_job_id: you submit a job to compass-ci and the job has some errors' opts.separator 'error_id: you want git bisect a error from the errors that you got' opts.separator '' @@ -36,7 +36,7 @@ end op.parse!(ARGV) items = parse_argv -unless items['job_id'] || items['error_id'] +unless items['bad_job_id'] || items['error_id'] system 'git-bisect -h' exit end diff --git a/src/delimiter/delimiter.rb b/src/delimiter/delimiter.rb index 3dcb3de..c1922e2 100644 --- a/src/delimiter/delimiter.rb +++ b/src/delimiter/delimiter.rb @@ -25,7 +25,7 @@ class Delimiter next end - %x(#{LKP_SRC}/sbin/submit job_id=#{task['job_id']} error_id=#{task['error_id']}) + %x(#{LKP_SRC}/sbin/submit bad_job_id=#{task['job_id']} error_id=#{task['error_id']} bisect.yaml) rescue StandardError => e puts e sleep(60) -- 2.23.0

1 0

[PATCH compass-ci] scheduler: prints logs in Sched class
by Wu Zhende 12 Dec '20

12 Dec '20

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 24 ++++++++-- src/scheduler/close_job.cr | 7 ++- src/scheduler/download_file.cr | 14 ++++++ src/scheduler/find_job_boot.cr | 6 ++- src/scheduler/find_next_job_boot.cr | 6 ++- src/scheduler/scheduler.cr | 64 +++++---------------------- src/scheduler/submit_job.cr | 10 +++-- src/scheduler/update_job_parameter.cr | 2 +- 8 files changed, 68 insertions(+), 65 deletions(-) create mode 100644 src/scheduler/download_file.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 6225ac2..10a51ab 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -21,6 +21,7 @@ require "../scheduler/close_job" require "../scheduler/request_cluster_state" require "../scheduler/update_job_parameter" require "../scheduler/create_job_cpio" +require "../scheduler/download_file" class Sched property es @@ -37,6 +38,15 @@ class Sched @log = env.log.as(JSONLogger) end + def debug_message(response) + @log.info(%({"from": "#{(a)env.request.remote_address}", "response": #{response.to_json}})) + end + + def alive(version) + debug_message("Env= {\n#{`export`}}") + "LKP Alive! The time is #{Time.local}, version = #{version}" + end + def normalize_mac(mac : String) mac.gsub(":", "-") end @@ -97,11 +107,19 @@ class Sched # json log hash["testbox"] = testbox - return hash.to_json + @log.info(hash.to_json) end - def report_ssh_port(testbox : String, ssh_port : String) - @redis.hash_set("sched/tbox2ssh_port", testbox, ssh_port) + def report_ssh_port + testbox = @env.params.query["tbox_name"] + ssh_port = @env.params.query["ssh_port"].to_s + job_id = @env.params.query["job_id"].to_s + + if testbox && ssh_port + @redis.hash_set("sched/tbox2ssh_port", testbox, ssh_port) + end + + @log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) end private def query_consumable_keys(shortest_queue_name) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 2c33ffd..579c2c0 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -2,7 +2,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def close_job(job_id : String) + def close_job + job_id = @env.params.query["job_id"]? + return unless job_id + job = @redis.get_job(job_id) delete_access_key_file(job) if job @@ -22,7 +25,7 @@ class Sched @redis.remove_finished_job(job_id) - return %({"job_id": "#{job_id}", "job_state": "complete"}) + @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) end def delete_access_key_file(job : Job) diff --git a/src/scheduler/download_file.cr b/src/scheduler/download_file.cr new file mode 100644 index 0000000..e1f0c9a --- /dev/null +++ b/src/scheduler/download_file.cr @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def download_file + job_id = @env.params.url["job_id"] + job_package = @env.params.url["job_package"] + file_path = ::File.join [Kemal.config.public_folder, job_id, job_package] + + @log.info(%({"job_id": "#{job_id}", "job_state": "download"})) + + send_file @env, file_path + end +end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index fb94e91..5a11523 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -16,7 +16,11 @@ class Sched host = value end - get_job_boot(host, boot_type) + response = get_job_boot(host, boot_type) + job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? + @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id + + response end # auto submit a job to collect the host information diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 09abbd5..0bdfe5b 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -9,6 +9,10 @@ class Sched hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) end - get_job_boot(hostname, "ipxe") + response = get_job_boot(hostname, "ipxe") + job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? + @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id + + response end end diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 4fde6e3..7ec149c 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -41,15 +41,9 @@ module Scheduler env.create_sched end - # for debug (maybe kemal debug|logger does better) - def self.debug_message(env, response) - env.log.info(%({"from": "#{env.request.remote_address}", "response": #{response.to_json}})) - end - # echo alive get "/" do |env| - debug_message(env, "Env= {\n#{`export`}}") - "LKP Alive! The time is #{Time.local}, version = #{VERSION}" + env.sched.alive(VERSION) end # for XXX_runner get job @@ -58,46 +52,24 @@ module Scheduler # /boot.xxx/host/${hostname} # /boot.yyy/mac/${mac} get "/boot.:boot_type/:parameter/:value" do |env| - response = env.sched.find_job_boot - - job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? - env.log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id - - response + env.sched.find_job_boot end # /~lkp/cgi-bin/gpxelinux.cgi?hostname=:hostname&mac=:mac&last_kernel=:last_kernel get "/~lkp/cgi-bin/gpxelinux.cgi" do |env| - response = env.sched.find_next_job_boot - - job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? - env.log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id - - response + env.sched.find_next_job_boot end # enqueue # - echo job_id to caller # -- job_id = "0" ? means failed post "/submit_job" do |env| - job_messages = env.sched.submit_job - - job_messages.each do |job_message| - env.log.info(job_message.to_json) - end - - job_messages.to_json + env.sched.submit_job.to_json end # file download server get "/job_initrd_tmpfs/:job_id/:job_package" do |env| - job_id = env.params.url["job_id"] - job_package = env.params.url["job_package"] - file_path = ::File.join [Kemal.config.public_folder, job_id, job_package] - - env.log.info(%({"job_id": "#{job_id}", "job_state": "download"})) - - send_file env, file_path + env.sched.download_file end # client(runner) report its hostname and mac @@ -155,7 +127,7 @@ module Scheduler # ?job_file=/lkp/scheduled/job.yaml&job_state=post_run&job_id=10 # ?job_file=/lkp/scheduled/job.yaml&loadavg=0.28 0.82 0.49 1/105 3389&start_time=1587725398&end_time=1587725698&job_id=10 get "/~lkp/cgi-bin/lkp-jobfile-append-var" do |env| - env.log.info(env.sched.update_job_parameter) + env.sched.update_job_parameter "Done" end @@ -178,42 +150,26 @@ module Scheduler # response: get "server ip" from cluster state, # return "server=<server ip>". get "/~lkp/cgi-bin/lkp-cluster-sync" do |env| - response = env.sched.request_cluster_state - - debug_message(env, response) - - response + env.sched.request_cluster_state end # client(runner) report job post_run finished # /~lkp/cgi-bin/lkp-post-run?job_file=/lkp/scheduled/job.yaml&job_id=40 # curl "http://localhost:3000/~lkp/cgi-bin/lkp-post-run?job_file=/lkp/scheduled/job…" get "/~lkp/cgi-bin/lkp-post-run" do |env| - # get job_id from request - job_id = env.params.query["job_id"]? - if job_id - env.log.info(env.sched.close_job(job_id)) - end + env.sched.close_job "Done" end get "/~lkp/cgi-bin/lkp-wtmp" do |env| - env.log.info(env.sched.update_tbox_wtmp) + env.sched.update_tbox_wtmp "Done" end get "/~lkp/cgi-bin/report_ssh_port" do |env| - testbox = env.params.query["tbox_name"] - ssh_port = env.params.query["ssh_port"].to_s - job_id = env.params.query["job_id"].to_s - - if testbox && ssh_port - env.sched.report_ssh_port(testbox, ssh_port) - end - - env.log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) + env.sched.report_ssh_port "Done" end diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 3909cb7..4303b1d 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -11,20 +11,24 @@ class Sched cluster_file = job["cluster"] if cluster_file.empty? || cluster_file == "cs-localhost" - return submit_single_job(job) + response = submit_single_job(job) else cluster_config = get_cluster_config(cluster_file, job.lkp_initrd_user, job.os_arch).not_nil! - return submit_cluster_job(job, cluster_config) + response = submit_cluster_job(job, cluster_config) end rescue ex @log.warn(ex.inspect_with_backtrace) - return [{ + response = [{ "job_id" => "0", "message" => ex.to_s, "job_state" => "submit", }] + ensure + response.each do |job_message| + @log.info(job_message.to_json) + end end # return: diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index cee8b9b..da528c1 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -29,6 +29,6 @@ class Sched # json log log = job_content.dup log["job_id"] = log.delete("id").not_nil! - return log.to_json + @log.info(log.to_json) end end -- 2.23.0

1 0

[PATCH compass-ci 2/2] mail-robot: update email message
by Luan Shengde 12 Dec '20

12 Dec '20

update email message with new key: my_uuid => my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index a88626c..834e96f 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -27,7 +27,7 @@ def build_apply_account_email(my_info) EOF mkdir -p ~/.config/compass-ci/include/lab cat >> ~/.config/compass-ci/include/lab/#{ENV['lab']}.yaml <<-EOF - my_uuid: #{my_info['my_uuid']} + my_token: #{my_info['my_token']} EOF 2) download lkp-tests and dependencies -- 2.23.0

1 0

[PATCH compass-ci 1/2] mail-robot: use new key my_token for apply account
by Luan Shengde 12 Dec '20

12 Dec '20

[why] key my_uuid is renamed to my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 0fff3b5..93920f4 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -31,6 +31,7 @@ SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 # - my_email # - my_name # - my_uuid +# - my_token # my_ssh_pubkey # store_account_info # call ESClient to store my_info @@ -38,6 +39,7 @@ SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 # - my_email # - my_name # - my_uuid +# - my_token # - my_commit_url # - my_login_name # - my_ssh_pubkey @@ -62,6 +64,7 @@ class ApplyAccount # in order to successfully send email for failed parsing # firstly get my_email before execute parse_mail_content is needed @my_info['my_email'] = @mail_content.from[0] + @my_info['lab'] = ENV['lab'] parse_mail_content apply_my_account @@ -92,14 +95,17 @@ class ApplyAccount apply_info = {} if my_account_es my_ssh_pubkey_new = @my_info.delete('my_ssh_pubkey') + # my_token will take the place of my_uuid. + # to make the change with no sense, will keep the my_uuid for now + apply_info['my_token'] = my_account_es['my_uuid'] if my_account_es['my_token'].nil? apply_info.update my_account_es apply_info.update @my_info apply_info['my_ssh_pubkey'] = (apply_info['my_ssh_pubkey'] + my_ssh_pubkey_new).uniq @my_info.update apply_info apply_info['is_update_account'] = true else - my_uuid = %x(uuidgen).chomp - @my_info['my_uuid'] = my_uuid + my_token = %x(uuidgen).chomp + @my_info['my_token'] = my_token apply_info.update @my_info end apply_new_account(apply_info, my_account_es) @@ -112,10 +118,6 @@ class ApplyAccount @my_info['my_login_name'] = acct_info['my_login_name'] unless my_account_es end - def check_account_es - ESQuery.new(index: 'accounts').query_by_id(@my_info['my_email']) - end - def store_account_info es = ESClient.new(index: 'accounts') es.put_source_by_id(@my_info['my_email'], @my_info) -- 2.23.0

1 0

[PATCH compass-ci 3/3] answerback-email: rename key my_uuid with my_token
by Luan Shengde 12 Dec '20

12 Dec '20

key my_uuid is renamed to my_token. when applying/updating account, will use my_token instead for this change with no sense for old user(update account) keep key my_uuid and add key my_token for new user delete my_uuid and just add key my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c08f10e..401bd8c 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -29,12 +29,13 @@ JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = 'z9' my_info = { 'my_email' => nil, 'my_name' => nil, 'my_commit_url' => nil, - 'my_uuid' => nil, + 'my_token' => nil, 'my_login_name' => nil, 'my_ssh_pubkey' => [] } @@ -200,6 +201,7 @@ def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) new_stdin_pubkey = stdin_info.delete 'new_ssh_pubkey' new_pubkey = new_stdin_pubkey || new_email_pubkey + my_info['my_token'] = my_info_es['my_uuid'] if my_info['my_token'].nil? my_info.update my_info_es unless my_info_es.empty? my_info.update email_info unless email_info.empty? my_info.update stdin_info unless stdin_info.empty? @@ -221,7 +223,7 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) end def check_server - return true if ENV['HOSTNAME'] == 'z9' + return true if ENV['HOSTNAME'] == LAB message = 'please run the tool on z9 server' puts message @@ -260,7 +262,7 @@ def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) return unless check_my_email(my_info) return unless check_email_assigned_account(conf_info, my_info_es) - my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] + my_info['my_token'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) return unless check_my_name_exist(my_info) -- 2.23.0

1 0