- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/2] tutorial.md: delete old content
by Wang Chenglong 14 Dec '20

14 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- doc/tutorial.md | 82 ------------------------------------------------- 1 file changed, 82 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index e6f8491..39d62e1 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -146,88 +146,6 @@ job yaml ��[YAML](http://yaml.org/YAM ### �� OS �� -�� initramfs �� cgz �� cgz �� rootfs�� I/O �� - -#### �� - - 1. �� os �� rootfs�� openEuler �� - - �� docker �� rootfs - - 1) �� openEuler �� docker �� - - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… - ``` - 2) �� docker �� - ```bash - docker load -i openEuler-docker.aarch64 - ``` - - 3) �� openEuler �� - ```bash - docker run -id openeuler-20.03-lts - ``` - 4) �� docker �� rootfs - ```bash - docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs - ``` - - �� qemu.img(qcow2��)�� rootfs (openEuler��) - - - 1) �� openEule r�� qcow2 �� - ```bash - wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… - ``` - 2) ��{compass-ci}/container/qcow2rootfs ��rootfs - ```bash - cd {compass-ci}/container/qcow2rootfs - ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs - ``` - 2. ��rootfs - 1. ��chroot�� rootfs (�� root ��) - ```bash - chroot openEuler-rootfs - ``` - 2. �� - - a. �� root �� - b. �� ssh �� - c. �� - d. �� docker �� osimage �� - > 1. �� - > 2. �� centos ��rpm�� - > 3. �� yum �� - > 4. �� docker �� - - 3. �� rootfs�� - ```bash - cd $rootfs - find . | coip -o -Hnewc |gzip -9 > $os_name.cgz - ``` -#### FAQ -1. �� Unable to mount root fs on unknown-block�� - - �� - ```bash - [ 0.390437] List of all partitions: - [ 0.390806] No filesystem could mount root, tried: - [ 0.391489] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) - ... - [ 0.399404] Memory Limit: none - [ 0.399749] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]--- - ``` - - �� - - 1�� - 2�� 644 �� - -2. �� - - - �� - - �� - - �� - - �� rootfs �� ### �� -- 2.23.0

1 0

[PATCH lkp-tests] tests: fix error when running host-info in vm
by Zhang Yu 14 Dec '20

14 Dec '20

[why] Error: Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory [how] Run ipmi command only on physical machines Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tests/host-info b/tests/host-info index 84f796dcb..7e6e71a13 100755 --- a/tests/host-info +++ b/tests/host-info @@ -1,5 +1,7 @@ #!/bin/bash +. $LKP_SRC/lib/env.sh + get_memory() { memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') @@ -102,7 +104,7 @@ get_ipmi_ip() # RMCP+ Cipher Suites : 0,1,2,3,17 # Cipher Suite Priv Max : XuuaXXXXXXXXXXX local ip=$(ipmitool lan print | awk 'NR==3{print $4}') - [ -n "$ip" ] && echo "ipmi_ip:" $ip + [ -n "$ip" ] && echo "ipmi_ip: $ip" } @@ -113,7 +115,7 @@ main() get_partitions get_mac get_cpu_info - get_ipmi_ip + is_virt || get_ipmi_ip } main -- 2.23.0

2 2

[PATCH v2 compass-ci] scheduler: prints logs in Sched class
by Wu Zhende 14 Dec '20

14 Dec '20

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 24 ++++++++-- src/scheduler/close_job.cr | 7 ++- src/scheduler/download_file.cr | 14 ++++++ src/scheduler/find_job_boot.cr | 6 ++- src/scheduler/find_next_job_boot.cr | 6 ++- src/scheduler/scheduler.cr | 64 +++++---------------------- src/scheduler/submit_job.cr | 10 +++-- src/scheduler/update_job_parameter.cr | 2 +- 8 files changed, 68 insertions(+), 65 deletions(-) create mode 100644 src/scheduler/download_file.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 6225ac2..10a51ab 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -21,6 +21,7 @@ require "../scheduler/close_job" require "../scheduler/request_cluster_state" require "../scheduler/update_job_parameter" require "../scheduler/create_job_cpio" +require "../scheduler/download_file" class Sched property es @@ -37,6 +38,15 @@ class Sched @log = env.log.as(JSONLogger) end + def debug_message(response) + @log.info(%({"from": "#{(a)env.request.remote_address}", "response": #{response.to_json}})) + end + + def alive(version) + debug_message("Env= {\n#{`export`}}") + "LKP Alive! The time is #{Time.local}, version = #{version}" + end + def normalize_mac(mac : String) mac.gsub(":", "-") end @@ -97,11 +107,19 @@ class Sched # json log hash["testbox"] = testbox - return hash.to_json + @log.info(hash.to_json) end - def report_ssh_port(testbox : String, ssh_port : String) - @redis.hash_set("sched/tbox2ssh_port", testbox, ssh_port) + def report_ssh_port + testbox = @env.params.query["tbox_name"] + ssh_port = @env.params.query["ssh_port"].to_s + job_id = @env.params.query["job_id"].to_s + + if testbox && ssh_port + @redis.hash_set("sched/tbox2ssh_port", testbox, ssh_port) + end + + @log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) end private def query_consumable_keys(shortest_queue_name) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 2c33ffd..579c2c0 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -2,7 +2,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def close_job(job_id : String) + def close_job + job_id = @env.params.query["job_id"]? + return unless job_id + job = @redis.get_job(job_id) delete_access_key_file(job) if job @@ -22,7 +25,7 @@ class Sched @redis.remove_finished_job(job_id) - return %({"job_id": "#{job_id}", "job_state": "complete"}) + @log.info(%({"job_id": "#{job_id}", "job_state": "complete"})) end def delete_access_key_file(job : Job) diff --git a/src/scheduler/download_file.cr b/src/scheduler/download_file.cr new file mode 100644 index 0000000..e1f0c9a --- /dev/null +++ b/src/scheduler/download_file.cr @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def download_file + job_id = @env.params.url["job_id"] + job_package = @env.params.url["job_package"] + file_path = ::File.join [Kemal.config.public_folder, job_id, job_package] + + @log.info(%({"job_id": "#{job_id}", "job_state": "download"})) + + send_file @env, file_path + end +end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index fb94e91..5a11523 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -16,7 +16,11 @@ class Sched host = value end - get_job_boot(host, boot_type) + response = get_job_boot(host, boot_type) + job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? + @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id + + response end # auto submit a job to collect the host information diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 09abbd5..0bdfe5b 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -9,6 +9,10 @@ class Sched hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) end - get_job_boot(hostname, "ipxe") + response = get_job_boot(hostname, "ipxe") + job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? + @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id + + response end end diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 4fde6e3..7ec149c 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -41,15 +41,9 @@ module Scheduler env.create_sched end - # for debug (maybe kemal debug|logger does better) - def self.debug_message(env, response) - env.log.info(%({"from": "#{env.request.remote_address}", "response": #{response.to_json}})) - end - # echo alive get "/" do |env| - debug_message(env, "Env= {\n#{`export`}}") - "LKP Alive! The time is #{Time.local}, version = #{VERSION}" + env.sched.alive(VERSION) end # for XXX_runner get job @@ -58,46 +52,24 @@ module Scheduler # /boot.xxx/host/${hostname} # /boot.yyy/mac/${mac} get "/boot.:boot_type/:parameter/:value" do |env| - response = env.sched.find_job_boot - - job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? - env.log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id - - response + env.sched.find_job_boot end # /~lkp/cgi-bin/gpxelinux.cgi?hostname=:hostname&mac=:mac&last_kernel=:last_kernel get "/~lkp/cgi-bin/gpxelinux.cgi" do |env| - response = env.sched.find_next_job_boot - - job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? - env.log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id - - response + env.sched.find_next_job_boot end # enqueue # - echo job_id to caller # -- job_id = "0" ? means failed post "/submit_job" do |env| - job_messages = env.sched.submit_job - - job_messages.each do |job_message| - env.log.info(job_message.to_json) - end - - job_messages.to_json + env.sched.submit_job.to_json end # file download server get "/job_initrd_tmpfs/:job_id/:job_package" do |env| - job_id = env.params.url["job_id"] - job_package = env.params.url["job_package"] - file_path = ::File.join [Kemal.config.public_folder, job_id, job_package] - - env.log.info(%({"job_id": "#{job_id}", "job_state": "download"})) - - send_file env, file_path + env.sched.download_file end # client(runner) report its hostname and mac @@ -155,7 +127,7 @@ module Scheduler # ?job_file=/lkp/scheduled/job.yaml&job_state=post_run&job_id=10 # ?job_file=/lkp/scheduled/job.yaml&loadavg=0.28 0.82 0.49 1/105 3389&start_time=1587725398&end_time=1587725698&job_id=10 get "/~lkp/cgi-bin/lkp-jobfile-append-var" do |env| - env.log.info(env.sched.update_job_parameter) + env.sched.update_job_parameter "Done" end @@ -178,42 +150,26 @@ module Scheduler # response: get "server ip" from cluster state, # return "server=<server ip>". get "/~lkp/cgi-bin/lkp-cluster-sync" do |env| - response = env.sched.request_cluster_state - - debug_message(env, response) - - response + env.sched.request_cluster_state end # client(runner) report job post_run finished # /~lkp/cgi-bin/lkp-post-run?job_file=/lkp/scheduled/job.yaml&job_id=40 # curl "http://localhost:3000/~lkp/cgi-bin/lkp-post-run?job_file=/lkp/scheduled/job…" get "/~lkp/cgi-bin/lkp-post-run" do |env| - # get job_id from request - job_id = env.params.query["job_id"]? - if job_id - env.log.info(env.sched.close_job(job_id)) - end + env.sched.close_job "Done" end get "/~lkp/cgi-bin/lkp-wtmp" do |env| - env.log.info(env.sched.update_tbox_wtmp) + env.sched.update_tbox_wtmp "Done" end get "/~lkp/cgi-bin/report_ssh_port" do |env| - testbox = env.params.query["tbox_name"] - ssh_port = env.params.query["ssh_port"].to_s - job_id = env.params.query["job_id"].to_s - - if testbox && ssh_port - env.sched.report_ssh_port(testbox, ssh_port) - end - - env.log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) + env.sched.report_ssh_port "Done" end diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 3909cb7..4303b1d 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -11,20 +11,24 @@ class Sched cluster_file = job["cluster"] if cluster_file.empty? || cluster_file == "cs-localhost" - return submit_single_job(job) + response = submit_single_job(job) else cluster_config = get_cluster_config(cluster_file, job.lkp_initrd_user, job.os_arch).not_nil! - return submit_cluster_job(job, cluster_config) + response = submit_cluster_job(job, cluster_config) end rescue ex @log.warn(ex.inspect_with_backtrace) - return [{ + response = [{ "job_id" => "0", "message" => ex.to_s, "job_state" => "submit", }] + ensure + response.each do |job_message| + @log.info(job_message.to_json) + end end # return: diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index cee8b9b..da528c1 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -29,6 +29,6 @@ class Sched # json log log = job_content.dup log["job_id"] = log.delete("id").not_nil! - return log.to_json + @log.info(log.to_json) end end -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] answerback-email: rename key my_uuid with my_token
by Luan Shengde 14 Dec '20

14 Dec '20

key my_uuid is renamed to my_token. when applying/updating account, will use my_token instead to do this change with no sense for old user(update account) keep key my_uuid and add key my_token for new user just add key my_token Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 21 ++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index c08f10e..eb906b3 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -21,6 +21,7 @@ names = Set.new %w[ JUMPER_PORT SEND_MAIL_HOST SEND_MAIL_PORT + lab ] defaults = relevant_defaults(names) @@ -29,12 +30,14 @@ JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +LAB = defaults['lab'] +HOST_SERVER = 'crystal.ci' my_info = { 'my_email' => nil, 'my_name' => nil, 'my_commit_url' => nil, - 'my_uuid' => nil, + 'my_token' => nil, 'my_login_name' => nil, 'my_ssh_pubkey' => [] } @@ -179,6 +182,7 @@ def apply_account(my_info, conf_info) apply_info = {} apply_info.update my_info apply_info.update conf_info + apply_info['lab'] = LAB account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str @@ -200,9 +204,7 @@ def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) new_stdin_pubkey = stdin_info.delete 'new_ssh_pubkey' new_pubkey = new_stdin_pubkey || new_email_pubkey - my_info.update my_info_es unless my_info_es.empty? - my_info.update email_info unless email_info.empty? - my_info.update stdin_info unless stdin_info.empty? + update_my_info(my_info, my_info_es, email_info, stdin_info) return if new_pubkey.nil? return if my_info['my_ssh_pubkey'].include? new_pubkey @@ -210,6 +212,13 @@ def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) my_info['my_ssh_pubkey'].insert(0, new_pubkey) end +def update_my_info(my_info, my_info_es, email_info, stdin_info) + my_info['my_token'] = my_info_es['my_uuid'] if my_info['my_token'].nil? + my_info.update my_info_es unless my_info_es.empty? + my_info.update email_info unless email_info.empty? + my_info.update stdin_info unless stdin_info.empty? +end + def build_my_info_from_account_info(my_info, account_info, conf_info) unless account_info['my_jumper_pubkey'].nil? return if my_info['my_ssh_pubkey'][-1] == account_info['my_jumper_pubkey'] @@ -221,7 +230,7 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) end def check_server - return true if ENV['HOSTNAME'] == 'z9' + return true if ENV['HOSTNAME'] == HOST_SERVER message = 'please run the tool on z9 server' puts message @@ -260,7 +269,7 @@ def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) return unless check_my_email(my_info) return unless check_email_assigned_account(conf_info, my_info_es) - my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] + my_info['my_token'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) return unless check_my_name_exist(my_info) -- 2.23.0

2 2

[PATCH v2 lkp-tests] tests/build-pkg: fix git clone error
by Wang Yong 12 Dec '20

12 Dec '20

[before] '"${var}"' [after] '${var}' " will be treated as a character, it's redundant --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index aeaf17787..518298228 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -107,7 +107,7 @@ replace_source() do echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { - url="\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" } } url=$(echo "$url" | sed 's|https://|http://|g') -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix git clone error
by Wang Yong 12 Dec '20

12 Dec '20

[before] '${var}' [after] '"${var}"' character " is redundant --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index aeaf17787..518298228 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -107,7 +107,7 @@ replace_source() do echo "$url" | egrep 'git\+|\.git$' && { [ -n "$repo_dir" ] && { - url="\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + url="${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}" } } url=$(echo "$url" | sed 's|https://|http://|g') -- 2.23.0

2 2

[PATCH v3 compass-ci 3/3] lib/compare_data_format.rb: limit the number of table columns in a single row
by Lu Kaiyi 12 Dec '20

12 Dec '20

[example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - meminfo.Active - meminfo.Memused filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 4k|1G | 4k|10G | 4k|80G | 4k|100G | 16k|1G | 16k|10G | 16k|100G | 32k|1G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1051645.34 | 704958.00 | 706798.00 | 657669.27 | 1030484.85 | 979289.00 | 948576.00 | 1262591.17 | | standard_deviation openeuler 20.03 | 74.00 | 0.00 | 0.00 | 8.00 | 46.00 | 0.00 | 2.00 | 42.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 64k|1G | 64k|100G | 128k|1G | 128k|100G | 256k|1G | 256k|100G | 512k|1G | 512k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 1418731.53 | 1854231.33 | 2004321.76 | 2960171.75 | 2979326.50 | 4775340.00 | 5003555.53 | 5630845.75 | | standard_deviation openeuler 20.03 | 47.00 | 10.00 | 65.00 | 18.00 | 88.00 | 8.00 | 108.00 | 15.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | meminfo.Active | 1024k|1G | 1024k|100G | 2048k|100G | 4096k|100G | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ | average openeuler 20.03 | 8974190.91 | 9641039.00 | 5480763.67 | 7302509.67 | | standard_deviation openeuler 20.03 | 118.00 | 36.00 | 86.00 | 67.00 | +------------------------------------+------------+------------+------------+------------+------------+------------+------------+------------+ ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 42 ++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 9 deletions(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index c1f7023..0226a65 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -72,19 +72,20 @@ end # format compare template results into a table format # class FormatTableData - def initialize(result_hash) + def initialize(result_hash, row_size=8) @title = result_hash['title'] @tables = result_hash['tables'] @unit = result_hash['unit'] @x_name = result_hash['x_name'] + @row_size = row_size end def show_table @tables.each do |table_title, table| @tb = Terminal::Table.new set_table_title - set_field_names(table_title, table) - add_rows(table) + row_num = get_row_num(table) + split_data_column(table_title, table, row_num) set_align_column print_table end @@ -94,14 +95,37 @@ class FormatTableData @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" end - def set_field_names(table_title, table) + def get_row_num(table) + data_column_size = table['average']['source'][0].size + row_num = data_column_size / @row_size + row_rem = data_column_size % @row_size + if row_rem > 0 + row_num += 1 + end + row_num + end + + def split_data_column(table_title, table, row_num) + row_num.times do |row| + starts = 1 + row * @row_size + ends = starts + @row_size + set_field_names(table_title, table, starts, ends) + add_rows(table, starts, ends) + break if row == row_num - 1 + + @tb.add_separator + @tb.add_separator + end + end + + def set_field_names(table_title, table, starts, ends) field_names = [table_title] - field_names.concat(table['average']['source'][0]) + field_names.concat(table['average']['source'][0][starts-1...ends-1]) @tb.add_row(field_names) @tb.add_separator end - def add_rows(table) + def add_rows(table, starts, ends) row_names = %w[average standard_deviation change] max_size = row_names.map(&:size).max row_names.each do |row_name| @@ -109,15 +133,15 @@ class FormatTableData dimensions_size = table[row_name]['dimensions'].size (1...dimensions_size).each do |index| - add_row(table, row_name, index, max_size) + add_row(table, row_name, index, max_size, starts, ends) end end end - def add_row(table, row_name, index, max_size) + def add_row(table, row_name, index, max_size, starts, ends) row = table[row_name]['source'][index] row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') - format_data_row = row[1..-1] + format_data_row = row[starts...ends] if row_name == 'change' format_data_row.map! { |data| format('%.1f%%', data) } else -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] add options for compare_by_template
by Lu Kaiyi 12 Dec '20

12 Dec '20

if want json format results, use command like below: compare -t compare_template.yaml --theme json otherwise, print table format results by default: compare -t compare_template.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 11 ++++++++--- sbin/compare | 2 +- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index f34e239..db264f8 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -96,11 +96,11 @@ end # unit: KB/s # -def compare_by_template(template) +def compare_by_template(template, options) template_params = load_template(template) groups_matrices = create_groups_matrices(template_params) compare_results = compare_metrics_values(groups_matrices) - show_compare_result(compare_results, template_params) + show_compare_result(compare_results, template_params, options) end def load_template(template) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 6a60f2b..0e08ec7 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -507,11 +507,16 @@ def get_dimensions_combination(dimension_list) dims end -def show_compare_result(metrics_compare_results, template_params) +def show_compare_result(metrics_compare_results, template_params, options) + theme = options[:theme] formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - table_results = FormatTableData.new(echart_results) - table_results.show_table + if theme == 'json' + print JSON.pretty_generate(echart_results) + else + table_results = FormatTableData.new(echart_results) + table_results.show_table + end end # Format Fields diff --git a/sbin/compare b/sbin/compare index c13ec1a..e8d1ac4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -67,7 +67,7 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful if template - compare_by_template(template) + compare_by_template(template, options) elsif is_group compare_group(argv, dimensions, options) else -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] lib: friendly show compare template result in command line
by Lu Kaiyi 12 Dec '20

12 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the results of "compare -t <compare_template.yaml>" with terminal-table in Ruby. [example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: centos os_version: 7.6 - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | average openeuler 20.03 | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | standard_deviation openeuler 20.03 | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | Hackbench Performance Testing (unit: KB/s, x_name: bs|test_size) | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | average centos 7.6 | 345243.03 | 142698.79 | 62108.35 | 34747.73 | 26330.19 | 10317.85 | 7471.71 | 3558.30 | | average openeuler 20.03 | 122003.54 | 33528.53 | 31469.06 | 13870.14 | 8249.71 | 4329.45 | 1976.54 | 1141.00 | | standard_deviation centos 7.6 | 97.00 | 95.00 | 122.00 | 125.00 | 100.00 | 130.00 | 101.00 | 103.00 | | standard_deviation openeuler 20.03 | 174.00 | 188.00 | 171.00 | 197.00 | 181.00 | 175.00 | 170.00 | 176.00 | | change centos 7.6 vs openeuler 20.03 | 183.0% | 325.6% | 97.4% | 150.5% | 219.2% | 138.3% | 278.0% | 211.9% | +--------------------------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 73 ++++++++++++++++++++++++++++++++++++++ lib/compare_matrixes.rb | 3 +- 2 files changed, 75 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index bf4013b..c1f7023 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -2,6 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'terminal-table' + # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # @@ -66,6 +68,77 @@ class FormatEchartData end end +# ---------------------------------------------------------------------------------------------------- +# format compare template results into a table format +# +class FormatTableData + def initialize(result_hash) + @title = result_hash['title'] + @tables = result_hash['tables'] + @unit = result_hash['unit'] + @x_name = result_hash['x_name'] + end + + def show_table + @tables.each do |table_title, table| + @tb = Terminal::Table.new + set_table_title + set_field_names(table_title, table) + add_rows(table) + set_align_column + print_table + end + end + + def set_table_title + @tb.title = "#{@title} (unit: #{@unit}, x_name: #{@x_name})" + end + + def set_field_names(table_title, table) + field_names = [table_title] + field_names.concat(table['average']['source'][0]) + @tb.add_row(field_names) + @tb.add_separator + end + + def add_rows(table) + row_names = %w[average standard_deviation change] + max_size = row_names.map(&:size).max + row_names.each do |row_name| + next unless table[row_name] + + dimensions_size = table[row_name]['dimensions'].size + (1...dimensions_size).each do |index| + add_row(table, row_name, index, max_size) + end + end + end + + def add_row(table, row_name, index, max_size) + row = table[row_name]['source'][index] + row_title = [row_name + ' ' * (max_size - row_name.size), row[0]].join(' ') + format_data_row = row[1..-1] + if row_name == 'change' + format_data_row.map! { |data| format('%.1f%%', data) } + else + format_data_row.map! { |data| format('%.2f', data) } + end + @tb.add_row([row_title, *format_data_row]) + end + + def set_align_column + @tb.number_of_columns.times do |index| + @tb.align_column(index + 1, :right) + end + @tb.align_column(0, :left) + end + + def print_table + puts @tb + puts + end +end + # input: x_params_list # eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] # output: diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 5293989..6a60f2b 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -510,7 +510,8 @@ end def show_compare_result(metrics_compare_results, template_params) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - print JSON.pretty_generate(echart_results) + table_results = FormatTableData.new(echart_results) + table_results.show_table end # Format Fields -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] assign-account: rename my_uuid to my_token
by Luan Shengde 12 Dec '20

12 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index c0753b0..4ed95c0 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -24,7 +24,7 @@ get '/assign_account' do end def assign_jumper_account(data) - lacked_info = %w[my_email my_name my_uuid] - data.keys + lacked_info = %w[my_email my_name my_token] - data.keys error_message = "lack of my infos: #{lacked_info.join(', ')}." raise error_message unless lacked_info.empty? -- 2.23.0

1 0