- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/2] lib: friendly show compare template result in command line
by Lu Kaiyi 11 Dec '20

11 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the json format result of "compare -t <compare_template.yaml>" with terminal-table in Ruby. [usage] compare -t <compare_template.yaml> [example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | Hackbench Performance Testing | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | openeuler 20.03 average | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | openeuler 20.03 standard_deviation | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | Hackbench Performance Testing | +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | centos 7.6 average | 345243.03 | 142698.79 | 62108.35 | 34747.73 | 26330.19 | 10317.85 | 7471.71 | 3558.30 | | centos 7.6 standard_deviation | 97.00 | 95.00 | 122.00 | 125.00 | 100.00 | 130.00 | 101.00 | 103.00 | +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 53 ++++++++++++++++++++++++++++++++++++++ lib/compare_matrixes.rb | 3 ++- 2 files changed, 55 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index bf4013b..aa5179f 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'terminal-table' # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # @@ -66,6 +67,58 @@ class FormatEchartData end end +class FormatTableData + def initialize(result_hash) + @title = result_hash['title'] + @tables = result_hash['tables'] + end + + def show_table + @tables.each do |table_title, table| + @tb = Terminal::Table.new + set_field_names(table_title, table) + set_table_title + add_row(table) + set_align_column + print_table + end + end + + def set_table_title + @tb.title = @title + end + + def set_field_names(table_title, table) + field_names = [table_title] + field_names.concat(table['average']['source'][0]) + @tb.add_row(field_names) + @tb.add_separator + end + + def add_row(table) + row_names = %w[average standard_deviation] + row_names.each do |row_name| + row = table[row_name]['source'][1] + row_title = [row[0], row_name].join(' ') + format_data_row = row[1..-1] + format_data_row.map! { |data| format('%.2f', data) } + @tb.add_row([row_title, *format_data_row]) + end + end + + def set_align_column + @tb.number_of_columns.times do |index| + @tb.align_column(index + 1, :right) + end + @tb.align_column(0, :left) + end + + def print_table + puts @tb + puts + end +end + # input: x_params_list # eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] # output: diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 5293989..f7bb56a 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -510,7 +510,8 @@ end def show_compare_result(metrics_compare_results, template_params) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - print JSON.pretty_generate(echart_results) + table_data = FormatTableData.new(echart_results) + table_data.show_table end # Format Fields -- 2.23.0

1 0

help star
by Wu Fengguang 11 Dec '20

11 Dec '20

Xueliang is our help star, thanks Xueliang! wfg@crystal ~/todo% bin/help-star people/*/credit/2020-12-0[34].md 22.7 Cao Xueliang 20.0 Wei Jihui 19.8 Wu Zhende 16.0 Luan Shengde 13.6 Wu Fengguang 13.3 Lu Weitao 11.9 Xiao Shenwei 11.0 Bai Jing 9.3 Yu Chuan 7.7 Zhang Yu 7.4 Ren Wen 6.7 Zhang Yale 6.7 Cui Yili 6.2 Xu Xijian 6.2 Lin Jiaxin 5.7 Liu Yinsi 5.6 Zhang Dewan 5.6 Wang Yong 5.1 Lu Kaiyi 4.4 Li Ping 4.2 Hu Xuejiao 3.8 Liu Shaofei 3.5 Du Kaitian 2.6 Sun Yukui 0.9 Li Yuanchao 0.0 Tong Qunfeng 0.0 Wang Chenglong 0.0 Wang Shuai 0.0 Cui Xucui

1 0

review star
by Wu Fengguang 11 Dec '20

11 Dec '20

Shenwei is our review star, thanks Shenwei! wfg@crystal ~/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 19 12 34 21 Xiao Shenwei 95 30 44 19 Wei Jihui 28 16 25 17 Liu Shaofei 79 13 16 16 Cao Xueliang 20 4 18 13 Wang Yong 41 16 36 10 Wu Fengguang 34 9 10 9 Zhang Yu 22 7 18 9 Luan Shengde 30 4 17 8 Yu Chuan 26 3 8 7 Sun Yukui 19 5 7 6 Wu Zhende 18 4 4 5 Cui Yili 8 8 8 4 Li Ping 33 6 5 4 Hu Xuejiao 63 7 11 4 Wang Chenglong 19 7 15 4 Lu Kaiyi 6 2 4 3 Xu Xijian 13 1 2 2 Lu Weitao 14 4 5 2 Lin Jiaxin 17 5 5 2 Liu Yinsi 15 1 1 1 Bai Jing 4 0 0 0 Li Yuanchao 33 2 0 0 Zhang Yale 2 0 0 0 Zhang Dewan 20 0 0 0 Ren Wen

1 0

git count
by Wu Fengguang 11 Dec '20

11 Dec '20

Zhende is our code star again, thanks Zhende! wfg@crystal ~/todo% bin/git-count 13 Wu Fengguang 12 Wu Zhende 9 Cao Xueliang 9 Zhang Yale 7 Hu Xuejiao 6 Bai Jing 6 Liu Shaofei 6 Lu Weitao 6 Liu Yinsi 6 Wei Jihui 6 Zhang Yu 6 Luan Shengde 5 Yu Chuan 5 Cui Yili 5 Wang Chenglong 5 Li Yuanchao 5 Lu Kaiyi 4 Wang Yong 4 Lin Jiaxin 3 Sun Yukui 3 Ren Wen 2 Xu Xijian 2 Xiao Shenwei 1 Li Ping 0 Tong Qunfeng 0 Du Kaitian 0 Zhang Dewan

1 0

[PATCH compass-ci] logging: fix time difference of eight hours
by Wu Zhende 11 Dec '20

11 Dec '20

If the time zone information is not contained, the ES saves time as the UTC time by default. As a result, the kibana display time is eight hours later. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 2 +- src/lib/json_logger.cr | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index b3daeb4..7bf2229 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -11,7 +11,7 @@ @type record_transformer enable_ruby <record> - time ${time.strftime('%Y-%m-%dT%H:%M:%S')} + time ${time.strftime('%Y-%m-%dT%H:%M:%S.%3N+0800')} </record> </filter> diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index dd87020..6d61197 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -18,7 +18,7 @@ class JSONLogger < Logger Logger::Formatter.new do | severity, datetime, progname, msg, io| get_env_info(@env.as(HTTP::Server::Context)) if @env level_num = severity.to_i32 - datetime = datetime.to_s("%Y-%m-%dT%H:%M:%S") + datetime = datetime.to_s("%Y-%m-%dT%H:%M:%S.%3N+0800") logger_hash = JSON.parse(%({"level_num": #{level_num}, "level": "#{severity}", "time": "#{datetime}" -- 2.23.0

2 1

[PATCH lkp-tests] fix: depends/lkp: add ntpdate server
by Wang Chenglong 11 Dec '20

11 Dec '20

[Why] In $LKP_SRC/lib/bootstrap.s use ntpdate command to synchronize time. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- distro/depends/lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/lkp b/distro/depends/lkp index 6c8b6bf2..750b5c59 100644 --- a/distro/depends/lkp +++ b/distro/depends/lkp @@ -4,3 +4,4 @@ time kmod gzip which +ntpdate -- 2.23.0

1 0

[PATCH compass-ci 1/2] scheduler: subqueue: consume subqueue
by Ren Wen 11 Dec '20

11 Dec '20

Find subqueues related to the host default queues of all users, and consume the job from random one. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 20 +++++++++++++++++++- src/scheduler/redis_client.cr | 4 ++++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 6f51640..77071db 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -56,7 +56,25 @@ class Sched queues << item.strip end - return rand_queues(queues) + user_queues = [] of String + queues.each do |queue| + matched_queues = @redis.keys("*/sched/#{queue}*") + matched_queues.each do |mq| + user_queues << "#{mq}" + end + end + + user_queues = Set.new(user_queues) + user_queues = user_queues.map do |queue| + if queue =~ /sched\/(.+)\/ready$/ + $1 + else + "" + end + end + user_queues.reject!(&.empty?) + + return rand_queues(user_queues) end def get_job_from_queues(queues, testbox) diff --git a/src/scheduler/redis_client.cr b/src/scheduler/redis_client.cr index b816d67..1253108 100644 --- a/src/scheduler/redis_client.cr +++ b/src/scheduler/redis_client.cr @@ -16,6 +16,10 @@ class Redis::Client @client = Redis::PooledClient.new(host: host, port: port, pool_size: 25, pool_timeout: 0.01) end + def keys(pattern) + @client.keys(pattern) + end + def hash_set(key : String, field, value) @client.hset(key, field.to_s, value.to_s) end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] scheduler: subqueue: change queue to queue/subqueue
by Ren Wen 11 Dec '20

11 Dec '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d071d69..2c33ffd 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,7 +14,7 @@ class Sched end response = @task_queue.hand_over_task( - "sched/#{job.queue}", "extract_stats", job_id + "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] scheduler: subqueue: submit user's job to subqueue
by Ren Wen 11 Dec '20

11 Dec '20

Every user has a subqueue, user's email by default. Queue will be "queue/subqueue" when submitting job. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 6 ++++++ src/scheduler/submit_job.cr | 2 +- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7ded6d8..0083a7d 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -80,6 +80,7 @@ class Job testbox lab queue + subqueue initrd_pkg initrd_deps initrds_uri @@ -161,6 +162,7 @@ class Job set_lkp_server() set_sshr_info() set_queue() + set_subqueue() end private def set_kernel @@ -266,6 +268,10 @@ class Job end end + private def set_subqueue + self["subqueue"] = self["my_email"] + end + # if not assign tbox_group, set it to a match result from testbox # ?if job special testbox, should we just set tbox_group=textbox private def update_tbox_group_from_testbox diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index f93ad2b..3909cb7 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -107,7 +107,7 @@ class Sched # success: [{"job_id" => job_id, "message" => "", job_state => "submit"}] # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) - queue = job.queue + queue = "#{job.queue}/#{job.subqueue}" # only single job will has "idle job" and "execute rate limiter" if job["idle_job"].empty? -- 2.23.0

1 0

Re: [PATCH v4 compass-ci] doc/job: add result_root.md
by Zhang Yu 11 Dec '20

11 Dec '20

>>+ result_root: /result/host-info/2020-12-09/vm-2p8g/openeuler-20.03-aarch64/crystal.166995 > > the example missing the field $pp_params. good Thanks, Zhangyu > >>-- >>2.23.0 >>

1 0