[PATCH v4 compass-ci 1/2] auto group jobs with user-defined template
auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: auto group jobs_list Signed-off-by: Lu Weitao <luweitaobe@163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list) + get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + new_job = get_new_job(job, metrics) + next if new_job.empty? + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + value = find_param_in_job(job, param) + group_key_list << value if value + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +def find_param_in_job(job, param) + return job[param] if job.key?(param) + + job.each_value do |v| + return v[param] if v.is_a?(Hash) && v.key?(param) + end +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job(job, metrics) + return {} unless job['stats'] + + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0
On Wed, Nov 04, 2020 at 06:15:49PM +0800, Lu Weitao wrote:
auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are:
are => is
load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list
Do you mean load compare_template.yaml? Or load_compare_template is a fuction name? Then should not add '.yaml'.
---> create groups_matrices ---> compare_values by each metrics ---> format/show results
current patch is the step: auto group jobs_list
What current patch do is: auto group jobs_list Thanks, Yuanchao
On Thu, Nov 05, 2020 at 03:33:10PM +0800, Li Yuanchao wrote:
On Wed, Nov 04, 2020 at 06:15:49PM +0800, Lu Weitao wrote:
auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are:
are => is
OK
load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list
Do you mean load compare_template.yaml? Or load_compare_template is a fuction name? Then should not add '.yaml'.
that should be load compare_template.yaml
---> create groups_matrices ---> compare_values by each metrics ---> format/show results
current patch is the step: auto group jobs_list
What current patch do is: auto group jobs_list
godd iead Thanks, Weitao
Thanks, Yuanchao
On Wed, Nov 04, 2020 at 06:15:49PM +0800, Lu Weitao wrote:
auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results
current patch is the step: auto group jobs_list
Signed-off-by: Lu Weitao <luweitaobe@163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-)
diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true
-# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list)
job_list is too similar to jobs_list. You can name them job_list: extracted_jobs jobs_list: filtered_jobs //it means you query it from es by filters
+ get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + new_job = get_new_job(job, metrics) + next if new_job.empty? + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + value = find_param_in_job(job, param) + group_key_list << value if value + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +def find_param_in_job(job, param) + return job[param] if job.key?(param) + + job.each_value do |v| + return v[param] if v.is_a?(Hash) && v.key?(param) + end +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job(job, metrics)
+ return {} unless job['stats']
The above line is not needed. Thanks, Yuanchao
+ + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0
On Thu, Nov 05, 2020 at 03:43:38PM +0800, Li Yuanchao wrote:
On Wed, Nov 04, 2020 at 06:15:49PM +0800, Lu Weitao wrote:
auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results
current patch is the step: auto group jobs_list
Signed-off-by: Lu Weitao <luweitaobe@163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-)
diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true
-# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list)
job_list is too similar to jobs_list.
You can name them job_list: extracted_jobs jobs_list: filtered_jobs //it means you query it from es by filters
More choices: "jobs", "jobs_list". Thanks, Zhang Yuhang
On Thu, Nov 05, 2020 at 04:00:11PM +0800, Zhang Yuhang wrote:
On Thu, Nov 05, 2020 at 03:43:38PM +0800, Li Yuanchao wrote:
On Wed, Nov 04, 2020 at 06:15:49PM +0800, Lu Weitao wrote:
auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results
current patch is the step: auto group jobs_list
Signed-off-by: Lu Weitao <luweitaobe@163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-)
diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true
-# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list)
job_list is too similar to jobs_list.
You can name them job_list: extracted_jobs jobs_list: filtered_jobs //it means you query it from es by filters
More choices: "jobs", "jobs_list".
seem it's also ok Thanks, Weitao
Thanks, Zhang Yuhang
On Thu, Nov 05, 2020 at 03:43:38PM +0800, Li Yuanchao wrote:
On Wed, Nov 04, 2020 at 06:15:49PM +0800, Lu Weitao wrote:
auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results
current patch is the step: auto group jobs_list
Signed-off-by: Lu Weitao <luweitaobe@163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-)
diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true
-# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list)
job_list is too similar to jobs_list.
how about use jobs_list again
You can name them job_list: extracted_jobs jobs_list: filtered_jobs //it means you query it from es by filters
+ get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + new_job = get_new_job(job, metrics) + next if new_job.empty? + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + value = find_param_in_job(job, param) + group_key_list << value if value + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +def find_param_in_job(job, param) + return job[param] if job.key?(param) + + job.each_value do |v| + return v[param] if v.is_a?(Hash) && v.key?(param) + end +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job(job, metrics)
+ return {} unless job['stats']
The above line is not needed.
ok, let me try
Thanks, Yuanchao
+ + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0
On Thu, Nov 05, 2020 at 05:13:18PM +0800, Lu Weitao wrote:
On Thu, Nov 05, 2020 at 03:43:38PM +0800, Li Yuanchao wrote:
On Wed, Nov 04, 2020 at 06:15:49PM +0800, Lu Weitao wrote:
auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results
current patch is the step: auto group jobs_list
Signed-off-by: Lu Weitao <luweitaobe@163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-)
diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true
-# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list)
job_list is too similar to jobs_list.
how about use jobs_list again
Technically, it's ok. I just wander would it be confused. Thanks, Yuanchao
You can name them job_list: extracted_jobs jobs_list: filtered_jobs //it means you query it from es by filters
On Thu, Nov 05, 2020 at 03:43:38PM +0800, Li Yuanchao wrote:
On Wed, Nov 04, 2020 at 06:15:49PM +0800, Lu Weitao wrote:
auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results
current patch is the step: auto group jobs_list
Signed-off-by: Lu Weitao <luweitaobe@163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-)
diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true
-# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list)
job_list is too similar to jobs_list.
You can name them job_list: extracted_jobs jobs_list: filtered_jobs //it means you query it from es by filters
+ get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + new_job = get_new_job(job, metrics) + next if new_job.empty? + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + value = find_param_in_job(job, param) + group_key_list << value if value + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +def find_param_in_job(job, param) + return job[param] if job.key?(param) + + job.each_value do |v| + return v[param] if v.is_a?(Hash) && v.key?(param) + end +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job(job, metrics)
+ return {} unless job['stats']
The above line is not needed.
lib/params_group.rb:223:3: C: Layout/EmptyLineAfterGuardClause: Add empty line after guard clause. return {} unless job['stats'] rubocop required a empty line. and if job['stats'] does exists, we needn't foreach metric Thanks, Weitao
Thanks, Yuanchao
+ + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0
participants (3)
-
Li Yuanchao -
Lu Weitao -
Zhang Yuhang