On Wed, Oct 28, 2020 at 08:00:22PM +0800, Cao Xueliang wrote:
On Wed, Oct 28, 2020 at 06:18:52PM +0800, Lu Weitao wrote:
auto group jobs with user-defined template to support template compare mode background: before compare, create job-groups for all jobs that query from ES with user's filter then, create matrices-groups according job-groups finally, compare each group matrices
Signed-off-by: Lu Weitao luweitaobe@163.com
lib/params_group.rb | 120 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+)
diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..4d10dbe 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -113,3 +113,123 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end
+# -------------------------------------------------------------------------------------------------- +# auto group job_list by user's template +# Exammple: +# Input:
Example, pls use F3 do the spell check.
ok, many thanks!
+# 1. jobs_list. +# 2. params from template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4k|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'contos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } +#
+def auto_group_by_template(jobs_list, group_params, dimensions, metrics)
- job_list = extract_jobs_list(jobs_list)
- get_group_by_template(job_list, group_params, dimensions, metrics)
+end
+def get_group_by_template(job_list, group_params, dimensions, metrics)
- groups = {}
- job_list.each do |job|
- next unless job['stats']
- group_key = get_user_group_key(job, group_params)
- dimension = get_user_dimension(job, dimensions)
- next unless group_key && dimension
- new_job = get_new_job_by_metrics(job, metrics)
- next if new_job.empty?
- groups[group_key] ||= {}
- groups[group_key][dimension] ||= []
- groups[group_key][dimension] << new_job
- end
- groups
+end
+# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params)
- group_key_list = []
- group_params.each do |param|
- if job.key?(param)
group_key_list << job[param]next- end
- job.each_value do |v|
if v.is_a?(Hash) && v.key?(param)group_key_list << v[param]breakend- end
- end
- return nil if group_key_list.size < group_params.size
- group_key_list.join('|')
+end
+# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions)
- dimension_list = []
- dimensions.each do |dim|
- dim.each_key do |key|
if job[key] == dim[key]dimension_list << job[key]end- end
- break unless dimension_list.empty?
- end
- return nil if dimension_list.size < dimensions.size
If dimensions.size is 3, but the each dimension only has 2 keys, why need the return?
yes, that's a issue good question
- dimension_list.join(' ')
+end
+# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job_by_metrics(job, metrics)
- new_job = {}
- metrics.each do |metric|
- if job['stats'].key?(metric)
How does this statement work?
you means which statement?
metric is one key of job['stats'], we just need compare user-defined metrics, so we filter by each metric in job
Thanks, Weitao
Thanks, Xueliang
new_job['stats'] ||= {}new_job['stats'][metric] = job['stats'][metric]- end
- end
- new_job
+end
2.23.0