[Compass-ci] Re: [PATCH v2 compass-ci 2/2] add template compare mode

On Wed, Nov 04, 2020 at 06:34:38PM +0800, Lu Weitao wrote:

add template compare mode in sbin/compare(command-line tool) by:

• load and parse compare.yaml
• calls lib/es_query.rb, query jobs data set from ES
• calls lib/matrix.rb, convert jobs data set to group matrices
• calls lib/compare_matrixes.rb, use groups matrices to compare

background: To support compare with user-defined template feature, the work-flow of user-defined template feature:

work-flow --> workflow user-defined --> user defined

   	load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list

load_compare_template.yaml --> load compare_template.yaml

       ---> create groups_matrices ---> compare_values by each metrics  ---> format/show results

example: compare -t/--template compare_template.yaml, the yaml like: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size

title: Hackbench Performance Testing unit: KB/s

output:{ "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "bs|test_size", "tables": { "4k|1G": { "fio.write_iops": { "average": { "openeuler 20.03": 308298.5524761111, "debian sid": 220289.07563 }, "standard_deviation": { "openeuler 20.03": null, "debian sid": null }, "change": { "debian sid vs openeuler 20.03": -28.5 } } }, "16k|1G": {...}, ... } }

Signed-off-by: Lu Weitao luweitaobe@163.com

lib/compare.rb | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++ sbin/compare | 10 +++++++++- 2 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/lib/compare.rb b/lib/compare.rb index 2837edc..9edf049 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -6,6 +6,7 @@ require_relative './es_query.rb' require_relative './matrix2.rb' require_relative './compare_matrixes.rb' require_relative './constants.rb' +require 'yaml'

# ------------------------------------------------------------------------------------------- # compare_matrices_list @@ -66,3 +67,53 @@ def create_groups_matrices_list(conditions, dims) query_results = es.multi_field_query(conditions) combine_group_query_data(query_results, dims) end

+# ------------------------------------------------------------------------------------------- +# compare with user-defined compare_template.yaml

^

+# the work-flow are:

^

+# load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list

^

Thanks, Zhang Yuhang

+# ---> create groups_matrices ---> compare_values by each metrics ---> format/show results +#

+def compare_by_template(template)

• template_params = load_template(template)
• groups_matrices = create_groups_matrices(template_params)
• compare_values = compare_metrics_values(groups_matrices)
• chart_results = format_for_chart(compare_values, template_params)
• show_chart_compare_result(chart_results)

+end

+def load_template(template)

• unless File.file?(template)
• warn 'template does not exist'
• exit
• end
• YAML.load_file(template)

+end

+# input: template_params: Hash +# eg: +# { +# "compare_metrics"=>["fio.write_iops", "fio.read_iops"], +# "filter"=>[ +# {"suite"=>["fio-bisic"]}, +# {"os_arch"=>["aarch_64"]} +# ], +# "compare_dimensions"=>[ +# {"os"=>"openeuler", "os_version"=>20.03}, +# {"os"=>"centos", "os_version"=>7.6} +# ], +# "x_params"=>["block_size", "package_size"], +# "title"=>"Hackbench Performance Testing", +# "unit"=>"KB/s" +# } +def create_groups_matrices(template_params)

• es = ESQuery.new
• query_results = es.multi_field_query(template_params['filter'])
• combine_group_jobs_list(
• query_results,
• template_params['x_params'],
• template_params['compare_dimensions'],
• template_params['compare_metrics']
• )

+end diff --git a/sbin/compare b/sbin/compare index cf3980c..b13682e 100755 --- a/sbin/compare +++ b/sbin/compare @@ -6,6 +6,7 @@ # Usage: # compare "conditions_1" "conditions_2" ... -c "common_conditions" # compare "conditions" -d "dimensions" +# compare -t template.yaml # Eg: # compare "id=6000,6001" "id=7000,7001" # compare "commit=a12d232e" "commit=b3bacc31" @@ -21,6 +22,7 @@ is_group = false dimensions = nil colorful = nil options = {} +template = nil

opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: compare "conditions" ... [option]' @@ -45,6 +47,10 @@ opt_parser = OptionParser.new do |opts| colorful = color end

• opts.on('-t', '--template template', 'compare with user-defined template') do |t|
• template = t
• end
• opts.on_tail('-h', '--help', 'show this message') do puts opts exit

@@ -60,7 +66,9 @@ opt_parser.parse!(argv)

options = { theme: colorful } if colorful

-if is_group +if template

• compare_by_template(template)

+elsif is_group compare_group(argv, dimensions, options) else compare_matrices_list(argv, common_conditions, options) -- 2.23.0