- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] logging: fix time difference of eight hours
by Wu Zhende 12 Dec '20

12 Dec '20

If the time zone information is not contained, the ES saves time as the UTC time by default. As a result, the kibana display time is eight hours later. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 2 +- src/lib/json_logger.cr | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index b3daeb4..7bf2229 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -11,7 +11,7 @@ @type record_transformer enable_ruby <record> - time ${time.strftime('%Y-%m-%dT%H:%M:%S')} + time ${time.strftime('%Y-%m-%dT%H:%M:%S.%3N+0800')} </record> </filter> diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index dd87020..6d61197 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -18,7 +18,7 @@ class JSONLogger < Logger Logger::Formatter.new do | severity, datetime, progname, msg, io| get_env_info(@env.as(HTTP::Server::Context)) if @env level_num = severity.to_i32 - datetime = datetime.to_s("%Y-%m-%dT%H:%M:%S") + datetime = datetime.to_s("%Y-%m-%dT%H:%M:%S.%3N+0800") logger_hash = JSON.parse(%({"level_num": #{level_num}, "level": "#{severity}", "time": "#{datetime}" -- 2.23.0

1 0

[PATCH compass-ci] README.md: modify the link address
by Li Ping 11 Dec '20

11 Dec '20

1 0

[PATCH compass-ci] container/defconfig.rb: split functions
by Liu Yinsi 11 Dec '20

11 Dec '20

split into 2 functions. function 1 -> meminfo_hash: create a more generic function by returning a hash, function 2 -> get_available_memory: create a dedicated function for use by es and logging-es container Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index d353bb3..482dbc8 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -34,10 +34,14 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end +def meminfo_hash + File.readlines('/proc/meminfo').map { |s| k, v, kb = s.split; { k.chomp(':') => v } } +end + def get_available_memory - memory = File.readlines('/proc/meminfo')[0].chomp.split[1].to_f / 1048576 + memtotal = meminfo_hash[0]['MemTotal'].to_f / 1048576 # set container available memory size, minimum size is 1024m, maximum size is 30720m, # take the middle value according to the system memory size. - [1024, 30720, Math.sqrt(memory) * 1024].sort[1].to_i + [1024, 30720, Math.sqrt(memtotal) * 1024].sort[1].to_i end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container/defconfig.rb: limit container memory usage
by Liu Yinsi 11 Dec '20

11 Dec '20

use Math.sqrt(memory) to set container occupied memory according to system memory size, instead of setting fixed size, more reasonable. In addition, '-Xms' not support floating number, so use to_i. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index 05c56db..026a183 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -35,3 +35,11 @@ def docker_rm(container) system "docker stop #{container} && docker rm -f #{container}" end + +def set_available_memory + memory = `awk '($1 == "MemTotal:"){print $2/1048576}' /proc/meminfo`.to_f + + # set container available memory size, minimum size is 1g, maximum size is 30g, + # take the middle value according to the system memory size. + [1, 30, Math.sqrt(memory)].sort[1].to_i +end -- 2.23.0

2 2

[PATCH compass-ci] README.zh.md: modify the link address
by Li Ping 11 Dec '20

11 Dec '20

1 0

[PATCH lkp-tests] bin/lkp-setup-rootfs: modify test-machine is not restarted
by Hu Xuejiao 11 Dec '20

11 Dec '20

[why] When force_reboot is not added to the job.yaml, the test-machine is "no KERNEL found". Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- bin/lkp-setup-rootfs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 0eabea75..cca1b53b 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -61,7 +61,10 @@ job_does_not_need_reboot() job_force_reboot() { - grep -q "^force_reboot: \+1$" $job + grep -q "^force_reboot: 1$" $job && return 0 + grep -q "^do_not_reboot: 1$" $job || return 0 + + return 1 } reboot_for_next_job() -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix syntax errors caused by missing double quotes
by Lin Jiaxin 11 Dec '20

11 Dec '20

error: PKGBUILD.src: line 2: syntax error near unexpected token `&' PKGBUILD.src: line 2: `glassfish-5.1.0.zip::http://www.eclipse.org/downloads/download.php?file=/glassfish/glassfish-5.1.0.zip&r=1' PKGBUILD.src: line 3: glassfish5.service: command not found PKGBUILD.src: line 4: syntax error near unexpected token `)' PKGBUILD.src: line 4: `)' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 8a138055..0b2794b1 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -112,7 +112,7 @@ replace_source() } url=$(echo "$url" | sed 's|https://|http://|g') - echo "$url" >> $1 + echo "\"$url\"" >> $1 done echo ")" >> $1 } -- 2.23.0

3 3

[PATCH v2 lkp-tests] tests/build-pkg: fix syntax errors caused by missing quotes
by Lin Jiaxin 11 Dec '20

11 Dec '20

error: PKGBUILD.src: line 2: syntax error near unexpected token `&' PKGBUILD.src: line 2: `glassfish-5.1.0.zip::http://www.eclipse.org/downloads/download.php?file=/glassfish/glassfish-5.1.0.zip&r=1' PKGBUILD.src: line 3: glassfish5.service: command not found PKGBUILD.src: line 4: syntax error near unexpected token `)' PKGBUILD.src: line 4: `)' Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 8a138055..aeaf1778 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -112,7 +112,7 @@ replace_source() } url=$(echo "$url" | sed 's|https://|http://|g') - echo "$url" >> $1 + echo \'$url\' >> $1 done echo ")" >> $1 } -- 2.23.0

1 0

[PATCH compass-ci 2/2] add options for compare_by_template
by Lu Kaiyi 11 Dec '20

11 Dec '20

if theme is json like below: compare -t compare_template.yaml --theme json display json format output result, otherwise, display table format output result by default. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 11 ++++++++--- sbin/compare | 2 +- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index f34e239..db264f8 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -96,11 +96,11 @@ end # unit: KB/s # -def compare_by_template(template) +def compare_by_template(template, options) template_params = load_template(template) groups_matrices = create_groups_matrices(template_params) compare_results = compare_metrics_values(groups_matrices) - show_compare_result(compare_results, template_params) + show_compare_result(compare_results, template_params, options) end def load_template(template) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index f7bb56a..db102f1 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -507,11 +507,16 @@ def get_dimensions_combination(dimension_list) dims end -def show_compare_result(metrics_compare_results, template_params) +def show_compare_result(metrics_compare_results, template_params, options) + theme = options[:theme] formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - table_data = FormatTableData.new(echart_results) - table_data.show_table + if theme == 'json' + print JSON.pretty_generate(echart_results) + else + table_data = FormatTableData.new(echart_results) + table_data.show_table + end end # Format Fields diff --git a/sbin/compare b/sbin/compare index c13ec1a..e8d1ac4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -67,7 +67,7 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful if template - compare_by_template(template) + compare_by_template(template, options) elsif is_group compare_group(argv, dimensions, options) else -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: friendly show compare template result in command line
by Lu Kaiyi 11 Dec '20

11 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the json format result of "compare -t <compare_template.yaml>" with terminal-table in Ruby. [usage] compare -t <compare_template.yaml> [example] compare -t compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | Hackbench Performance Testing | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | openeuler 20.03 average | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | openeuler 20.03 standard_deviation | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | Hackbench Performance Testing | +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ | centos 7.6 average | 345243.03 | 142698.79 | 62108.35 | 34747.73 | 26330.19 | 10317.85 | 7471.71 | 3558.30 | | centos 7.6 standard_deviation | 97.00 | 95.00 | 122.00 | 125.00 | 100.00 | 130.00 | 101.00 | 103.00 | +-------------------------------+-----------+-----------+----------+----------+----------+----------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 53 ++++++++++++++++++++++++++++++++++++++ lib/compare_matrixes.rb | 3 ++- 2 files changed, 55 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index bf4013b..aa5179f 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'terminal-table' # ---------------------------------------------------------------------------------------------------- # format compare results for a specific format # @@ -66,6 +67,58 @@ class FormatEchartData end end +class FormatTableData + def initialize(result_hash) + @title = result_hash['title'] + @tables = result_hash['tables'] + end + + def show_table + @tables.each do |table_title, table| + @tb = Terminal::Table.new + set_field_names(table_title, table) + set_table_title + add_row(table) + set_align_column + print_table + end + end + + def set_table_title + @tb.title = @title + end + + def set_field_names(table_title, table) + field_names = [table_title] + field_names.concat(table['average']['source'][0]) + @tb.add_row(field_names) + @tb.add_separator + end + + def add_row(table) + row_names = %w[average standard_deviation] + row_names.each do |row_name| + row = table[row_name]['source'][1] + row_title = [row[0], row_name].join(' ') + format_data_row = row[1..-1] + format_data_row.map! { |data| format('%.2f', data) } + @tb.add_row([row_title, *format_data_row]) + end + end + + def set_align_column + @tb.number_of_columns.times do |index| + @tb.align_column(index + 1, :right) + end + @tb.align_column(0, :left) + end + + def print_table + puts @tb + puts + end +end + # input: x_params_list # eg: ["1G|4K", "1G|1024k", "1G|128k", 2G|4k] # output: diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 5293989..f7bb56a 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -510,7 +510,8 @@ end def show_compare_result(metrics_compare_results, template_params) formatter = FormatEchartData.new(metrics_compare_results, template_params) echart_results = formatter.format_for_echart - print JSON.pretty_generate(echart_results) + table_data = FormatTableData.new(echart_results) + table_data.show_table end # Format Fields -- 2.23.0

1 0