- Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests 2/3] sbin/install-dependencies.sh: modify the way to add ruby env-variables
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/install-dependencies.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 7df3652f..3436ee44 100644 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -1,6 +1,7 @@ #!/bin/bash . lib/env.sh +. sbin/set-env.sh # choose install function base on # DISTRIBUTION @@ -30,8 +31,7 @@ esac mac_dep() { brew install ruby - echo "export PATH=/usr/local/opt/ruby/bin:$PATH" >> $(shell_profile) - source $(shell_profile) + write_shell_profile "export PATH=/usr/local/opt/ruby/bin:$PATH" gem install git activesupport rest-client } -- 2.23.0

1 0

[PATCH compass-ci 1/4] docker-rootfs: build rootfs with one command
by Wang Chenglong 09 Dec '20

09 Dec '20

generate rootfs through docker image with one command. we also can customize the pre-installed software by configuring ./packages-to-install Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/docker-rootfs/run diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run new file mode 100755 index 0000000..de9c3a7 --- /dev/null +++ b/container/docker-rootfs/run @@ -0,0 +1,32 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +source "$(dirname $(realpath $0))/common" + +root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + +get_host_arch +check_cmd_input "$@" +check_passwd_file "$root_pwd_file" +load_docker_img "$1" +echo $docker_name_tag +IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) + +start=( + docker run + --privileged=true + -v $RUN_DIR/setup-image:/usr/bin/setup-image + -v $RUN_DIR/packages-to-install:/tmp/packages-to-install + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e IMAGE_PACK=$IMAGE_PACK + --name rootfs-docker + $docker_name_tag + /usr/bin/setup-image +) + +"${start[@]}" + +cp_rootfs $IMAGE_PACK $2 +#docker rm -f rootfs-docker +create_links_vmlinuz_initrd $2 -- 2.23.0

4 6

[PATCH compass-ci] lib/json_logger: fix initializate without env
by Wu Zhende 09 Dec '20

09 Dec '20

[Error] In lib/json_logger.cr:9:61 9 | def initialize(logdev = STDOUT, formatter = my_formatter, @env = nil) Error: instance variable @env of JSONLogger was inferred to be Nil, but Nil alone provides no information This error is reported when "JSONLogger.new" is used. Fix it. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index 9596243..dd87020 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -4,6 +4,9 @@ require "logger" require "json" require "any_merge" +require "kemal" + +add_context_storage_type(Time::Span) class JSONLogger < Logger def initialize(logdev = STDOUT, formatter = my_formatter, @env = nil) -- 2.23.0

1 0

[PATCH compass-ci 2/4] docker-rootfs: setup rootfs in container
by Wang Chenglong 09 Dec '20

09 Dec '20

install software, set password and other actions in container. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/setup-image | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/docker-rootfs/setup-image diff --git a/container/docker-rootfs/setup-image b/container/docker-rootfs/setup-image new file mode 100755 index 0000000..0fa1eb6 --- /dev/null +++ b/container/docker-rootfs/setup-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +remove_file() +{ + rm -rf /.dockerenv +} + +setup_login() +{ + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find ./ ! -path "./tmp/${IMAGE_PACK}" | cpio -o -Hnewc | gzip -9 >./tmp/$IMAGE_PACK + chmod 644 /tmp/${IMAGE_PACK} +} + +install_packages +remove_file +setup_login +pack_cgz -- 2.23.0

2 2

[PATCH compass-ci] container/kibana: optimize logtrail
by Wu Zhende 09 Dec '20

09 Dec '20

1. show all ES index 2. set the maximum number of logs to be displayed to 10000 3. set the display of logs generated within 30 days Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/logtrail.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/kibana/logtrail.json b/container/kibana/logtrail.json index 304a4b3..c63f2ac 100644 --- a/container/kibana/logtrail.json +++ b/container/kibana/logtrail.json @@ -2,7 +2,7 @@ "version": 2, "index_patterns": [{ "es": { - "default_index": "scheduler-3016", + "default_index": "*", "allow_url_parameter": false, "timezone": "CST" }, @@ -12,9 +12,9 @@ "display_timestamp_format": "YYYY MMM DD HH:mm:ss", "max_buckets": 500, "nested_objects": false, - "default_time_range_in_days": 0, + "default_time_range_in_days": 30, "max_hosts": 100, - "max_events_to_keep_in_viewer": 5000, + "max_events_to_keep_in_viewer": 10000, "default_search": "", "fields": { "mapping": { -- 2.23.0

1 0

[PATCH lkp-tests 4/5] sbin/set-env.sh: take apart the method of adding env-variables
by Hu Xuejiao 09 Dec '20

09 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/set-env.sh | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/sbin/set-env.sh b/sbin/set-env.sh index a8b161fa..ce1ad427 100644 --- a/sbin/set-env.sh +++ b/sbin/set-env.sh @@ -2,10 +2,17 @@ . lib/env.sh -write_shellrc() +submit_env() +{ + echo $@ >> $(shell_profile) + source $(shell_profile) +} + +write_env() { echo "export LKP_SRC=$PWD" >> $(shell_profile) echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $(shell_profile) + source $(shell_profile) } write_host() @@ -28,5 +35,5 @@ write_host() EOF } -write_shellrc +write_env write_host -- 2.23.0

4 3

[PATCH v3 compass-ci 2/3] lib/compare_data_format.rb: fix NoMethodError
by Lu Kaiyi 09 Dec '20

09 Dec '20

[why] @metrics_compare_results[x_param][metric][value_type] maybe nil, if so, @metrics_compare_results[x_param][metric][value_type][dimension] will get the error like below: /home/lukaiyi/compass-ci/lib/compare_data_format.rb:57:in `block (2 levels) in assign_echart_source': undefined method `[]' for nil:NilClass (NoMethodError) from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:55:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:55:in `block in assign_echart_source' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:53:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:53:in `assign_echart_source' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:46:in `block in assign_echart_data_set' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:39:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:39:in `assign_echart_data_set' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:30:in `block (2 levels) in convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:29:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:29:in `block in convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:28:in `each_value' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:28:in `convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:21:in `format_for_echart' from /home/lukaiyi/compass-ci/lib/compare_matrixes.rb:512:in `show_compare_result' from /home/lukaiyi/compass-ci/lib/compare.rb:103:in `compare_by_template' from /home/lukaiyi/compass-ci/sbin/compare:70:in `<main>' [how] also judge the value of @metrics_compare_results[x_param][metric][value_type] before use. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index 1959045..bf4013b 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -53,7 +53,7 @@ class FormatEchartData dimensions.each do |dimension| dimension_values = [dimension] @x_params.each do |x_param| - if @metrics_compare_results[x_param][metric] + if @metrics_compare_results[x_param][metric] && @metrics_compare_results[x_param][metric][value_type] dimension_values << @metrics_compare_results[x_param][metric][value_type][dimension] else source[0].delete(x_param) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] sbin/compare_template: friendly show compare template result in command line
by Lu Kaiyi 09 Dec '20

09 Dec '20

[why] friendly show compare template result in command line. [how] rehandle the json format result of "compare -t <compare_template.yaml>" with third-party module in python3. [usage] compare_template <compare_template.yaml> [example] compare_template compare_template.yaml in z9 [input] a compare_template.yaml like below: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s [output] Hackbench Performance Testing +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | fio.read_iops | 4k|1G | 4k|80G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ | openeuler 20.03 average | 144076.29 | 11601.10 | 37865.30 | 21145.10 | 14010.34 | 6701.24 | 3205.08 | 1367.48 | 673.33 | | openeuler 20.03 standard_deviation | 195.00 | 0.00 | 214.00 | 205.00 | 188.00 | 183.00 | 180.00 | 191.00 | 191.00 | +------------------------------------+-----------+----------+----------+----------+----------+---------+---------+---------+----------+ Hackbench Performance Testing +------------------------------------+-----------+----------+----------+----------+---------+---------+---------+----------+ | fio.write_iops | 4k|1G | 16k|1G | 32k|1G | 64k|1G | 128k|1G | 256k|1G | 512k|1G | 1024k|1G | +------------------------------------+-----------+----------+----------+----------+---------+---------+---------+----------+ | openeuler 20.03 average | 122003.54 | 33528.53 | 31469.06 | 13870.14 | 8249.71 | 4329.45 | 1976.54 | 1141.00 | | openeuler 20.03 standard_deviation | 174.00 | 188.00 | 171.00 | 197.00 | 181.00 | 175.00 | 170.00 | 176.00 | +------------------------------------+-----------+----------+----------+----------+---------+---------+---------+----------+ Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- sbin/compare_template | 53 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 sbin/compare_template diff --git a/sbin/compare_template b/sbin/compare_template new file mode 100755 index 0000000..6b702c0 --- /dev/null +++ b/sbin/compare_template @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +import os +import sys +import json +import prettytable as pt + +# receive compare_template.yaml and auto pretty show compare results +class TableShow: + def __init__(self, result_dict): + self.title = result_dict['title'] + self.tables = result_dict['tables'] + self.tb = None + + def show_table(self): + for (table_title, table) in self.tables.items(): + self.tb = pt.PrettyTable() + self.set_field_names(table, table_title) + self.set_align(table_title) + self.add_row(table) + self.print_table() + + def set_field_names(self, table, table_title): + field_names = [table_title] + field_names.extend(table['average']['source'][0]) + self.tb.field_names = field_names + + def set_align(self, table_title): + for field_name in self.tb.field_names: + self.tb.align[field_name] = 'r' + self.tb.align[table_title] = 'l' + + def add_row(self, table): + row_names = ['average', 'standard_deviation'] + for row_name in row_names: + row = table[row_name]['source'][1] + row_title = ' '.join([row[0], row_name]) + format_data_row = ["%.2f" % data for data in row[1:]] + self.tb.add_row([row_title, *format_data_row]) + + def print_table(self): + print(self.title) + print(self.tb) + print() + + +if __name__ == '__main__': + template_yaml = sys.argv[1] + result_dict = json.loads(os.popen(f"compare -t {template_yaml}").read()) + table_show = TableShow(result_dict) + table_show.show_table() -- 2.23.0

1 0

[PATCH v3 compass-ci] src/lib: add delete function in job.cr
by Zhang Yu 09 Dec '20

09 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- src/lib/job.cr | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index a4345e0..196516f 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -136,6 +136,15 @@ class Job update(json.as_h) end + def delete(key : String) + initialized_keys = get_initialized_keys + if initialized_keys.includes?(key) + raise "Should not delete #{key}" + else + @hash.delete(key) + end + end + private def set_defaults append_init_field() set_user_lkp_src() @@ -324,7 +333,7 @@ class Job @hash.delete("my_uuid") end - private def initialized? + private def get_initialized_keys() initialized_keys = [] of String REQUIRED_KEYS.each do |key| @@ -350,7 +359,10 @@ class Job "kernel_uri", "kernel_params", "linux_vmlinuz_path"] + end + private def initialized? + initialized_keys = get_initialized_keys initialized_keys.each do |key| return false unless @hash.has_key?(key) end -- 2.23.0

2 2

[PATCH compass-ci] lib/compare_data_format.rb: fix NoMethodError
by Lu Kaiyi 09 Dec '20

09 Dec '20

[error] /home/lukaiyi/compass-ci/lib/compare_data_format.rb:57:in `block (2 levels) in assign_echart_source': undefined method `[]' for nil:NilClass (NoMethodError) from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:55:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:55:in `block in assign_echart_source' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:53:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:53:in `assign_echart_source' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:46:in `block in assign_echart_data_set' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:39:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:39:in `assign_echart_data_set' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:30:in `block (2 levels) in convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:29:in `each' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:29:in `block in convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:28:in `each_value' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:28:in `convert_to_echart_dataset' from /home/lukaiyi/compass-ci/lib/compare_data_format.rb:21:in `format_for_echart' from /home/lukaiyi/compass-ci/lib/compare_matrixes.rb:512:in `show_compare_result' from /home/lukaiyi/compass-ci/lib/compare.rb:103:in `compare_by_template' from /home/lukaiyi/compass-ci/sbin/compare:70:in `<main>' Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index 1959045..bf4013b 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -53,7 +53,7 @@ class FormatEchartData dimensions.each do |dimension| dimension_values = [dimension] @x_params.each do |x_param| - if @metrics_compare_results[x_param][metric] + if @metrics_compare_results[x_param][metric] && @metrics_compare_results[x_param][metric][value_type] dimension_values << @metrics_compare_results[x_param][metric][value_type][dimension] else source[0].delete(x_param) -- 2.23.0

2 1