- Compass-ci - mailweb.openeuler.org

[PATCH v6 lkp-tests] spec/submit_spec.rb: submit merge_yaml/*.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

We will use 'rspec spec/submit_spec.rb' to merge job.yaml, it support merging from external file by "<<: job.yaml" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- .../merge_yaml/merge_included_files.yaml | 1 + .../merge_included_files-defaults.yaml | 19 +++++++++++++++++++ spec/submit/merge_yaml/template_yaml | 3 +++ spec/submit_spec.rb | 4 ++++ 4 files changed, 27 insertions(+) create mode 100644 spec/submit/merge_yaml/merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml create mode 100644 spec/submit/merge_yaml/template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml new file mode 100644 index 00000000..8d459a32 --- /dev/null +++ b/spec/submit/merge_yaml/merge_included_files.yaml @@ -0,0 +1 @@ +<<: template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml new file mode 100644 index 00000000..42b53c13 --- /dev/null +++ b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml @@ -0,0 +1,19 @@ +--- +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit/merge_yaml/template_yaml b/spec/submit/merge_yaml/template_yaml new file mode 100644 index 00000000..331cb470 --- /dev/null +++ b/spec/submit/merge_yaml/template_yaml @@ -0,0 +1,3 @@ +suite: atomic + +atomic: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 169d7812..d71ed891 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -48,4 +48,8 @@ describe 'submit job spec' do it 'separate yaml spec' do submit_job('separate_yaml') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH v3 compass-ci] doc/manual: how to check the result files
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- ...\273\273\345\212\241\347\273\223\346\236\234" | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 "doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" diff --git "a/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" new file mode 100644 index 0000000..7a8cf8b --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" @@ -0,0 +1,16 @@ +# �� + ��https://compass-ci.openeuler.org/jobs�� + ��suite��id��job state�� + +# ��compass-ci�� +## job.yaml�� + job.yaml��job�� + �� + +## output�� + output��check_exit_code��0�� + +## stats.json�� + �� + compass-ci��.json�� + stats.json��json�� -- 2.23.0

1 0

[PATCH v3 compass-ci] doc/job: add runtime.md
by Zhang Yale 26 Nov '20

26 Nov '20

Add a document for introducing the runtime field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/runtime.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 doc/job/runtime.md diff --git a/doc/job/runtime.md b/doc/job/runtime.md new file mode 100644 index 0000000..8e720f8 --- /dev/null +++ b/doc/job/runtime.md @@ -0,0 +1,32 @@ +# runtime + +Meaning: +- Run time for test: seconds(s), hours(h) or days(d). +- Limit the test's run time in $runtime. + +Usage example: + +```bash +#!/bin/sh +# - runtime + +## delay for a specified amount of time +echo sleep started +exec sleep ${1:-$runtime} +echo sleep finished +``` + +`runtime` is a parameter of test script. +For any test case needing parameter(s), +need such comment line and follow exactly the +same format `# - parameter`. + +```yaml +suite: borrow +testcase: borrow + +runtime: 1d + +sleep: +``` + -- 2.23.0

1 0

[PATCH v2 compass-ci] fill 0 if standard_deviation is nil
by Lu Weitao 26 Nov '20

26 Nov '20

[Why] With template compare mode, it's not friendly for echart that return null for nil standard_deviation of a metric like: { "tables": { "vmstat.system.cs": { "standard_deviation": { "dimensions": ["compare_dimension", "openeuler 20.03"], "source": [ ["4G", "8G", "16G"], ["openeuler 20.03", 23, null, null] ] } } } } [How] fill 0 if standard_deviation is nil. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 1550435..dd9b86e 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -461,7 +461,7 @@ def assign_metric_values(metrics_values, dim, metric, values) metrics_values[metric]['standard_deviation'] ||= {} metric_value = get_values(values, true) metrics_values[metric]['average'][dim] = metric_value[:average] - metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] || 0 end def assign_metric_change(metrics_values) -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: how to check the result files
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- ...273\273\345\212\241\347\273\223\346\236\234" | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 "doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" diff --git "a/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" new file mode 100644 index 0000000..db6cc75 --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" @@ -0,0 +1,17 @@ +# �� + ��https://compass-ci.openeuler.org�� + ��id��job state�� + +## job.yaml�� + job.yaml��job�� + �� + +## output�� + output��check_exit_code��0�� + +## �� + �� + +## stats.json�� + ��xxx.json�� + stats.json��json�� -- 2.23.0

3 2

[PATCH v2 compass-ci] doc/manual: how to check the result files
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- ...3\273\345\212\241\347\273\223\346\236\234" | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 "doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" diff --git "a/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" new file mode 100644 index 0000000..855e9be --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" @@ -0,0 +1,19 @@ +# �� + ��https://compass-ci.openeuler.org�� + ��suite��id��job state�� + +# ��compass-ci�� +## job.yaml�� + job.yaml��job�� + �� + +## output�� + output��check_exit_code��0�� + +## �� + �� + ��iperf,��iperf��iperf�� + +## stats.json�� + ��xxx.json�� + stats.json��json�� -- 2.23.0

1 0

[PATCH compass-ci] kernel_version.md: update kernel information
by Xu Xijian 26 Nov '20

26 Nov '20

1. delete redundant soft links of kernel files I've created before and update this information in kernel_version.md. 2. the kernel version of centos/aarch64/7.6.1810 we use has been changed into 4.14.0-115.el7.0.1.aarch64. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md index 3e94ca8..4a193fb 100644 --- a/doc/job/kernel_version.md +++ b/doc/job/kernel_version.md @@ -6,10 +6,10 @@ Meaning: - If kernel_version is not given by users, it will use the default one. - Here are the kernel versions we support for different os: openeuler/aarch64/1.0 - - 4.19.90-2001(default) + - 4.19.90-vhulk2001.1.0.0026.aarch64(default) openeuler/aarch64/20.03 - - 4.19.90-2003(default) + - 4.19.90-2003.4.0.0036.oe1.aarch64(default) - 4.19.90-mysql - 4.19.90-nginx @@ -18,7 +18,7 @@ Meaning: - 5.8.0-1-arm64 centos/aarch64/7.6.1810 - - 4.14.0-115.el7a.0.1(default) + - 4.14.0-115.el7.0.1.aarch64(default) centos/aarch64/7.8.2003 - 4.18.0-147.8.1.el7(default) @@ -29,10 +29,10 @@ Meaning: Related files: - In initramfs boot process, every kernel version is related with a vmlinuz, module and headers. - Files like below under $boot_dir, an example $boot_dir can be "/srv/os/openeuler/aarch64/20.03/boot". -├── headers-4.19.90-2003.cgz -├── modules-4.19.90-2003.cgz -├── vmlinuz-4.19.90-2003 -└── vmlinuz -> vmlinuz-4.19.90-2003 +├── headers-4.19.90-2003.4.0.0036.oe1.aarch64.cgz +├── modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz +├── vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 +└── vmlinuz -> vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 Usage example: -- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 +- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003.4.0.0036.oe1.aarch64 -- 2.23.0

1 0

[PATCH compass-ci] fill 0 for nil standard_deviation with template compare mode
by Lu Weitao 26 Nov '20

26 Nov '20

[Why] With template compare mode, it's not friendly for echart that return null for nil standard_deviation of a metric like: { "tables": { "vmstat.system.cs": { "standard_deviation": { "dimensions": ["compare_dimension", "openeuler 20.03"], "source": [ ["4G", "8G", "16G"], ["openeuler 20.03", 23, null, null] ] } } } } [How] fill 0 for nil standard_deviation, instead of "null" Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 1550435..dd9b86e 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -461,7 +461,7 @@ def assign_metric_values(metrics_values, dim, metric, values) metrics_values[metric]['standard_deviation'] ||= {} metric_value = get_values(values, true) metrics_values[metric]['average'][dim] = metric_value[:average] - metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] || 0 end def assign_metric_change(metrics_values) -- 2.23.0

2 2

[PATCH v3 compass-ci] add compare.md
by Lu Weitao 26 Nov '20

26 Nov '20

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- doc/compare.md | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 doc/compare.md diff --git a/doc/compare.md b/doc/compare.md new file mode 100644 index 0000000..02161f0 --- /dev/null +++ b/doc/compare.md @@ -0,0 +1,49 @@ +# Introduction of compare feature + The compare feature used to analyze the results of different jobs, showing the performance waves and changes + under different influencing factors, for users to analyze performance influencing factor. + +## url: https://compass-ci.openeuler.org/compare + +## options directions +### filters: + - suite: iperf, netperf, mysql, ... + - OS: openeuler 20.03, centos 7.6, ... + - os_arch: aarch64, x86 + - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... + we can combine the above options arbitrarily to limit compare scope. + choose at least one option as filter. + +### dimension + Dimension can select: os, os_version, os_arch, suite, tbox_group. + Within filter, we will compare all different job result by dimension + and keep other test conditions are same. + +## example: + filter: suite = iperf + dimension: os_version + + result: +os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + + + 20.09 20.03 metric +-------------------- ------------------------------ ------------------------------ + fails:runs change fails:runs + | | | + 3:3 -100.0% 0:3 last_state.exit_fail + 3:3 -100.0% 0:3 last_state.is_incomplete_run + 3:3 -100.0% 0:3 last_state.test.iperf.exit_code.127 + 3:3 -100.0% 0:3 stderr.perf_command_failed + + + + 20.09 20.03 metric +-------------------- ------------------------------ ------------------------------ + %stddev change %stddev + \ | \ +4.461021e+10 ± 6% -17.4% 3.686392e+10 ± 12% iperf.tcp.receiver.bps +4.461112e+10 ± 6% -17.4% 3.686935e+10 ± 12% iperf.tcp.sender.bps + 94.82 -44.0% 53.10 boot-time.boot + 123.11 -58.4% 51.19 boot-time.idle + 0.00 0 4.87 boot-time.kernel_boot + 4165.50 ± 12% -99.9% 5.00 interrupts.38:GICv3.36.Level.virtio0 -- 2.23.0

1 0

[PATCH v2 compass-ci] add compare.md
by Lu Weitao 26 Nov '20

26 Nov '20

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- doc/compare.md | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 doc/compare.md diff --git a/doc/compare.md b/doc/compare.md new file mode 100644 index 0000000..94b139a --- /dev/null +++ b/doc/compare.md @@ -0,0 +1,49 @@ +# Introduction of compare feature + The compare feature used to analyze the results of different jobs, showing the performance waves and changes + under different influencing factors, for users to analyze performance influencing factor. + +## url: https://compass-ci.openeuler.org/compare + +## options directions +### filters: + - suite: iperf, netperf, mysql, ... + - OS: openeuler 20.03, centos 7.6, ... + - os_arch: aarch64, x86 + - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... + we can combine the above options arbitrarily to limit compare scope. + choose at least one option as filter. + +### dimension + Dimension can select: os, os_version, os_arch, suite, tbox_group. + Within filter, we will compare all different job result by dimension + and keep other test conditions are same. + +## example: + filter: suite = iperf | os_arch = aarch64 | tbox_group = vm-2p8g + dimension: os_version + + result: +os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + + + 20.09 20.03 metric +-------------------- ------------------------------ ------------------------------ + fails:runs change fails:runs + | | | + 3:3 -100.0% 0:3 last_state.exit_fail + 3:3 -100.0% 0:3 last_state.is_incomplete_run + 3:3 -100.0% 0:3 last_state.test.iperf.exit_code.127 + 3:3 -100.0% 0:3 stderr.perf_command_failed + + + + 20.09 20.03 metric +-------------------- ------------------------------ ------------------------------ + %stddev change %stddev + \ | \ +4.461021e+10 ± 6% -17.4% 3.686392e+10 ± 12% iperf.tcp.receiver.bps +4.461112e+10 ± 6% -17.4% 3.686935e+10 ± 12% iperf.tcp.sender.bps + 94.82 -44.0% 53.10 boot-time.boot + 123.11 -58.4% 51.19 boot-time.idle + 0.00 0 4.87 boot-time.kernel_boot + 4165.50 ± 12% -99.9% 5.00 interrupts.38:GICv3.36.Level.virtio0 -- 2.23.0

1 0