- Compass-ci - mailweb.openeuler.org

[PATCH v6 lkp-tests] spec/submit_spec.rb: submit merge_yaml/*.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

We will use 'rspec spec/submit_spec.rb' to merge job.yaml, it support merging from external file by "<<: job.yaml" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- .../merge_yaml/merge_included_files.yaml | 1 + .../merge_included_files-defaults.yaml | 19 +++++++++++++++++++ spec/submit/merge_yaml/template_yaml | 3 +++ spec/submit_spec.rb | 4 ++++ 4 files changed, 27 insertions(+) create mode 100644 spec/submit/merge_yaml/merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml create mode 100644 spec/submit/merge_yaml/template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml new file mode 100644 index 00000000..8d459a32 --- /dev/null +++ b/spec/submit/merge_yaml/merge_included_files.yaml @@ -0,0 +1 @@ +<<: template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml new file mode 100644 index 00000000..42b53c13 --- /dev/null +++ b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml @@ -0,0 +1,19 @@ +--- +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit/merge_yaml/template_yaml b/spec/submit/merge_yaml/template_yaml new file mode 100644 index 00000000..331cb470 --- /dev/null +++ b/spec/submit/merge_yaml/template_yaml @@ -0,0 +1,3 @@ +suite: atomic + +atomic: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 169d7812..d71ed891 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -48,4 +48,8 @@ describe 'submit job spec' do it 'separate yaml spec' do submit_job('separate_yaml') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH v3 compass-ci] doc/manual: how to check the result files
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- ...\273\273\345\212\241\347\273\223\346\236\234" | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 "doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" diff --git "a/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" new file mode 100644 index 0000000..7a8cf8b --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" @@ -0,0 +1,16 @@ +# �� + ��https://compass-ci.openeuler.org/jobs�� + ��suite��id��job state�� + +# ��compass-ci�� +## job.yaml�� + job.yaml��job�� + �� + +## output�� + output��check_exit_code��0�� + +## stats.json�� + �� + compass-ci��.json�� + stats.json��json�� -- 2.23.0

1 0

[PATCH v3 compass-ci] doc/job: add runtime.md
by Zhang Yale 26 Nov '20

26 Nov '20

Add a document for introducing the runtime field. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/job/runtime.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 doc/job/runtime.md diff --git a/doc/job/runtime.md b/doc/job/runtime.md new file mode 100644 index 0000000..8e720f8 --- /dev/null +++ b/doc/job/runtime.md @@ -0,0 +1,32 @@ +# runtime + +Meaning: +- Run time for test: seconds(s), hours(h) or days(d). +- Limit the test's run time in $runtime. + +Usage example: + +```bash +#!/bin/sh +# - runtime + +## delay for a specified amount of time +echo sleep started +exec sleep ${1:-$runtime} +echo sleep finished +``` + +`runtime` is a parameter of test script. +For any test case needing parameter(s), +need such comment line and follow exactly the +same format `# - parameter`. + +```yaml +suite: borrow +testcase: borrow + +runtime: 1d + +sleep: +``` + -- 2.23.0

1 0

[PATCH v2 compass-ci] fill 0 if standard_deviation is nil
by Lu Weitao 26 Nov '20

26 Nov '20

[Why] With template compare mode, it's not friendly for echart that return null for nil standard_deviation of a metric like: { "tables": { "vmstat.system.cs": { "standard_deviation": { "dimensions": ["compare_dimension", "openeuler 20.03"], "source": [ ["4G", "8G", "16G"], ["openeuler 20.03", 23, null, null] ] } } } } [How] fill 0 if standard_deviation is nil. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 1550435..dd9b86e 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -461,7 +461,7 @@ def assign_metric_values(metrics_values, dim, metric, values) metrics_values[metric]['standard_deviation'] ||= {} metric_value = get_values(values, true) metrics_values[metric]['average'][dim] = metric_value[:average] - metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] || 0 end def assign_metric_change(metrics_values) -- 2.23.0

1 0

[PATCH compass-ci] doc/manual: how to check the result files
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- ...273\273\345\212\241\347\273\223\346\236\234" | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 "doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" diff --git "a/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" new file mode 100644 index 0000000..db6cc75 --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" @@ -0,0 +1,17 @@ +# �� + ��https://compass-ci.openeuler.org�� + ��id��job state�� + +## job.yaml�� + job.yaml��job�� + �� + +## output�� + output��check_exit_code��0�� + +## �� + �� + +## stats.json�� + ��xxx.json�� + stats.json��json�� -- 2.23.0

3 2

[PATCH v2 compass-ci] doc/manual: how to check the result files
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- ...3\273\345\212\241\347\273\223\346\236\234" | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 "doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" diff --git "a/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" new file mode 100644 index 0000000..855e9be --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" @@ -0,0 +1,19 @@ +# �� + ��https://compass-ci.openeuler.org�� + ��suite��id��job state�� + +# ��compass-ci�� +## job.yaml�� + job.yaml��job�� + �� + +## output�� + output��check_exit_code��0�� + +## �� + �� + ��iperf,��iperf��iperf�� + +## stats.json�� + ��xxx.json�� + stats.json��json�� -- 2.23.0

1 0

[PATCH compass-ci] kernel_version.md: update kernel information
by Xu Xijian 26 Nov '20

26 Nov '20

1. delete redundant soft links of kernel files I've created before and update this information in kernel_version.md. 2. the kernel version of centos/aarch64/7.6.1810 we use has been changed into 4.14.0-115.el7.0.1.aarch64. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md index 3e94ca8..4a193fb 100644 --- a/doc/job/kernel_version.md +++ b/doc/job/kernel_version.md @@ -6,10 +6,10 @@ Meaning: - If kernel_version is not given by users, it will use the default one. - Here are the kernel versions we support for different os: openeuler/aarch64/1.0 - - 4.19.90-2001(default) + - 4.19.90-vhulk2001.1.0.0026.aarch64(default) openeuler/aarch64/20.03 - - 4.19.90-2003(default) + - 4.19.90-2003.4.0.0036.oe1.aarch64(default) - 4.19.90-mysql - 4.19.90-nginx @@ -18,7 +18,7 @@ Meaning: - 5.8.0-1-arm64 centos/aarch64/7.6.1810 - - 4.14.0-115.el7a.0.1(default) + - 4.14.0-115.el7.0.1.aarch64(default) centos/aarch64/7.8.2003 - 4.18.0-147.8.1.el7(default) @@ -29,10 +29,10 @@ Meaning: Related files: - In initramfs boot process, every kernel version is related with a vmlinuz, module and headers. - Files like below under $boot_dir, an example $boot_dir can be "/srv/os/openeuler/aarch64/20.03/boot". -├── headers-4.19.90-2003.cgz -├── modules-4.19.90-2003.cgz -├── vmlinuz-4.19.90-2003 -└── vmlinuz -> vmlinuz-4.19.90-2003 +├── headers-4.19.90-2003.4.0.0036.oe1.aarch64.cgz +├── modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz +├── vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 +└── vmlinuz -> vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 Usage example: -- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 +- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003.4.0.0036.oe1.aarch64 -- 2.23.0

1 0

[PATCH compass-ci] fill 0 for nil standard_deviation with template compare mode
by Lu Weitao 26 Nov '20

26 Nov '20

[Why] With template compare mode, it's not friendly for echart that return null for nil standard_deviation of a metric like: { "tables": { "vmstat.system.cs": { "standard_deviation": { "dimensions": ["compare_dimension", "openeuler 20.03"], "source": [ ["4G", "8G", "16G"], ["openeuler 20.03", 23, null, null] ] } } } } [How] fill 0 for nil standard_deviation, instead of "null" Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 1550435..dd9b86e 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -461,7 +461,7 @@ def assign_metric_values(metrics_values, dim, metric, values) metrics_values[metric]['standard_deviation'] ||= {} metric_value = get_values(values, true) metrics_values[metric]['average'][dim] = metric_value[:average] - metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] || 0 end def assign_metric_change(metrics_values) -- 2.23.0

2 2

[PATCH v3 compass-ci] add compare.md
by Lu Weitao 26 Nov '20

26 Nov '20

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- doc/compare.md | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 doc/compare.md diff --git a/doc/compare.md b/doc/compare.md new file mode 100644 index 0000000..02161f0 --- /dev/null +++ b/doc/compare.md @@ -0,0 +1,49 @@ +# Introduction of compare feature + The compare feature used to analyze the results of different jobs, showing the performance waves and changes + under different influencing factors, for users to analyze performance influencing factor. + +## url: https://compass-ci.openeuler.org/compare + +## options directions +### filters: + - suite: iperf, netperf, mysql, ... + - OS: openeuler 20.03, centos 7.6, ... + - os_arch: aarch64, x86 + - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... + we can combine the above options arbitrarily to limit compare scope. + choose at least one option as filter. + +### dimension + Dimension can select: os, os_version, os_arch, suite, tbox_group. + Within filter, we will compare all different job result by dimension + and keep other test conditions are same. + +## example: + filter: suite = iperf + dimension: os_version + + result: +os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + + + 20.09 20.03 metric +-------------------- ------------------------------ ------------------------------ + fails:runs change fails:runs + | | | + 3:3 -100.0% 0:3 last_state.exit_fail + 3:3 -100.0% 0:3 last_state.is_incomplete_run + 3:3 -100.0% 0:3 last_state.test.iperf.exit_code.127 + 3:3 -100.0% 0:3 stderr.perf_command_failed + + + + 20.09 20.03 metric +-------------------- ------------------------------ ------------------------------ + %stddev change %stddev + \ | \ +4.461021e+10 ± 6% -17.4% 3.686392e+10 ± 12% iperf.tcp.receiver.bps +4.461112e+10 ± 6% -17.4% 3.686935e+10 ± 12% iperf.tcp.sender.bps + 94.82 -44.0% 53.10 boot-time.boot + 123.11 -58.4% 51.19 boot-time.idle + 0.00 0 4.87 boot-time.kernel_boot + 4165.50 ± 12% -99.9% 5.00 interrupts.38:GICv3.36.Level.virtio0 -- 2.23.0

1 0

[PATCH v2 compass-ci] add compare.md
by Lu Weitao 26 Nov '20

26 Nov '20

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- doc/compare.md | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 doc/compare.md diff --git a/doc/compare.md b/doc/compare.md new file mode 100644 index 0000000..94b139a --- /dev/null +++ b/doc/compare.md @@ -0,0 +1,49 @@ +# Introduction of compare feature + The compare feature used to analyze the results of different jobs, showing the performance waves and changes + under different influencing factors, for users to analyze performance influencing factor. + +## url: https://compass-ci.openeuler.org/compare + +## options directions +### filters: + - suite: iperf, netperf, mysql, ... + - OS: openeuler 20.03, centos 7.6, ... + - os_arch: aarch64, x86 + - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... + we can combine the above options arbitrarily to limit compare scope. + choose at least one option as filter. + +### dimension + Dimension can select: os, os_version, os_arch, suite, tbox_group. + Within filter, we will compare all different job result by dimension + and keep other test conditions are same. + +## example: + filter: suite = iperf | os_arch = aarch64 | tbox_group = vm-2p8g + dimension: os_version + + result: +os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + + + 20.09 20.03 metric +-------------------- ------------------------------ ------------------------------ + fails:runs change fails:runs + | | | + 3:3 -100.0% 0:3 last_state.exit_fail + 3:3 -100.0% 0:3 last_state.is_incomplete_run + 3:3 -100.0% 0:3 last_state.test.iperf.exit_code.127 + 3:3 -100.0% 0:3 stderr.perf_command_failed + + + + 20.09 20.03 metric +-------------------- ------------------------------ ------------------------------ + %stddev change %stddev + \ | \ +4.461021e+10 ± 6% -17.4% 3.686392e+10 ± 12% iperf.tcp.receiver.bps +4.461112e+10 ± 6% -17.4% 3.686935e+10 ± 12% iperf.tcp.sender.bps + 94.82 -44.0% 53.10 boot-time.boot + 123.11 -58.4% 51.19 boot-time.idle + 0.00 0 4.87 boot-time.kernel_boot + 4165.50 ± 12% -99.9% 5.00 interrupts.38:GICv3.36.Level.virtio0 -- 2.23.0

1 0

[PATCH compass-ci] add compare.md
by Lu Weitao 26 Nov '20

26 Nov '20

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- doc/compare.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 doc/compare.md diff --git a/doc/compare.md b/doc/compare.md new file mode 100644 index 0000000..24b8f73 --- /dev/null +++ b/doc/compare.md @@ -0,0 +1,46 @@ +# Introduction of compare feature + +## url: https://compass-ci.openeuler.org/compare + +## options directions +### filters: + - suite: iperf, netperf, mysql, ... + - OS: openeuler 20.03, centos 7.6, ... + - os_arch: aarch64, x86 + - tbox_group: vm-2p8g, taishan200-2880-2s48p-256g, ... + we can combine the above options arbitrarily to limit compare scope + +### dimension + dimension can select: os, os_version, os_arch, suite, box_group. + within compare scope, we will compare all defferent job result by dimension + and keep other test conditions are same. + +## example: + filter: suite=iperf | os_arch=aarch63 | tbox_group=2p8g + dimension: os_version + + result: +os=openeuler/os_arch=aarch64/pp.iperf.protocol=tcp/pp.iperf.runtime=20/tbox_group=vm-2p8g # other test conditions keep same + + + 20.09 20.03 metric +-------------------- ------------------------------ ------------------------------ + fails:runs change fails:runs + | | | + 3:3 -100.0% 0:3 last_state.exit_fail + 3:3 -100.0% 0:3 last_state.is_incomplete_run + 3:3 -100.0% 0:3 last_state.test.iperf.exit_code.127 + 3:3 -100.0% 0:3 stderr.perf_command_failed + + + + 20.09 20.03 metric +-------------------- ------------------------------ ------------------------------ + %stddev change %stddev + \ | \ +4.461021e+10 ± 6% -17.4% 3.686392e+10 ± 12% iperf.tcp.receiver.bps +4.461112e+10 ± 6% -17.4% 3.686935e+10 ± 12% iperf.tcp.sender.bps + 94.82 -44.0% 53.10 boot-time.boot + 123.11 -58.4% 51.19 boot-time.idle + 0.00 0 4.87 boot-time.kernel_boot + 4165.50 ± 12% -99.9% 5.00 interrupts.38:GICv3.36.Level.virtio0 -- 2.23.0

3 4

[PATCH v6 lkp-tests] spec/submit_spec.rb: submit merge_yaml/*.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

We will use 'rspec spec/submit_spec.rb' to merge job.yaml, it support merging from external file by "<<: job.yaml" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template_yaml | 3 +++ .../merge_yaml/yaml_merge_included_files.yaml | 1 + .../yaml_merge_included_files-defaults.yaml | 19 +++++++++++++++++++ spec/submit_spec.rb | 4 ++++ 4 files changed, 27 insertions(+) create mode 100644 spec/submit/merge_yaml/template_yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml diff --git a/spec/submit/merge_yaml/template_yaml b/spec/submit/merge_yaml/template_yaml new file mode 100644 index 00000000..331cb470 --- /dev/null +++ b/spec/submit/merge_yaml/template_yaml @@ -0,0 +1,3 @@ +suite: atomic + +atomic: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..8d459a32 --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template_yaml diff --git a/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml new file mode 100644 index 00000000..42b53c13 --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml @@ -0,0 +1,19 @@ +--- +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 169d7812..d71ed891 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -48,4 +48,8 @@ describe 'submit job spec' do it 'separate yaml spec' do submit_job('separate_yaml') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH v3 lkp-tests] spec/submit_spec.rb: submit job_on_fail.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

We will use 'rspec spec/submit_spec.rb' to debug job_on_fail.yaml, and after the task is failed, it is be waiting state. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/job_on_fail/job_on_fail.yaml | 5 +++++ .../job_on_fail/job_on_fail-3000.yaml | 22 +++++++++++++++++++ spec/submit_spec.rb | 4 ++++ 3 files changed, 31 insertions(+) create mode 100644 spec/submit/job_on_fail/job_on_fail.yaml create mode 100644 spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml diff --git a/spec/submit/job_on_fail/job_on_fail.yaml b/spec/submit/job_on_fail/job_on_fail.yaml new file mode 100644 index 00000000..bf68ff40 --- /dev/null +++ b/spec/submit/job_on_fail/job_on_fail.yaml @@ -0,0 +1,5 @@ +suite: atomic + +on_fail: + sleep: 3000 +atomic: diff --git a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml b/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml new file mode 100644 index 00000000..3d7ad9b5 --- /dev/null +++ b/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml @@ -0,0 +1,22 @@ +--- +suite: atomic +on_fail: + sleep: 3000 +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + sleep: 3000 + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 169d7812..52d9a158 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -48,4 +48,8 @@ describe 'submit job spec' do it 'separate yaml spec' do submit_job('separate_yaml') end + + it 'job on fail' do + submit_job('job_on_fail') + end end -- 2.23.0

1 0

[PATCH compass-ci] add a document for adding open source project
by Li Yuanchao 26 Nov '20

26 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- ...-compass-ci-to-test-open-source-project.md | 51 +++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 doc/how-to-use-compass-ci-to-test-open-source-project.md diff --git a/doc/how-to-use-compass-ci-to-test-open-source-project.md b/doc/how-to-use-compass-ci-to-test-open-source-project.md new file mode 100644 index 0000000..219cd0c --- /dev/null +++ b/doc/how-to-use-compass-ci-to-test-open-source-project.md @@ -0,0 +1,51 @@ +### 首先，将想要测试的代码仓信息以yaml格式文件的形式添加到upstream-repos仓库 + +仓库地址为：https://gitee.com/wu_fengguang/upstream-repos。 + +你可以先fork这个仓库，新建文件之后，通过Pull Request的方式完成添加。 + +以仓库backlight为例： + +它的url为：https://github.com/baskerville/backlight +1. 以首字母和仓库名创建路径 +``` +mkdir -p b/backlight +``` +2. 在该目录下新建同名文件backlight： +``` +cd b/backlight +touch backlight +``` +3. 使用vim写入url： +``` +vim backlight +``` +内容格式为 +``` +--- +url: +- https://github.com/baskerville/backlight +``` +请保持格式一致，若不确定，可参考upstream-repos仓库中已有文件。 + +pkgbuild_repo字段暂不需要，因为自助添加pkgbuild的功能尚未开放。 + +### 然后，将测试case以yaml文件的形式添加到lkp-tests仓库的jobs目录 + +仓库地址为 https://gitee.com/wu_fengguang/lkp-tests + +这其中有一些已经适配好的测试case，如果你想要测试的case其中正好有，那就可以直接使用。使用方式以iperf为例。 + +iperf是已经适配的一个测试case，在lkp-tests仓库的jobs下有一个iperf.yaml文件，里面有一些测试的基本参数。 + +然后，你要在compass-ci仓库下面的 sbin/auto_submit.yaml 文件中添加 +``` +b/backlight/backlight: +- testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml +``` +并通过Pull Request的方式提交。 + +参考文档： +- [compass-ci测试平台使用教程--submit命令详解](https://gitee.com/wu_fengguang/compass-ci/t… +- [如何适配测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-test… + -- 2.23.0

1 0

[PATCH compass-ci] doc: how to check the result files
by Wei Jihui 26 Nov '20

26 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- ...273\273\345\212\241\347\273\223\346\236\234" | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 "doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" diff --git "a/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" new file mode 100644 index 0000000..c4a2a55 --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\346\237\245\347\234\213\344\273\273\345\212\241\347\273\223\346\236\234" @@ -0,0 +1,17 @@ +# 如何查看任务结果 + 完测试用例执行完成后，可以登录到https://compass-ci.openeuler.org查看用例的结果。 + 在网站上可以根据任务id等内容搜索到自己的任务，点击任务对应的job state列即可查看。 + +## job.yaml文件 + job.yaml文件中部分字段是用户提交上来的，还有一部分字段是根据提交的job自动添加进来。 + 此文件包含了测试任务需要的所有参数。 + +## output文件 + output文件记录了用例的执行过程，文件最后部分一般会有check_exit_code这个状态码，非0时代表测试用例错误。 + +## 测试项结果文件 + 测试过程中不仅会执行测试用例，还可能会监控系统信息，这些测试项会生成一个与测试项同名的文件，记录它们命令与输出。 + +## stats.json文件 + 对于测试项结果文件，会对每一个文件进行提取，生成与文件同名的xxx.json文件。 + stats.json是所有的json文件的汇总。 -- 2.23.0

2 1

[PATCH v2 compass-ci 4/4] assign-account: attach new direcroty for the container
by Luan Shengde 26 Nov '20

26 Nov '20

[why] enable execute command: usermod -s shell to enable/disable login permission Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/start | 16 ++-------------- 1 file changed, 2 insertions(+), 14 deletions(-) diff --git a/container/assign-account/start b/container/assign-account/start index 2290982..793191a 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -. $CCI_SRC/container/defconfig.sh +. ../defconfig.sh docker_rm assign-account @@ -11,8 +11,7 @@ cmd=( --restart=always --name=assign-account -d - -v /etc/passwd:/etc/passwd:ro - -v /etc/group:/etc/group:ro + -v /etc:/etc:rw -v /srv/account-info:/opt/account_data/:rw -v /home:/home:rw -p 29999:29999 @@ -20,14 +19,3 @@ cmd=( ) "${cmd[@]}" - -cat << EOF -port: 29999 ---- -Client: -./assign-tool.rb email_file - -email subject: "apply ssh account" - -Refer to compass-ci/doc/apply-ssh-account.md for more information. -EOF -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] assign-account: build email message for updating configuration
by Luan Shengde 26 Nov '20

26 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../assign-account/build-update-email.rb | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 container/assign-account/build-update-email.rb diff --git a/container/assign-account/build-update-email.rb b/container/assign-account/build-update-email.rb new file mode 100755 index 0000000..40d6b3b --- /dev/null +++ b/container/assign-account/build-update-email.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_update_message(email, account_info, conf_info) + message = if conf_info['enable_login'] + + <<~EMAIL_MESSAGE + To: #{email} + Subject: [compass-ci] account update notice + + Dear user: + + Your account has been update successfully. + You can use the following command to login the server. + + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} + + regards + compass-ci + EMAIL_MESSAGE + else + <<~EMAIL_MESSAGE + To: #{email} + Subject: [compass-ci] account expiration notice + + Dear user: + + your account has expired. + consult the manager for help if you want to continue using the account. + + regards + compass-ci + EMAIL_MESSAGE + end + + return message +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] assign-account: add new function and optimize
by Luan Shengde 26 Nov '20

26 Nov '20

[why] meet new options for answerback-email meet update configuration function for answerback-email [how] add new function for the new options for answerback-email optimize current function to enable update Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 117 +++++++++++++++---- 1 file changed, 92 insertions(+), 25 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 0e91057..95bec14 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -29,11 +29,14 @@ API: call graph: setup_jumper_account_info - read_account_info - build_account_name + read account + read_account_info + build_account_info + reread_account_info read_jumper_info config_default_yaml config_authorized_key + permit_login_config generate_ssh_key the returned data for setup_jumper_account_info like: @@ -41,12 +44,14 @@ the returned data for setup_jumper_account_info like: "my_login_name" => "login_name", "password" => "password", "jumper_host" => "0.0.0.0", - "jumper_port" => "10000" + "jumper_port" => "10000", + "my_jumper_pubkey" => my_jumper_pubkey } =end require 'fileutils' +require 'yaml' # get jumper and account info class AccountStorage @@ -95,14 +100,21 @@ class AccountStorage end def setup_jumper_account_info - login_name, password = read_account_info + if @data.key?('is_update_account') && @data['is_update_account'] + login_name = @data['my_login_name'] + password = reread_account_info + else + login_name, password = read_account_info + end + jumper_host, jumper_port = read_jumper_info - pub_key = @data['my_ssh_pubkey'][0] - ssh_dir = File.join('/home/', login_name, '.ssh') - config_authorized_key(login_name, pub_key, ssh_dir) unless pub_key.nil? + config_authorized_key(login_name) config_default_yaml(login_name) - my_jumper_pubkey = generate_ssh_key(login_name, ssh_dir) if @data['gen_sshkey'].eql? true + permit_login_config(login_name) + if @data.key?('gen_sshkey') && @data['gen_sshkey'] + my_jumper_pubkey = generate_ssh_key(login_name) + end jumper_account_info = { 'my_login_name' => login_name, @@ -115,34 +127,89 @@ class AccountStorage return jumper_account_info end - def generate_ssh_key(login_name, ssh_dir) + def permit_login_config(login_name) + if @data.key?('enable_login') && @data['enable_login'] + %x(usermod -s /usr/bin/zsh #{login_name}) + else + %x(usermod -s /sbin/nologin #{login_name}) + end + end + + def generate_ssh_key(login_name) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + pub_key_file = File.join(ssh_dir, 'id_rsa.pub') + + return if File.exist?(pub_key_file) && \ + @data['my_ssh_pubkey'].include?(File.read(pub_key_file).strip) + %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@account-vm") - %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) - File.read("/home/#{login_name}/.ssh/id_rsa.pub") + + FileUtils.chown_R(login_name, login_name, ssh_dir) + File.read("/home/#{login_name}/.ssh/id_rsa.pub").strip end - def config_default_yaml(login_name) + def touch_default_yaml(login_name) default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') - FileUtils.mkdir_p default_yaml_dir + FileUtils.mkdir_p default_yaml_dir unless File.exist? default_yaml_dir + + default_yaml = File.join(default_yaml_dir, 'account.yaml') + FileUtils.touch default_yaml unless File.exist? default_yaml + default_yaml + end + def config_default_yaml(login_name) + default_yaml = touch_default_yaml(login_name) + + account_yaml = YAML.load_file(default_yaml) || {} # my_email, my_name, my_uuid is required to config default yaml file # they are added along with 'my_ssh_pubkey' when sending assign account request - File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| - file.puts "my_email: #{@data['my_email']}" - file.puts "my_name: #{@data['my_name']}" - file.puts "my_uuid: #{@data['my_uuid']}" - end - %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + account_yaml['my_email'] = @data['my_email'] + account_yaml['my_name'] = @data['my_name'] + account_yaml['my_uuid'] = @data['my_uuid'] + + f = File.new(default_yaml, 'w') + f.puts account_yaml.to_yaml + f.close + FileUtils.chown_R(login_name, login_name, "/home/#{login_name}/.config") + end + + def config_authorized_key(login_name) + pub_key = @data['my_ssh_pubkey'][0] + + return if pub_key.nil? + return if pub_key.strip.end_with?('account-vm') + + ssh_dir = File.join('/home/', login_name, '.ssh') + Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + + authorized_file = File.join(ssh_dir, 'authorized_keys') + FileUtils.touch authorized_file unless File.exist? authorized_file + + store_pubkey(ssh_dir, login_name, authorized_file, pub_key) end - def config_authorized_key(login_name, pub_key, ssh_dir) - Dir.mkdir ssh_dir, 0o700 - Dir.chdir ssh_dir - f = File.new('authorized_keys', 'w') + def store_pubkey(ssh_dir, login_name, authorized_file, pub_key) + authorized_keys = File.read(authorized_file).split("\n") + + return if authorized_keys.include?(pub_key.strip) + + f = File.new(authorized_file, 'a') f.puts pub_key f.close - File.chmod 0o600, 'authorized_keys' - %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) + + FileUtils.chown_R(login_name, login_name, ssh_dir) + File.chmod 0o600, authorized_file + end + + def reread_account_info + my_login_name_file = File.join(@account_dir, 'assigned-users', @data['my_login_name']) + + message = "No such assigned account exists: #{my_login_name_file}." + raise message unless File.exist? my_login_name_file + + password = File.readlines(my_login_name_file)[0].chomp + + return password end end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] answerback-email: add new options
by Luan Shengde 26 Nov '20

26 Nov '20

add option for answerback-email tool -n|--name name specify my_name for user when assigning account -l|--login value enable/disable user login allowed values: y|Y|n|N -u|--update enable update configuration for assigned account -u option enable update for: my_name my_ssh_pubkey generate sshkey on jumper reset user's login permission Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 153 +++++++++++++++---- 1 file changed, 123 insertions(+), 30 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 5ddc451..40284e4 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -14,6 +14,7 @@ require 'optparse' require_relative '../defconfig' require_relative '../../lib/es_client' require_relative 'build-send-account-email' +require_relative 'build-update-email' names = Set.new %w[ JUMPER_HOST @@ -24,7 +25,7 @@ names = Set.new %w[ defaults = relevant_defaults(names) -JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 @@ -33,25 +34,61 @@ my_info = { 'my_email' => nil, 'my_name' => nil, 'my_commit_url' => nil, - 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => [], - 'gen_sshkey' => false + 'my_uuid' => nil, + 'my_login_name' => nil, + 'my_ssh_pubkey' => [] } -def init_info(mail_content, my_info) +# stdin_info is used to store infos added with option: +# -e email_address +# -n name +# -s pubkey_file +# email_info is used to store infos parsed from email_file with option: +# -f email_file +# my_info_es is used to store info read from ES when update user +# when use --update option +# stdin_info, email_info, my_info_es have different priority +# stdin_info > email_info > my_info_es +# when assigning account or update conf for account +# if they have the same key, my_info will use the value with higher priority. +# conf_info is used to store keys used to config the account +stdin_info = {} +email_info = {} +my_info_es = {} +conf_info = { + 'gen_sshkey' => false, + 'enable_login' => true, + 'is_update_account' => false +} + +def init_info(mail_content, email_info, my_info) my_info['my_email'] = mail_content.from[0] - my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + email_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') return if mail_content.attachments.empty? - my_info['my_ssh_pubkey'] << mail_content.attachments[0].body.decoded + email_info['new_email_pubkey'] = mail_content.attachments[0].body.decoded.strip +end + +def read_my_login_name(my_email, my_info_es) + my_account_info_str = %x(curl -XGET localhost:9200/accounts/_doc/#{my_email}) + my_account_info = YAML.safe_load my_account_info_str + message = "No such email found from the ES: #{my_email}" + raise message unless my_account_info['found'] + + my_info_es.update my_account_info['_source'] end options = OptionParser.new do |opts| - opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' - opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file] [-g|--gen-sshkey]\n" - opts.banner += " -e or -f is required\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] [-n|--name name] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-g|--gen-sshkey] [--login y|Y|n|N] [--update]\n" + opts.banner += ' answerback-mail.rb [-f|--raw-email email_file] ' + opts.banner += "[-g|--gen-sshkey] [--login y|Y|n|N] [--update]\n" + opts.banner += " -e|-f is required when applying account or updating account\n" + opts.banner += " -n is required when assigning account with -e\n" opts.banner += " -s is optional when use -e\n" - opts.banner += ' -g is optional, used to generate sshkey for user' + opts.banner += " -g is optional, used to generate sshkey for user\n" + opts.banner += " -u is required when updating an account\n" + opts.banner += ' -l is optional, used to enable/disable login permission' opts.separator '' opts.separator 'options:' @@ -60,18 +97,37 @@ options = OptionParser.new do |opts| my_info['my_email'] = email_address end + opts.on('-n name', '--name name', 'appoint name') do |name| + stdin_info['my_name'] = name + end + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ 'ssh pub_key file, enable password-less login') do |pub_key_file| - my_info['my_ssh_pubkey'] << File.read(pub_key_file).chomp + stdin_info['new_ssh_pubkey'] = File.read(pub_key_file).strip end opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| mail_content = Mail.read(email_file) - init_info(mail_content, my_info) + init_info(mail_content, email_info, my_info) end - opts.on('-g', '--gen-sshkey', 'generate jumper ras public/private key and return pubkey') do - my_info['gen_sshkey'] = true + opts.on('-g', '--gen-sshkey', 'generate jumper rsa public/private key and return pubkey') do + conf_info['gen_sshkey'] = true + end + + opts.on('-u', '--update', 'updata configurations') do + read_my_login_name(my_info['my_email'], my_info_es) + conf_info['is_update_account'] = true + end + + opts.on('-l value', '--login value', 'enable/disable login, value: y|Y|n|N') do |value| + if value.downcase == 'y' + conf_info['enable_login'] = true + elsif value.downcase == 'n' + conf_info['enable_login'] = false + else + raise 'invalid parameter, please use y|Y|n|N' + end end opts.on_tail('-h', '--help', 'show this message') do @@ -82,33 +138,70 @@ end options.parse!(ARGV) -def apply_account(my_info) - account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') +def apply_account(my_info, conf_info) + apply_info = {} + apply_info.update my_info + apply_info.update conf_info + + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') JSON.parse account_info_str end -def send_account(my_info) +def check_my_email(my_info) message = "No email address specified\n" - message += "use -e to add a email address\n" - message += 'or use -f to add a email file' - raise message if my_info['my_email'].nil? + message += "use -e to add an email address for applying account\n" + message += 'or use -f to add an email file' - account_info = apply_account(my_info) - my_info['my_login_name'] = account_info['my_login_name'] + raise message if my_info['my_email'].empty? +end +def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) + new_email_pubkey = email_info.delete 'new_email_pubkey' + new_stdin_pubkey = stdin_info.delete 'new_ssh_pubkey' + new_pubkey = new_stdin_pubkey || new_email_pubkey + + my_info.update my_info_es unless my_info_es.empty? + my_info.update email_info unless email_info.empty? + my_info.update stdin_info unless stdin_info.empty? + + return if new_pubkey.nil? + return if my_info['my_ssh_pubkey'].include? new_pubkey + + my_info['my_ssh_pubkey'].insert(0, new_pubkey) +end + +def build_my_info_from_account_info(my_info, account_info, conf_info) unless account_info['my_jumper_pubkey'].nil? - my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'].chomp + return if my_info['my_ssh_pubkey'][-1] == account_info['my_jumper_pubkey'] + + my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] end - my_info.delete 'gen_sshkey' + my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] +end + +def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) + check_my_email(my_info) + my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] + build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) + + message = 'No my_name found, please use -n to add one' + raise message if my_info['my_name'].nil? + + account_info = apply_account(my_info, conf_info) + build_my_info_from_account_info(my_info, account_info, conf_info) + store_account_info(my_info) - send_mail(my_info, account_info) + send_mail(my_info, account_info, conf_info) end -def send_mail(my_info, account_info) - message = build_message(my_info['my_email'], account_info) - +def send_mail(my_info, account_info, conf_info) + message = if conf_info['is_update_account'] + build_update_message(my_info['my_email'], account_info, conf_info) + else + build_message(my_info['my_email'], account_info) + end %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end @@ -117,4 +210,4 @@ def store_account_info(my_info) es.put_source_by_id(my_info['my_email'], my_info) end -send_account(my_info) +send_account(my_info, conf_info, email_info, my_info_es, stdin_info) -- 2.23.0

1 0

[PATCH compass-ci] sched: fix cast from Nil to Hash(k, v) failed
by Cao Xueliang 26 Nov '20

26 Nov '20

The program {} => Nil in the commit 4301d1ece960dd0eff1f2e10c2d967e0e6150643 of lkp-tests, sched need to adapt to this case. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 378fd00..d88aa40 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -460,11 +460,11 @@ class Job private def get_program_params program_params = Hash(String, JSON::Any).new - if @hash["monitors"]? + if @hash["monitors"]? != nil program_params.merge!(@hash["monitors"].as_h) end - if @hash["pp"]? + if @hash["pp"]? != nil program_params.merge!(@hash["pp"].as_h) end -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: fix when to regard the job as a cluster job
by Ren Wen 26 Nov '20

26 Nov '20

If a job has a cluster field and its' value is not "cs-localhost" or empty, will regard it as a cluster job. When running a cluster job with cluster value "localhost", its' actions depend on field "node_roles", default is "server client". Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 8838e40..2836275 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -9,10 +9,8 @@ class Sched job = Job.new(job_content, job_content["id"]?) job["commit_date"] = get_commit_date(job) - # it is not a cluster job if cluster field is empty or - # field's prefix is 'cs-localhost' cluster_file = job["cluster"] - if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") + if cluster_file.empty? || cluster_file == "cs-localhost" return submit_single_job(job) else cluster_config = get_cluster_config(cluster_file, -- 2.23.0

1 0

[PATCH v5 lkp-tests] spec/submit_spec.rb: submit merge_yaml/*.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

We will use 'rspec spec/submit_spec.rb' to merge job.yaml, it support merging from external file by "<<: job.yaml" Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 3 +++ .../template/template-defaults.yaml | 19 +++++++++++++++++++ .../merge_yaml/yaml_merge_included_files.yaml | 1 + .../yaml_merge_included_files-defaults.yaml | 19 +++++++++++++++++++ spec/submit_spec.rb | 4 ++++ 5 files changed, 46 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/template/template-defaults.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..331cb470 --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,3 @@ +suite: atomic + +atomic: diff --git a/spec/submit/merge_yaml/template/template-defaults.yaml b/spec/submit/merge_yaml/template/template-defaults.yaml new file mode 100644 index 00000000..42b53c13 --- /dev/null +++ b/spec/submit/merge_yaml/template/template-defaults.yaml @@ -0,0 +1,19 @@ +--- +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml diff --git a/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml new file mode 100644 index 00000000..42b53c13 --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml @@ -0,0 +1,19 @@ +--- +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 169d7812..d71ed891 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -48,4 +48,8 @@ describe 'submit job spec' do it 'separate yaml spec' do submit_job('separate_yaml') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH compass-ci] update document: apply-ssh-account.md
by Luan Shengde 26 Nov '20

26 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/apply-ssh-account.md | 44 ++++++++-------------------------------- 1 file changed, 8 insertions(+), 36 deletions(-) diff --git a/doc/apply-ssh-account.md b/doc/apply-ssh-account.md index a4902f5..bb9ec76 100644 --- a/doc/apply-ssh-account.md +++ b/doc/apply-ssh-account.md @@ -1,35 +1,6 @@ +# apply account -# apply for jumper account - -## purpose - -The jumper account is used to login the jumper VM, where you can -- submit a job -- ssh into a testbox -- read output of the job - -## steps overview -### manually assign account -for internal user -1. run assign account tool - usage: - answerback-mail.rb [-e|--email email] [-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file] [-g|--gen-sshkey] - - login command: - ssh -p port account@server_ip - -2. login authentication - - use pub key - - use password (if no pub key) - -3. submit jobs - - prepare job.yaml - - submit job - - command: - submit job.yaml - -### apply account with email -for external user +## steps 1. send apply account email - send apply account email to: compass-ci(a)qq.com attention: @@ -52,17 +23,18 @@ for external user my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… # attach your ssh pub key to the email as attachment -2. successfully applied account email - it will return an email with following information: +2. receive email from compass-ci(a)qq.com + the email contains following information: - my_email - my_name - my_uuid - SCHED_HOST - SCHED_PORT -3. environment configuration +3. local environment configuration follow steps in the email to finish the following configuration - setup default yaml ~/.config/compass-ci/default/account.yaml - - download and install lkp-tests - - prepare to submit jobs + +for more information: how to use submit + https://gitee.com/openeuler/compass-ci/blob/master/doc/compass-ci测试平台使用教程--… -- 2.23.0

1 0

[PATCH v2 lkp-tests] spec/submit_spec.rb: submit debug_yaml/job_fail.yaml
by Hu Xuejiao 26 Nov '20

26 Nov '20

We will use 'rspec spec/submit_spec.rb' to debug job.yaml, and after the task is failed, it is be waiting state. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/debug_yaml/job_fail.yaml | 5 +++++ .../debug_yaml/job_fail/job_fail-3000.yaml | 22 +++++++++++++++++++ spec/submit_spec.rb | 4 ++++ 3 files changed, 31 insertions(+) create mode 100644 spec/submit/debug_yaml/job_fail.yaml create mode 100644 spec/submit/debug_yaml/job_fail/job_fail-3000.yaml diff --git a/spec/submit/debug_yaml/job_fail.yaml b/spec/submit/debug_yaml/job_fail.yaml new file mode 100644 index 00000000..16dcae3f --- /dev/null +++ b/spec/submit/debug_yaml/job_fail.yaml @@ -0,0 +1,5 @@ +suite: atomic + +on_fail: + sleep: 3000 +atomic: diff --git a/spec/submit/debug_yaml/job_fail/job_fail-3000.yaml b/spec/submit/debug_yaml/job_fail/job_fail-3000.yaml new file mode 100644 index 00000000..3d7ad9b5 --- /dev/null +++ b/spec/submit/debug_yaml/job_fail/job_fail-3000.yaml @@ -0,0 +1,22 @@ +--- +suite: atomic +on_fail: + sleep: 3000 +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + sleep: 3000 + atomic: + cpufreq_governor: +monitors: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 169d7812..d9b931e2 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -48,4 +48,8 @@ describe 'submit job spec' do it 'separate yaml spec' do submit_job('separate_yaml') end + + it 'debug yaml spec' do + submit_job('debug_yaml') + end end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/5] openeuler: configure files about build image with one command
by Wang Chenglong 26 Nov '20

26 Nov '20

[Contains] openeule-${os_version}.repo: yum configuration files files-to-exclude: list of files that need to be excluded packages-to-install: list of packages that need to install README.md: descriptive documentation Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/README.md | 15 +++++++ container/osimage/openeuler/files-to-exclude | 11 +++++ .../openeuler/openeuler-20.03-lts.repo | 43 +++++++++++++++++++ .../osimage/openeuler/openeuler-20.09.repo | 43 +++++++++++++++++++ .../osimage/openeuler/packages-to-install | 10 +++++ 5 files changed, 122 insertions(+) create mode 100644 container/osimage/openeuler/README.md create mode 100644 container/osimage/openeuler/files-to-exclude create mode 100644 container/osimage/openeuler/openeuler-20.03-lts.repo create mode 100644 container/osimage/openeuler/openeuler-20.09.repo create mode 100644 container/osimage/openeuler/packages-to-install diff --git a/container/osimage/openeuler/README.md b/container/osimage/openeuler/README.md new file mode 100644 index 0000000..cea68f9 --- /dev/null +++ b/container/osimage/openeuler/README.md @@ -0,0 +1,15 @@ +# Use the tool to create a new openeuler-${os_version} initramfs image. + +Usage: + cd ${CCI_SRC}/rootfs/initramfs/openeuler/aarch64/${os_version}/ + ./build + +Some configuration items: +./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + +./files-to-exclude + If you want remove some unnecessary files, you can write the names in ./files-to-exclude + +$HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. diff --git a/container/osimage/openeuler/files-to-exclude b/container/osimage/openeuler/files-to-exclude new file mode 100644 index 0000000..3854947 --- /dev/null +++ b/container/osimage/openeuler/files-to-exclude @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/files-to-exclude +/tmp/packages-to-install diff --git a/container/osimage/openeuler/openeuler-20.03-lts.repo b/container/osimage/openeuler/openeuler-20.03-lts.repo new file mode 100644 index 0000000..691f438 --- /dev/null +++ b/container/osimage/openeuler/openeuler-20.03-lts.repo @@ -0,0 +1,43 @@ +#generic-repos is licensed under the Mulan PSL v2. +#You can use this software according to the terms and conditions of the Mulan PSL v2. +#You may obtain a copy of Mulan PSL v2 at: +# http://license.coscl.org.cn/MulanPSL2 +#THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR +#IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR +#PURPOSE. +#See the Mulan PSL v2 for more details. + +[OS] +name=OS +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/OS/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/OS/$basearch/RPM-GPG-KEY-openEuler + +[everything] +name=everything +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/everything/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/everything/$basearch/RPM-GPG-KEY-openEuler + +[EPOL] +name=EPOL +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/EPOL/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/EPOL/$basearch/RPM-GPG-KEY-openEuler + +[debuginfo] +name=debuginfo +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/debuginfo/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/debuginfo/$basearch/RPM-GPG-KEY-openEuler + +[source] +name=source +baseurl=http://repo.openeuler.org/openEuler-20.03-LTS/source/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.03-LTS/source/RPM-GPG-KEY-openEuler diff --git a/container/osimage/openeuler/openeuler-20.09.repo b/container/osimage/openeuler/openeuler-20.09.repo new file mode 100644 index 0000000..1da7571 --- /dev/null +++ b/container/osimage/openeuler/openeuler-20.09.repo @@ -0,0 +1,43 @@ +#generic-repos is licensed under the Mulan PSL v2. +#You can use this software according to the terms and conditions of the Mulan PSL v2. +#You may obtain a copy of Mulan PSL v2 at: +# http://license.coscl.org.cn/MulanPSL2 +#THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR +#IMPLIED, INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY OR FIT FOR A PARTICULAR +#PURPOSE. +#See the Mulan PSL v2 for more details. + +[OS] +name=OS +baseurl=http://repo.openeuler.org/openEuler-20.09/OS/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/OS/$basearch/RPM-GPG-KEY-openEuler + +[everything] +name=everything +baseurl=http://repo.openeuler.org/openEuler-20.09/everything/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/everything/$basearch/RPM-GPG-KEY-openEuler + +[EPOL] +name=EPOL +baseurl=http://repo.openeuler.org/openEuler-20.09/EPOL/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/EPOL/$basearch/RPM-GPG-KEY-openEuler + +[debuginfo] +name=debuginfo +baseurl=http://repo.openeuler.org/openEuler-20.09/debuginfo/$basearch/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/debuginfo/$basearch/RPM-GPG-KEY-openEuler + +[source] +name=source +baseurl=http://repo.openeuler.org/openEuler-20.09/source/ +enabled=1 +gpgcheck=1 +gpgkey=http://repo.openeuler.org/openEuler-20.09/source/RPM-GPG-KEY-openEuler diff --git a/container/osimage/openeuler/packages-to-install b/container/osimage/openeuler/packages-to-install new file mode 100644 index 0000000..f17eee1 --- /dev/null +++ b/container/osimage/openeuler/packages-to-install @@ -0,0 +1,10 @@ +cpio +systemd-udev.aarch64 +cifs-utils +nfs-utils +hostname +curl +openssh-server +openssl +iputils +wget -- 2.23.0

2 2