- Compass-ci - mailweb.openeuler.org

[PATCH v4 lkp-tests] ispec/submit_spec.rb: update submit_job
by Hu Xuejiao 03 Dec '20

03 Dec '20

[why] Originally submit_job is too complicated, it need to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/job_on_fail/job-3000.yaml | 52 +++++++++++ .../{job_on_fail.yaml => job.yaml} | 1 + .../job_on_fail/job_on_fail-3000.yaml | 22 ----- .../link_job1/job-1000-2-100000ss-add.yaml | 57 ++++++++++++ .../link_job1/job-10000-2-100000ss-add.yaml | 57 ++++++++++++ .../link_job1/job-20-2-6000ss-cache.yaml | 57 ++++++++++++ .../submit/link_job1/job.yaml | 2 - ...68000000-1-65535-set_get-never-never.yaml} | 88 +++++++------------ .../submit/link_job2/job.yaml | 2 - spec/submit/link_jobs/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 29 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 29 ------ .../link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 29 ------ spec/submit/link_jobs/redis.yaml | 1 - .../matrix/job-1-300-whetstone-double.yaml | 55 ++++++++++++ spec/submit/matrix/job-3-300-shell1.yaml | 55 ++++++++++++ .../matrix/{unixbench-spec.yaml => job.yaml} | 0 ...unixbench-spec-1-300-whetstone-double.yaml | 25 ------ .../unixbench-spec-3-300-shell1.yaml | 25 ------ spec/submit/merge_yaml/job-defaults.yaml | 49 +++++++++++ spec/submit/merge_yaml/job.yaml | 1 + .../merge_yaml/merge_included_files.yaml | 1 - .../merge_included_files-defaults.yaml | 19 ---- .../ftq/ftq-1000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 ------ .../job-1000-2-100000ss-add.yaml | 57 ++++++++++++ .../job-10000-2-100000ss-add.yaml | 57 ++++++++++++ .../separate_yaml/job-20-2-6000ss-cache.yaml | 57 ++++++++++++ .../separate_yaml/{ftq.yaml => job.yaml} | 0 spec/submit_spec.rb | 40 ++------- 31 files changed, 596 insertions(+), 353 deletions(-) create mode 100644 spec/submit/job_on_fail/job-3000.yaml rename spec/submit/job_on_fail/{job_on_fail.yaml => job.yaml} (97%) delete mode 100644 spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml create mode 100644 spec/submit/link_job1/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/link_job1/job-20-2-6000ss-cache.yaml rename jobs/ftq.yaml => spec/submit/link_job1/job.yaml (80%) rename spec/submit/{link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml => link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml} (57%) rename jobs/redis.yaml => spec/submit/link_job2/job.yaml (92%) delete mode 120000 spec/submit/link_jobs/ftq.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml delete mode 120000 spec/submit/link_jobs/redis.yaml create mode 100644 spec/submit/matrix/job-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrix/job-3-300-shell1.yaml rename spec/submit/matrix/{unixbench-spec.yaml => job.yaml} (100%) delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml create mode 100644 spec/submit/merge_yaml/job-defaults.yaml create mode 100644 spec/submit/merge_yaml/job.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml create mode 100644 spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml rename spec/submit/separate_yaml/{ftq.yaml => job.yaml} (100%) diff --git a/spec/submit/job_on_fail/job-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml new file mode 100644 index 00000000..5f41e885 --- /dev/null +++ b/spec/submit/job_on_fail/job-3000.yaml @@ -0,0 +1,52 @@ +--- + +#! spec/submit/job_on_fail/job.yaml +suite: atomic +on_fail: + sleep: 3000 +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/job_on_fail/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + sleep: 3000 + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/job_on_fail/job_on_fail.yaml b/spec/submit/job_on_fail/job.yaml similarity index 97% rename from spec/submit/job_on_fail/job_on_fail.yaml rename to spec/submit/job_on_fail/job.yaml index bf68ff40..5aaedf30 100644 --- a/spec/submit/job_on_fail/job_on_fail.yaml +++ b/spec/submit/job_on_fail/job.yaml @@ -2,4 +2,5 @@ suite: atomic on_fail: sleep: 3000 + atomic: diff --git a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml b/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml deleted file mode 100644 index 3d7ad9b5..00000000 --- a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml +++ /dev/null @@ -1,22 +0,0 @@ ---- -suite: atomic -on_fail: - sleep: 3000 -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - sleep: 3000 - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/link_job1/job-1000-2-100000ss-add.yaml b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..ba732182 --- /dev/null +++ b/spec/submit/link_job1/job-1000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-10000-2-100000ss-add.yaml b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..4b6bbacc --- /dev/null +++ b/spec/submit/link_job1/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_job1/job-20-2-6000ss-cache.yaml b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..9476bf3c --- /dev/null +++ b/spec/submit/link_job1/job-20-2-6000ss-cache.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_job1/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/jobs/ftq.yaml b/spec/submit/link_job1/job.yaml similarity index 80% rename from jobs/ftq.yaml rename to spec/submit/link_job1/job.yaml index 64287a8a..209f065d 100644 --- a/jobs/ftq.yaml +++ b/spec/submit/link_job1/job.yaml @@ -1,6 +1,4 @@ suite: ftq -testcase: ftq -category: noise-benchmark nr_task: 100% diff --git a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 57% rename from spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 466a2054..e1f2bbf6 100644 --- a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_job2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -1,7 +1,7 @@ --- + +#! spec/submit/link_job2/job.yaml suite: redis -testcase: redis -category: benchmark need_memory: 62G nr_threads: 2 numactl: @@ -27,43 +27,43 @@ if role client: key_len: 68000000 lab: spec_lab testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_job2/job.yaml tbox_group: vm-2p8g--spec_submit node_roles: server client + +#! hosts/vm-2p8g provider: qemu template: kvm nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -kmsg: -boot-time: -uptime: -iostat: -heartbeat: -vmstat: -numa-numastat: -numa-vmstat: -numa-meminfo: -proc-vmstat: -proc-stat: -meminfo: -slabinfo: -interrupts: -lock_stat: -latency_stats: -softirqs: -bdi_dev_mapping: -diskstats: -nfsstat: -cpuidle: -cpufreq-stats: -sched_debug: -perf-stat: -mpstat: -perf-profile: -cpufreq_governor: + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() pp: numactl: all: 1 @@ -85,31 +85,7 @@ pp: key_len: 68000000 nr_processes: 4 cpu_node_bind: even - cpufreq_governor: + cpufreq_governor: + +#! auto generated by add_monitors() monitors: - kmsg: - boot-time: - uptime: - iostat: - heartbeat: - vmstat: - numa-numastat: - numa-vmstat: - numa-meminfo: - proc-vmstat: - proc-stat: - meminfo: - slabinfo: - interrupts: - lock_stat: - latency_stats: - softirqs: - bdi_dev_mapping: - diskstats: - nfsstat: - cpuidle: - cpufreq-stats: - sched_debug: - perf-stat: - mpstat: - perf-profile: diff --git a/jobs/redis.yaml b/spec/submit/link_job2/job.yaml similarity index 92% rename from jobs/redis.yaml rename to spec/submit/link_job2/job.yaml index 0049d60f..3694716f 100644 --- a/jobs/redis.yaml +++ b/spec/submit/link_job2/job.yaml @@ -1,6 +1,4 @@ suite: redis -testcase: redis -category: benchmark need_memory: 62G diff --git a/spec/submit/link_jobs/ftq.yaml b/spec/submit/link_jobs/ftq.yaml deleted file mode 120000 index b472f871..00000000 --- a/spec/submit/link_jobs/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index 9a613c00..00000000 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 8bf52d0f..00000000 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index 8cb78287..00000000 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/redis.yaml b/spec/submit/link_jobs/redis.yaml deleted file mode 120000 index 438bdab7..00000000 --- a/spec/submit/link_jobs/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit/matrix/job-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml new file mode 100644 index 00000000..ae23f10c --- /dev/null +++ b/spec/submit/matrix/job-1-300-whetstone-double.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/job-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml new file mode 100644 index 00000000..90e20c6b --- /dev/null +++ b/spec/submit/matrix/job-3-300-shell1.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/job.yaml similarity index 100% rename from spec/submit/matrix/unixbench-spec.yaml rename to spec/submit/matrix/job.yaml diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml deleted file mode 100644 index 9896bcef..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '1' -test: whetstone-double -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: whetstone-double - nr_task: '1' - cpufreq_governor: -monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml deleted file mode 100644 index 912d10ee..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '3' -test: shell1 -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: shell1 - nr_task: '3' - cpufreq_governor: -monitors: diff --git a/spec/submit/merge_yaml/job-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml new file mode 100644 index 00000000..4168a6ef --- /dev/null +++ b/spec/submit/merge_yaml/job-defaults.yaml @@ -0,0 +1,49 @@ +--- + +#! spec/submit/merge_yaml/job.yaml +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/merge_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/merge_yaml/job.yaml b/spec/submit/merge_yaml/job.yaml new file mode 100644 index 00000000..6d64c381 --- /dev/null +++ b/spec/submit/merge_yaml/job.yaml @@ -0,0 +1 @@ +<< : template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml deleted file mode 100644 index 8d459a32..00000000 --- a/spec/submit/merge_yaml/merge_included_files.yaml +++ /dev/null @@ -1 +0,0 @@ -<<: template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml deleted file mode 100644 index 42b53c13..00000000 --- a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml +++ /dev/null @@ -1,19 +0,0 @@ ---- -suite: atomic -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index f3e2c553..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 63bac5cf..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index a802dc1b..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..4dcd179d --- /dev/null +++ b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..d5566714 --- /dev/null +++ b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..dd177379 --- /dev/null +++ b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/job.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq.yaml rename to spec/submit/separate_yaml/job.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..357e643f 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -14,46 +14,24 @@ def stable_yaml_file(yaml_file) save_yaml(job, yaml_file) end -def traverse_file(output_dir) - Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - stable_yaml_file(yaml_file) +def traverse_file(dir) + Dir.entries(dir) do |yaml_file| + stable_yaml_file(yaml_file) unless File.basename(yaml_file) == 'job.yaml' end end -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*").each do |dir| submit_cmd = [ "#{LKP_SRC}/sbin/submit", - '-o', output_dir, + '-o', dir, '-s', 'lab: spec_lab', '-s', 'testbox: vm-2p8g--spec_submit', - yaml_file + "#{dir}/job.yaml" ] system(*submit_cmd) - traverse_file(output_dir) + traverse_file(dir) end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0

1 0

[PATCH lkp-tests] jobs: rm my_email field and set "pub_key" field as level-1 parameter in borrow-*.yaml
by Zhang Yale 03 Dec '20

03 Dec '20

[Why] my_email field is seted in .config/compass-ci/default/account.yaml, and now the "pub_key" field is not placed under the "sshd" field in yaml. Use "pub_key" to determine whether to password-free login. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-10d.yaml | 15 +++++++-------- jobs/borrow-1d.yaml | 16 +++++++--------- 2 files changed, 14 insertions(+), 17 deletions(-) diff --git a/jobs/borrow-10d.yaml b/jobs/borrow-10d.yaml index aeb55466..a2b50025 100644 --- a/jobs/borrow-10d.yaml +++ b/jobs/borrow-10d.yaml @@ -1,14 +1,13 @@ suite: borrow testcase: borrow +pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> sshd: - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - my_email: # sleep at the bottom sleep: 10d diff --git a/jobs/borrow-1d.yaml b/jobs/borrow-1d.yaml index 72949982..7767f2d7 100644 --- a/jobs/borrow-1d.yaml +++ b/jobs/borrow-1d.yaml @@ -3,15 +3,13 @@ testcase: borrow timeout: 1d +pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> sshd: - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - my_email: - # sleep at the bottom sleep: 1d -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/mail_job_result.rb: fix NoMethodError
by Lu Weitao 03 Dec '20

03 Dec '20

[Why] /home/lwt/compass-ci/lib/mail_job_result.rb:45:in `query_job': undefined method `[]' for nil:NilClass (NoMethodError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/mail_job_result.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/mail_job_result.rb b/lib/mail_job_result.rb index 29a9c1a..50e2f53 100644 --- a/lib/mail_job_result.rb +++ b/lib/mail_job_result.rb @@ -33,7 +33,7 @@ class MailJobResult def set_submitter_info job = query_job - exit unless job['email'] + exit unless job && job['email'] @submitter_email = job['email'] @result_root = job['result_root'] @@ -42,6 +42,11 @@ class MailJobResult def query_job es = ESQuery.new query_result = es.multi_field_query({ 'id' => @job_id }) + if query_result['hits']['hits'].empty? + warn "Non-existent job: #{@job_id}" + return nil + end + query_result['hits']['hits'][0]['_source'] end end -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: add methods to search testboxes
by Li Yuanchao 03 Dec '20

03 Dec '20

Query all testboxes from es, and return a non repetitive list Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index e548d61..fe9545f 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -435,3 +435,31 @@ def template_body(request_body) formatter = FormatEchartData.new(compare_results, request_body) formatter.format_for_echart.to_json end + +def search_testboxes + query = { size: 0, aggs: { testboxes: { terms: { size: 10000, field: 'testbox' } } } } + buckets = es_query(query)['aggregations']['testboxes']['buckets'] + testboxes = [] + buckets.each_index do |index| + testboxes[index] = buckets[index]['key'] + end + return testboxes, testboxes.length +end + +def testboxes_body + testboxes, total = search_testboxes + { + total: total, + testboxes: testboxes + }.to_json +end + +def query_testboxes + begin + body = testboxes_body + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

2 2

[PATCH compass-ci] doc: change file name nr_threads->nr_threads.md
by Xu Xijian 03 Dec '20

03 Dec '20

Specify it as a markdown file. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/{nr_threads => nr_threads.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename doc/job/{nr_threads => nr_threads.md} (100%) diff --git a/doc/job/nr_threads b/doc/job/nr_threads.md similarity index 100% rename from doc/job/nr_threads rename to doc/job/nr_threads.md -- 2.23.0

1 0

[PATCH v2 compass-ci] container: change vmlinuz path
by Xu Xijian 03 Dec '20

03 Dec '20

Since vmlinuz path changed from $rootfs_dir/vmlinuz to $rootfs_dir/boot/vmlinuz, here keep consistent. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/update-os-docker/lib/common.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/update-os-docker/lib/common.rb b/container/update-os-docker/lib/common.rb index 23c4fdc..5b1353d 100755 --- a/container/update-os-docker/lib/common.rb +++ b/container/update-os-docker/lib/common.rb @@ -27,7 +27,7 @@ end def check_argv(argv) usage(argv) rootfs_dir = Pathname.new(OS_PATH + argv[0]).realpath.to_s - raise 'Wrong vmlinuz path' unless File.exist?(rootfs_dir + '/vmlinuz') + raise 'Wrong vmlinuz path' unless File.exist?(rootfs_dir + '/boot/vmlinuz') return rootfs_dir end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: set ENV DOCKER_REGISTRY_HOST default value
by Liu Yinsi 03 Dec '20

03 Dec '20

$DOCKER_REGISTRY_HOST default value should keep consistent with sparrow/3-code/dev-env, use server ip. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2753b9d..74905f1 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -26,7 +26,9 @@ cat >> /etc/modules-load.d/cifs <<EOF cifs EOF -: ${DOCKER_REGISTRY_HOST:="127.0.0.1"} +server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') + +: ${DOCKER_REGISTRY_HOST:=$server_ip} : ${DOCKER_REGISTRY_PORT:=5001} cat > /etc/docker/daemon.json <<EOF { -- 2.23.0

1 0

[PATCH v3 compass-ci] answerback-email: add execute tool limit
by Luan Shengde 03 Dec '20

03 Dec '20

limit the scope of executing answerback-email to 'z9' [why] when execute answerback-email, the account info will be stored to the es installed at the localhost. It will cause validation to fail when submit a job if the data is not at the server 'z9'. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 31 +++++++++++++++++--- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 825d815..53c321d 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -187,11 +187,14 @@ def apply_account(my_info, conf_info) end def check_my_email(my_info) + return true if my_info['my_email'] + message = "No email address specified\n" message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' + puts message - raise message if my_info['my_email'].nil? + return false end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) @@ -219,13 +222,32 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] end +def check_server + return true if ENV['HOSTNAME'] == 'z9' + + message = 'please run the tool on z9 server' + puts message + + return false +end + +def check_my_name_exist(my_info) + return true if my_info['my_name'] + + message = 'No my_name found, please use -n to add one' + puts message + + return false +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) - check_my_email(my_info) + return unless check_server + return unless check_my_email(my_info) + my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) - message = 'No my_name found, please use -n to add one' - raise message if my_info['my_name'].nil? + return unless check_my_name_exist(my_info) account_info = apply_account(my_info, conf_info) build_my_info_from_account_info(my_info, account_info, conf_info) @@ -241,6 +263,7 @@ def send_mail(my_info, account_info, conf_info) else build_message(my_info['my_email'], account_info) end + %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/es_query.rb: add traverse es data of function
by Cui Yili 03 Dec '20

03 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- lib/es_query.rb | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index c027aa9..3827ea8 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -11,6 +11,7 @@ class ESQuery PORT = (ENV.key?('ES_PORT') ? ENV['ES_PORT'] : ES_PORT).to_i def initialize(host = HOST, port = PORT, index: 'jobs') @index = index + @scroll_id = '' @client = Elasticsearch::Client.new url: "http://#{host}:#{port}" raise 'Connect Elasticsearch error!' unless @client.ping end @@ -33,6 +34,25 @@ class ESQuery @client.search index: 'jobs*', body: query end + def traverse_field(size) + if @scroll_id.empty? + query = { + query: { + bool: { + must: { + match_all: {} + } + } + }, size: size + } + result = @client.search index: @index, scroll: '10m', body: query + @scroll_id = result['_scroll_id'] + return result + else + @client.scroll scroll: '10m', scroll_id: @scroll_id + end + end + def query_by_id(id) @client.get_source({ index: @index, type: '_doc', id: id }) rescue Elasticsearch::Transport::Transport::Errors::NotFound -- 2.23.0

1 0

[PATCH v2 compass-ci] doc/manual: 修改名字为如何登录测试机调测任务.md并增加和修改内容
by Zhang Yale 03 Dec '20

03 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- ...03\346\265\213\344\273\273\345\212\241.md" | 91 +++++++++++++------ 1 file changed, 65 insertions(+), 26 deletions(-) rename doc/manual/how-to-log-in-the-machine.md => "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" (56%) diff --git a/doc/manual/how-to-log-in-the-machine.md "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" similarity index 56% rename from doc/manual/how-to-log-in-the-machine.md rename to "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" index fefd718..22da3c5 100644 --- a/doc/manual/how-to-log-in-the-machine.md +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -1,17 +1,22 @@ -这篇文档将告诉你如何登陆测试环境 +这篇文档将告诉你如何登陆测试环境去调测任务 -# 1. 前提条件 -请先学习: +# 1. 前提条件： +## 请先学习: * [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 * [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 * [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 # 2. 操作方法 -## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： -```shell +## 2.1 如何修改yaml? +### 在测试机运行script之前进入测试机调测任务 + 可以在job.yaml里加上sshd和sleep字段，测试机在运行脚本之前sleep, + 并免密登录进去，手动输入命令或脚本进行调试，以host-info.yaml为例： + +```yaml suite: host-info category: functional + pub_key: <%= begin File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp @@ -19,6 +24,7 @@ nil end %> + sshd: # sleep at the bottom sleep: 1h @@ -26,31 +32,42 @@ host-info: ``` -## 2.2 以下有2种方式可以登录到测试机： -### 第一种：使用submit -m -c的方式： - 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 - 命令：submit -m -c host-info.yaml - 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： - 效果如下： +### 在测试任务运行失败时进入测试机调测任务 + 可以在job.yaml里加上pub_key和on_fail字段，并在on_fail下加上sshd和sleep字段, + 测试任务失败后，测试机保留并免密登录进去，以方便手动调试，以iperf.yaml为例： -```shell - hi6325@account-vm ~% submit -m -c atomic.yaml - submit atomic.yaml, got job_id=crystal.146528 - query=>{"job_id":["crystal.146528"]} - connect to ws://localhost:11310/filter - {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" - {"job_id": "crystal.146528", "job_state": "boot"} - {"job_id": "crystal.146528", "job_state": "download"} - "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} - {"job_state":"running","job_id":"crystal.146528"} - {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} +```yaml + suite: iperf + category: benchmark - root@vm-2p8g ~# + runtime: 300s + + cluster: cs-localhost + + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + + if role server: + iperf-server: + + if role client: + iperf: + protocol: + - tcp + - udp ``` -### 第二种：根据邮件信息使用ssh方式登录测试机： - 这种是使用submit方式提交的任务完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 - 命令： submit host-info.yaml +## 2.2 如何免密登录执行机？ +### 直接submit job.yaml或者submit加-m参数 + 可在收到的邮件里，按照邮件提示手动免密登录到执行机调测 + + 命令：submit iperf.yaml或者submit -m iperf.yaml + 任务执行完成后，系统发送邮件内容如下： ```shell @@ -84,3 +101,25 @@ root@vm-2p8g ~# ``` + +### submit job.yaml时加上-m -c + 该方法不用查看邮件，可自动免密登录到测试机： + + 命令：submit -m -c iperf.yaml + + 效果如下： + +```shell + hi6325@account-vm ~% submit -m -c iperf.yaml + submit iperf.yaml, got job_id=crystal.146528 + query=>{"job_id":["crystal.146528"]} + connect to ws://localhost:11310/filter + {"job_id": "crystal.146528", "result_root": "/srv/result/iperf/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "job_state": "boot"} + {"job_id": "crystal.146528", "job_state": "download"} + "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} + {"job_state":"running","job_id":"crystal.146528"} + {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} + + root@vm-2p8g ~# +``` -- 2.23.0

2 3