November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sbin/es-account.sh: rename
by Lu Weitao 30 Nov '20

30 Nov '20

[Why] keep same style with es-jobs-mapping.sh, es-repo-mapping.sh [How] es-account.sh => es-accounts-mapping.sh Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/{es-account.sh => es-accounts-mapping.sh} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename sbin/{es-account.sh => es-accounts-mapping.sh} (100%) diff --git a/sbin/es-account.sh b/sbin/es-accounts-mapping.sh similarity index 100% rename from sbin/es-account.sh rename to sbin/es-accounts-mapping.sh -- 2.23.0

1 0

[PATCH lkp-tests] jobs/borrow-1h.yaml: rm my_email field
by Zhang Yale 30 Nov '20

30 Nov '20

[why] The "my_email" field must be written in "~/.config/compass-ci/default/$user.yaml", If It it not written, the job fails to be submitted. Therefore, this field is not used in borrow-1h.yaml. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/borrow-1h.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/jobs/borrow-1h.yaml b/jobs/borrow-1h.yaml index 6c2284f3..be22b94a 100644 --- a/jobs/borrow-1h.yaml +++ b/jobs/borrow-1h.yaml @@ -9,6 +9,5 @@ sshd: nil end %> - my_email: # sleep at the bottom sleep: 1h -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.sh: add function run_target_stats
by Wei Jihui 30 Nov '20

30 Nov '20

if user define his own stats and puts his stats file to tbox by use submit -a or other method, he should exec stats in test scripts by himself. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.sh | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/job.sh b/lib/job.sh index b84e6b18..03c49e0a 100755 --- a/lib/job.sh +++ b/lib/job.sh @@ -313,3 +313,10 @@ run_test() fi } +# user can run self-define stats by: +# run_target_stats $(basename $0) +run_target_stats() +{ + local script_name=$1 + $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.json +} -- 2.23.0

1 0

[PATCH v3 lkp-tests] spec/submit_spec.rb: update submit_job
by Hu Xuejiao 30 Nov '20

30 Nov '20

[why] Originally submit_job is too complicated, it need to be simplified. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/job_on_fail/job-3000.yaml | 52 +++++++++++ .../{job_on_fail.yaml => job.yaml} | 1 + .../job_on_fail/job_on_fail-3000.yaml | 22 ----- spec/submit/link_jobs/ftq.yaml | 1 - .../ftq/ftq-1000-2-100000ss-add.yaml | 29 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 29 ------ .../link_jobs/ftq/ftq-20-2-6000ss-cache.yaml | 29 ------ spec/submit/link_jobs/redis.yaml | 1 - .../link_jobs1/job-10000-2-100000ss-add.yaml | 57 ++++++++++++ spec/submit/link_jobs1/job.yaml | 8 ++ ...68000000-1-65535-set_get-never-never.yaml} | 90 +++++++------------ spec/submit/link_jobs2/job.yaml | 26 ++++++ .../matrix/job-1-300-whetstone-double.yaml | 55 ++++++++++++ spec/submit/matrix/job-3-300-shell1.yaml | 55 ++++++++++++ .../matrix/{unixbench-spec.yaml => job.yaml} | 5 +- ...unixbench-spec-1-300-whetstone-double.yaml | 25 ------ .../unixbench-spec-3-300-shell1.yaml | 25 ------ spec/submit/merge_yaml/job-defaults.yaml | 49 ++++++++++ spec/submit/merge_yaml/job.yaml | 1 + .../merge_yaml/merge_included_files.yaml | 1 - .../merge_included_files-defaults.yaml | 19 ---- spec/submit/merge_yaml/template_yaml | 2 +- .../ftq/ftq-1000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-10000-2-100000ss-add.yaml | 27 ------ .../ftq/ftq-20-2-6000ss-cache.yaml | 27 ------ .../job-1000-2-100000ss-add.yaml | 57 ++++++++++++ .../job-10000-2-100000ss-add.yaml | 57 ++++++++++++ .../separate_yaml/job-20-2-6000ss-cache.yaml | 57 ++++++++++++ .../separate_yaml/{ftq.yaml => job.yaml} | 0 spec/submit_spec.rb | 41 ++------- 30 files changed, 520 insertions(+), 355 deletions(-) create mode 100644 spec/submit/job_on_fail/job-3000.yaml rename spec/submit/job_on_fail/{job_on_fail.yaml => job.yaml} (97%) delete mode 100644 spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml delete mode 120000 spec/submit/link_jobs/ftq.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml delete mode 120000 spec/submit/link_jobs/redis.yaml create mode 100644 spec/submit/link_jobs1/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/link_jobs1/job.yaml rename spec/submit/{link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml => link_jobs2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml} (56%) create mode 100644 spec/submit/link_jobs2/job.yaml create mode 100644 spec/submit/matrix/job-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrix/job-3-300-shell1.yaml rename spec/submit/matrix/{unixbench-spec.yaml => job.yaml} (54%) delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml delete mode 100644 spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml create mode 100644 spec/submit/merge_yaml/job-defaults.yaml create mode 100644 spec/submit/merge_yaml/job.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files.yaml delete mode 100644 spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml delete mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml create mode 100644 spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml rename spec/submit/separate_yaml/{ftq.yaml => job.yaml} (100%) diff --git a/spec/submit/job_on_fail/job-3000.yaml b/spec/submit/job_on_fail/job-3000.yaml new file mode 100644 index 00000000..5f41e885 --- /dev/null +++ b/spec/submit/job_on_fail/job-3000.yaml @@ -0,0 +1,52 @@ +--- + +#! spec/submit/job_on_fail/job.yaml +suite: atomic +on_fail: + sleep: 3000 +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/job_on_fail/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + sleep: 3000 + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/job_on_fail/job_on_fail.yaml b/spec/submit/job_on_fail/job.yaml similarity index 97% rename from spec/submit/job_on_fail/job_on_fail.yaml rename to spec/submit/job_on_fail/job.yaml index bf68ff40..5aaedf30 100644 --- a/spec/submit/job_on_fail/job_on_fail.yaml +++ b/spec/submit/job_on_fail/job.yaml @@ -2,4 +2,5 @@ suite: atomic on_fail: sleep: 3000 + atomic: diff --git a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml b/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml deleted file mode 100644 index 3d7ad9b5..00000000 --- a/spec/submit/job_on_fail/job_on_fail/job_on_fail-3000.yaml +++ /dev/null @@ -1,22 +0,0 @@ ---- -suite: atomic -on_fail: - sleep: 3000 -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - sleep: 3000 - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq.yaml b/spec/submit/link_jobs/ftq.yaml deleted file mode 120000 index b472f871..00000000 --- a/spec/submit/link_jobs/ftq.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/ftq.yaml \ No newline at end of file diff --git a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index 9a613c00..00000000 --- a/spec/submit/link_jobs/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 8bf52d0f..00000000 --- a/spec/submit/link_jobs/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index 8cb78287..00000000 --- a/spec/submit/link_jobs/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,29 +0,0 @@ ---- -suite: ftq -testcase: ftq -category: noise-benchmark -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/link_jobs/redis.yaml b/spec/submit/link_jobs/redis.yaml deleted file mode 120000 index 438bdab7..00000000 --- a/spec/submit/link_jobs/redis.yaml +++ /dev/null @@ -1 +0,0 @@ -../../../jobs/redis.yaml \ No newline at end of file diff --git a/spec/submit/link_jobs1/job-10000-2-100000ss-add.yaml b/spec/submit/link_jobs1/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..a36d77de --- /dev/null +++ b/spec/submit/link_jobs1/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/link_jobs1/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_jobs1/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/link_jobs1/job.yaml b/spec/submit/link_jobs1/job.yaml new file mode 100644 index 00000000..5624b122 --- /dev/null +++ b/spec/submit/link_jobs1/job.yaml @@ -0,0 +1,8 @@ +suite: ftq + +nr_task: 100% +samples: 100000ss + +ftq: + test: add + freq: 10000 diff --git a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml b/spec/submit/link_jobs2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml similarity index 56% rename from spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml rename to spec/submit/link_jobs2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml index 466a2054..6e023d71 100644 --- a/spec/submit/link_jobs/redis/redis-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml +++ b/spec/submit/link_jobs2/job-1-cs-localhost-even-1024-68000000-5-3-4-68000000-1-65535-set_get-never-never.yaml @@ -1,7 +1,7 @@ --- + +#! spec/submit/link_jobs2/job.yaml suite: redis -testcase: redis -category: benchmark need_memory: 62G nr_threads: 2 numactl: @@ -16,7 +16,7 @@ cluster: cs-localhost cpu_node_bind: even nr_processes: 4 if role server: - redis-server: + redis-server: if role client: redis: test: set,get @@ -27,43 +27,43 @@ if role client: key_len: 68000000 lab: spec_lab testbox: vm-2p8g--spec_submit +job_origin: spec/submit/link_jobs2/job.yaml tbox_group: vm-2p8g--spec_submit node_roles: server client + +#! hosts/vm-2p8g provider: qemu template: kvm nr_node: 1 nr_cpu: 2 memory: 8G arch: aarch64 -kmsg: -boot-time: -uptime: -iostat: -heartbeat: -vmstat: -numa-numastat: -numa-vmstat: -numa-meminfo: -proc-vmstat: -proc-stat: -meminfo: -slabinfo: -interrupts: -lock_stat: -latency_stats: -softirqs: -bdi_dev_mapping: -diskstats: -nfsstat: -cpuidle: -cpufreq-stats: -sched_debug: -perf-stat: -mpstat: -perf-profile: -cpufreq_governor: + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: need_kconfig: - CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() pp: numactl: all: 1 @@ -85,31 +85,7 @@ pp: key_len: 68000000 nr_processes: 4 cpu_node_bind: even - cpufreq_governor: + cpufreq_governor: + +#! auto generated by add_monitors() monitors: - kmsg: - boot-time: - uptime: - iostat: - heartbeat: - vmstat: - numa-numastat: - numa-vmstat: - numa-meminfo: - proc-vmstat: - proc-stat: - meminfo: - slabinfo: - interrupts: - lock_stat: - latency_stats: - softirqs: - bdi_dev_mapping: - diskstats: - nfsstat: - cpuidle: - cpufreq-stats: - sched_debug: - perf-stat: - mpstat: - perf-profile: diff --git a/spec/submit/link_jobs2/job.yaml b/spec/submit/link_jobs2/job.yaml new file mode 100644 index 00000000..bf96e654 --- /dev/null +++ b/spec/submit/link_jobs2/job.yaml @@ -0,0 +1,26 @@ +suite: redis + +need_memory: 62G +nr_threads: 100% +numactl: + all: 1 +sysctl: + sc_overcommit_memory: 1 + sc_somaxconn: 65535 +transparent_hugepage: + thp_enabled: never + thp_defrag: never +cluster: cs-localhost +cpu_node_bind: even +nr_processes: 4 + +if role server: + redis-server: +if role client: + redis: + test: set,get + data_size: 1024 + n_client: 5 + requests: 68000000 + n_pipeline: 3 + key_len: 68000000 diff --git a/spec/submit/matrix/job-1-300-whetstone-double.yaml b/spec/submit/matrix/job-1-300-whetstone-double.yaml new file mode 100644 index 00000000..ae23f10c --- /dev/null +++ b/spec/submit/matrix/job-1-300-whetstone-double.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/job-3-300-shell1.yaml b/spec/submit/matrix/job-3-300-shell1.yaml new file mode 100644 index 00000000..90e20c6b --- /dev/null +++ b/spec/submit/matrix/job-3-300-shell1.yaml @@ -0,0 +1,55 @@ +--- + +#! spec/submit/matrix/job.yaml +suite: unixbench +runtime: 300 +unixbench: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +job_origin: spec/submit/matrix/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/matrix/unixbench-spec.yaml b/spec/submit/matrix/job.yaml similarity index 54% rename from spec/submit/matrix/unixbench-spec.yaml rename to spec/submit/matrix/job.yaml index e17322b4..75864574 100644 --- a/spec/submit/matrix/unixbench-spec.yaml +++ b/spec/submit/matrix/job.yaml @@ -1,9 +1,8 @@ suite: unixbench runtime: 300s - nr_task test: - - 1 whetstone-double - - 3 shell1 +- 1 whetstone-double +- 3 shell1 unixbench: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml deleted file mode 100644 index 9896bcef..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '1' -test: whetstone-double -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: whetstone-double - nr_task: '1' - cpufreq_governor: -monitors: diff --git a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml deleted file mode 100644 index 912d10ee..00000000 --- a/spec/submit/matrix/unixbench-spec/unixbench-spec-3-300-shell1.yaml +++ /dev/null @@ -1,25 +0,0 @@ ---- -suite: unixbench -runtime: 300 -unixbench: -lab: spec_lab -testbox: vm-2p8g--spec_submit -nr_task: '3' -test: shell1 -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - unixbench: - runtime: 300 - test: shell1 - nr_task: '3' - cpufreq_governor: -monitors: diff --git a/spec/submit/merge_yaml/job-defaults.yaml b/spec/submit/merge_yaml/job-defaults.yaml new file mode 100644 index 00000000..4168a6ef --- /dev/null +++ b/spec/submit/merge_yaml/job-defaults.yaml @@ -0,0 +1,49 @@ +--- + +#! spec/submit/merge_yaml/job.yaml +suite: atomic +atomic: +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/merge_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + atomic: + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/merge_yaml/job.yaml b/spec/submit/merge_yaml/job.yaml new file mode 100644 index 00000000..6d64c381 --- /dev/null +++ b/spec/submit/merge_yaml/job.yaml @@ -0,0 +1 @@ +<< : template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files.yaml b/spec/submit/merge_yaml/merge_included_files.yaml deleted file mode 100644 index 8d459a32..00000000 --- a/spec/submit/merge_yaml/merge_included_files.yaml +++ /dev/null @@ -1 +0,0 @@ -<<: template_yaml diff --git a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml b/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml deleted file mode 100644 index 42b53c13..00000000 --- a/spec/submit/merge_yaml/merge_included_files/merge_included_files-defaults.yaml +++ /dev/null @@ -1,19 +0,0 @@ ---- -suite: atomic -atomic: -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - atomic: - cpufreq_governor: -monitors: diff --git a/spec/submit/merge_yaml/template_yaml b/spec/submit/merge_yaml/template_yaml index 331cb470..a4805648 100644 --- a/spec/submit/merge_yaml/template_yaml +++ b/spec/submit/merge_yaml/template_yaml @@ -1,3 +1,3 @@ suite: atomic -atomic: +atomic: diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml deleted file mode 100644 index f3e2c553..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 1000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 1000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml deleted file mode 100644 index 63bac5cf..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 100000ss -ftq: - test: add - freq: 10000 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 100000ss - freq: 10000 - test: add - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml deleted file mode 100644 index a802dc1b..00000000 --- a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml +++ /dev/null @@ -1,27 +0,0 @@ ---- -suite: ftq -nr_task: 2 -samples: 6000ss -ftq: - test: cache - freq: 20 -lab: spec_lab -testbox: vm-2p8g--spec_submit -tbox_group: vm-2p8g--spec_submit -provider: qemu -template: kvm -nr_node: 1 -nr_cpu: 2 -memory: 8G -arch: aarch64 -cpufreq_governor: -need_kconfig: -- CONFIG_KVM_GUEST=y -pp: - ftq: - nr_task: 2 - samples: 6000ss - freq: 20 - test: cache - cpufreq_governor: -monitors: diff --git a/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..4dcd179d --- /dev/null +++ b/spec/submit/separate_yaml/job-1000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..d5566714 --- /dev/null +++ b/spec/submit/separate_yaml/job-10000-2-100000ss-add.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..dd177379 --- /dev/null +++ b/spec/submit/separate_yaml/job-20-2-6000ss-cache.yaml @@ -0,0 +1,57 @@ +--- + +#! spec/submit/separate_yaml/job.yaml +suite: ftq +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +job_origin: spec/submit/separate_yaml/job.yaml +tbox_group: vm-2p8g--spec_submit + +#! hosts/vm-2p8g +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 + +#! /etc/compass-ci/defaults/crystal.yaml +SCHED_HOST: 172.168.131.113 +TASKQUEUE_HOST: 172.168.131.113 +LKP_SERVER: 172.168.131.113 +SCHED_PORT: 3000 +GIT_SERVER: 172.168.131.113 +GITCACHE_HOST: 172.168.131.113 +INITRD_HTTP_HOST: 172.168.131.113 +MASTER_FLUENTD_HOST: 172.168.131.113 +MASTER_FLUENTD_PORT: 24224 +my_email: team(a)crystal.ci +my_name: team +my_uuid: a426ef61-9afe-4679-bc2c-af0163393245 + +#! /home/huxuejiao/.config/compass-ci/defaults/huxuejiao.yaml +lkp_initrd_user: huxuejiao + +#! include/testbox/vm +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y + +#! user overrides + +#! auto generated by add_pp() +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: + +#! auto generated by add_monitors() +monitors: diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/job.yaml similarity index 100% rename from spec/submit/separate_yaml/ftq.yaml rename to spec/submit/separate_yaml/job.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index 4096cb73..312400a4 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -14,46 +14,23 @@ def stable_yaml_file(yaml_file) save_yaml(job, yaml_file) end -def traverse_file(output_dir) - Dir.glob("#{output_dir}/*.yaml").each do |yaml_file| - stable_yaml_file(yaml_file) - end +def traverse_file(yaml_file) + stable_yaml_file(yaml_file) unless File.basename(yaml_file) == 'job.yaml' end - -def submit_job(submit_item) - submit_yaml_path = File.join("#{LKP_SRC}/spec/submit", submit_item) - Dir.glob("#{submit_yaml_path}/*.yaml").each do |yaml_file| - output_dir = File.join(submit_yaml_path, File.basename(yaml_file, '.yaml')) + +def submit_job() + Dir.glob("#{LKP_SRC}/spec/submit/*").each do |dir| + yaml_file = "#{dir}/job.yaml" submit_cmd = [ "#{LKP_SRC}/sbin/submit", - '-o', output_dir, + '-o', dir, '-s', 'lab: spec_lab', '-s', 'testbox: vm-2p8g--spec_submit', yaml_file ] system(*submit_cmd) - traverse_file(output_dir) + traverse_file(yaml_file) end end -describe 'submit job spec' do - it 'link jobs spec' do - submit_job('link_jobs') - end - - it 'link matrix' do - submit_job('matrix') - end - - it 'separate yaml spec' do - submit_job('separate_yaml') - end - - it 'job on fail' do - submit_job('job_on_fail') - end - - it 'merge yaml' do - submit_job('merge_yaml') - end -end +submit_job -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: delete kernel_append_root in job.yaml
by Wu Zhende 30 Nov '20

30 Nov '20

kernel_params contains kernel_append_root delete kernel_append_root in job.yaml Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 2 -- src/scheduler/kernel_params.cr | 6 +++--- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 55bac7f..6f81172 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -92,7 +92,6 @@ class Job lkp_initrd_user user_lkp_src kernel_uri - kernel_append_root kernel_params docker_image kernel_version @@ -151,7 +150,6 @@ class Job set_kernel_version() set_initrds_uri() set_kernel_uri() - set_kernel_append_root() set_kernel_params() set_lkp_server() set_sshr_info() diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 2cc8172..0afe692 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -11,7 +11,7 @@ class Job return @hash["kernel_custom_params"] if @hash["kernel_custom_params"]? end - private def set_kernel_append_root + def kernel_append_root os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") fs2root = { @@ -22,7 +22,7 @@ class Job "container" => "", } - self["kernel_append_root"] = fs2root[os_mount] + fs2root[os_mount] end private def kernel_console @@ -30,7 +30,7 @@ class Job end private def set_kernel_params - self["kernel_params"] = " #{kernel_common_params()} #{kernel_custom_params()} #{kernel_append_root} #{kernel_console()}" + self["kernel_params"] = " #{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" end end -- 2.23.0

1 0

[PATCH v3 compass-ci] service/scheduler: initialize a Sched for each API
by Wu Zhende 30 Nov '20

30 Nov '20

So can use Sched's instance variable in Sched's any interface Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 7 +++--- src/scheduler/find_job_boot.cr | 6 ++--- src/scheduler/find_next_job_boot.cr | 6 ++--- src/scheduler/request_cluster_state.cr | 14 ++++++------ src/scheduler/scheduler.cr | 31 +++++++++++++++----------- src/scheduler/submit_job.cr | 4 ++-- src/scheduler/update_job_parameter.cr | 6 ++--- 7 files changed, 40 insertions(+), 34 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 2be7d41..26ddd92 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -25,12 +25,13 @@ class Sched property redis property block_helper - def initialize + def initialize(env : HTTP::Server::Context) @es = Elasticsearch::Client.new @redis = Redis::Client.new @task_queue = TaskQueueAPI.new @block_helper = BlockHelper.new @rgc = RemoteGitClient.new + @env = env end def normalize_mac(mac : String) @@ -67,7 +68,7 @@ class Sched extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end - def update_tbox_wtmp(env : HTTP::Server::Context) + def update_tbox_wtmp testbox = "" hash = Hash(String, String).new @@ -75,7 +76,7 @@ class Sched hash["time"] = time %w(mac ip job_id tbox_name tbox_state).each do |parameter| - if (value = env.params.query[parameter]?) + if (value = @env.params.query[parameter]?) case parameter when "tbox_name" testbox = value diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b5a23c5..9805464 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -2,9 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def find_job_boot(env : HTTP::Server::Context) - value = env.params.url["value"] - boot_type = env.params.url["boot_type"] + def find_job_boot + value = @env.params.url["value"] + boot_type = @env.params.url["boot_type"] case boot_type when "ipxe" diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 807a9ae..09abbd5 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -2,9 +2,9 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def find_next_job_boot(env) - hostname = env.params.query["hostname"]? - mac = env.params.query["mac"]? + def find_next_job_boot + hostname = @env.params.query["hostname"]? + mac = @env.params.query["mac"]? if !hostname && mac hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) end diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 72e3810..79a13cc 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -6,9 +6,9 @@ class Sched # all request states: # wait_ready | abort | failed | finished | wait_finish | # write_state | roles_ip - def request_cluster_state(env) - request_state = env.params.query["state"] - job_id = env.params.query["job_id"] + def request_cluster_state + request_state = @env.params.query["state"] + job_id = @env.params.query["job_id"] cluster_id = @redis.hash_get("sched/id2cluster", job_id).not_nil! cluster_state = "" @@ -41,10 +41,10 @@ class Sched return cluster_state when "write_state" - node_roles = env.params.query["node_roles"] - node_ip = env.params.query["ip"] - direct_ips = env.params.query["direct_ips"] - direct_macs = env.params.query["direct_macs"] + node_roles = @env.params.query["node_roles"] + node_ip = @env.params.query["ip"] + direct_ips = @env.params.query["direct_ips"] + direct_macs = @env.params.query["direct_macs"] job_info = {"roles" => node_roles, "ip" => node_ip, diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 5f97567..98b8e2e 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -30,12 +30,17 @@ require "../lib/sched" module Scheduler VERSION = "0.2.0" - sched = Sched.new + add_context_storage_type(Sched) before_all do |env| + env.set "sched", Sched.new(env) env.response.headers["Connection"] = "close" end + def self.get_sched_from_env(env) + env.get("sched").as(Sched) + end + # for debug (maybe kemal debug|logger does better) def self.debug_message(env, response) puts %({"from": "#{env.request.remote_address}", "response": #{response.to_json}}) @@ -53,7 +58,7 @@ module Scheduler # /boot.xxx/host/${hostname} # /boot.yyy/mac/${mac} get "/boot.:boot_type/:parameter/:value" do |env| - response = sched.find_job_boot(env) + response = get_sched_from_env(env).find_job_boot job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? puts %({"job_id": "#{job_id}", "job_state": "boot"}) if job_id @@ -63,7 +68,7 @@ module Scheduler # /~lkp/cgi-bin/gpxelinux.cgi?hostname=:hostname&mac=:mac&last_kernel=:last_kernel get "/~lkp/cgi-bin/gpxelinux.cgi" do |env| - response = sched.find_next_job_boot(env) + response = get_sched_from_env(env).find_next_job_boot job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? puts %({"job_id": "#{job_id}", "job_state": "boot"}) if job_id @@ -75,7 +80,7 @@ module Scheduler # - echo job_id to caller # -- job_id = "0" ? means failed post "/submit_job" do |env| - job_messages = sched.submit_job(env) + job_messages = get_sched_from_env(env).submit_job job_messages.each do |job_message| puts job_message.to_json @@ -103,7 +108,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/set_host_mac?hostname=wfg&mac=00-01-02-03-04-05" put "/set_host_mac" do |env| if (client_hostname = env.params.query["hostname"]?) && (client_mac = env.params.query["mac"]?) - sched.set_host_mac(client_mac, client_hostname) + get_sched_from_env(env).set_host_mac(client_mac, client_hostname) "Done" else @@ -114,7 +119,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" put "/set_host2queues" do |env| if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) - sched.set_host2queues(client_host, client_queues) + get_sched_from_env(env).set_host2queues(client_host, client_queues) "Done" else @@ -125,7 +130,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| if client_mac = env.params.query["mac"]? - sched.del_host_mac(client_mac) + get_sched_from_env(env).del_host_mac(client_mac) "Done" else @@ -136,7 +141,7 @@ module Scheduler # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" put "/del_host2queues" do |env| if client_host = env.params.query["host"]? - sched.del_host2queues(client_host) + get_sched_from_env(env).del_host2queues(client_host) "Done" else @@ -150,7 +155,7 @@ module Scheduler # ?job_file=/lkp/scheduled/job.yaml&job_state=post_run&job_id=10 # ?job_file=/lkp/scheduled/job.yaml&loadavg=0.28 0.82 0.49 1/105 3389&start_time=1587725398&end_time=1587725698&job_id=10 get "/~lkp/cgi-bin/lkp-jobfile-append-var" do |env| - puts sched.update_job_parameter(env) + puts get_sched_from_env(env).update_job_parameter "Done" end @@ -173,7 +178,7 @@ module Scheduler # response: get "server ip" from cluster state, # return "server=<server ip>". get "/~lkp/cgi-bin/lkp-cluster-sync" do |env| - response = sched.request_cluster_state(env) + response = get_sched_from_env(env).request_cluster_state debug_message(env, response) @@ -187,14 +192,14 @@ module Scheduler # get job_id from request job_id = env.params.query["job_id"]? if job_id - puts sched.close_job(job_id) + puts get_sched_from_env(env).close_job(job_id) end "Done" end get "/~lkp/cgi-bin/lkp-wtmp" do |env| - puts sched.update_tbox_wtmp(env) + puts get_sched_from_env(env).update_tbox_wtmp "Done" end @@ -205,7 +210,7 @@ module Scheduler job_id = env.params.query["job_id"].to_s if testbox && ssh_port - sched.report_ssh_port(testbox, ssh_port) + get_sched_from_env(env).report_ssh_port(testbox, ssh_port) end puts %({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"}) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 2836275..4e78102 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -2,8 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def submit_job(env : HTTP::Server::Context) - body = env.request.body.not_nil!.gets_to_end + def submit_job + body = @env.request.body.not_nil!.gets_to_end job_content = JSON.parse(body) job = Job.new(job_content, job_content["id"]?) diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index 0692025..cee8b9b 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -2,8 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Sched - def update_job_parameter(env : HTTP::Server::Context) - job_id = env.params.query["job_id"]? + def update_job_parameter + job_id = @env.params.query["job_id"]? if !job_id return false end @@ -13,7 +13,7 @@ class Sched job_content["id"] = job_id (%w(start_time end_time loadavg job_state)).each do |parameter| - value = env.params.query[parameter]? + value = @env.params.query[parameter]? if !value || value == "" next end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] git_mirror.rb: change git_repo of upstream-repos
by Li Yuanchao 30 Nov '20

30 Nov '20

Change git_repo of upstream-repos from 'upstream-repos/upstream-repos' to 'u/upstream-repos/upstream-repos'. It's to keep style the same with other repos. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5b6b889..edee331 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -174,7 +174,7 @@ class MirrorMain update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - return reload_fork_info if git_repo == 'upstream-repos/upstream-repos' + return reload_fork_info if git_repo == 'u/upstream-repos/upstream-repos' new_refs = check_new_refs(git_repo) return if new_refs[:heads].empty? @@ -254,7 +254,7 @@ class MirrorMain end def reload_fork_info - upstream_repos = 'upstream-repos/upstream-repos' + upstream_repos = 'u/upstream-repos/upstream-repos' if @git_info[upstream_repos][:cur_refs].empty? @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) else -- 2.23.0

1 0

[PATCH v2 compass-ci] container/netdata-slave: monitoring HW server
by Xiao Shenwei 30 Nov '20

30 Nov '20

[monitored object] file-server : master crystal.ci : slave z9 : slave ibmc-mgnt : slave backup-server: slave jumper-server: slave startup by multi-qemu or multi-docker job [function] slave node will collect host machine metrics, send to master slave node does not provider data storage, alarm, web services. [usage] config/stream.conf destination = master-HOST[:PORT] api key = refer-to: https://github.com/netdata/netdata/tree/master/streaming --- container/netdata-slave/Dockerfile | 10 ++++++ container/netdata-slave/build | 5 +++ container/netdata-slave/config/netdata.conf | 5 +++ container/netdata-slave/config/stream.conf | 10 ++++++ container/netdata-slave/start | 34 +++++++++++++++++++++ 5 files changed, 64 insertions(+) create mode 100755 container/netdata-slave/Dockerfile create mode 100755 container/netdata-slave/build create mode 100644 container/netdata-slave/config/netdata.conf create mode 100644 container/netdata-slave/config/stream.conf create mode 100755 container/netdata-slave/start diff --git a/container/netdata-slave/Dockerfile b/container/netdata-slave/Dockerfile new file mode 100755 index 0000000..5c2c02f --- /dev/null +++ b/container/netdata-slave/Dockerfile @@ -0,0 +1,10 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +FROM netdata/netdata + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN usermod -u 1090 netdata && \ + groupmod -g 1090 netdata diff --git a/container/netdata-slave/build b/container/netdata-slave/build new file mode 100755 index 0000000..1a84550 --- /dev/null +++ b/container/netdata-slave/build @@ -0,0 +1,5 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +docker build -t netdata/netdata-slave . diff --git a/container/netdata-slave/config/netdata.conf b/container/netdata-slave/config/netdata.conf new file mode 100644 index 0000000..456fe02 --- /dev/null +++ b/container/netdata-slave/config/netdata.conf @@ -0,0 +1,5 @@ +[global] + memory mode = none + +[health] + enabled = no diff --git a/container/netdata-slave/config/stream.conf b/container/netdata-slave/config/stream.conf new file mode 100644 index 0000000..ac91a7d --- /dev/null +++ b/container/netdata-slave/config/stream.conf @@ -0,0 +1,10 @@ +[stream] + enabled = yes + destination = + api key = + timeout seconds = 60 + default port = 19999 + send charts matching = * + buffer size bytes = 1048576 + reconnect delay seconds = 5 + initial clock resync iterations = 60 diff --git a/container/netdata-slave/start b/container/netdata-slave/start new file mode 100755 index 0000000..e790298 --- /dev/null +++ b/container/netdata-slave/start @@ -0,0 +1,34 @@ +#!/bin/bash +# Origin: https://hub.docker.com/r/netdata/netdata +# SPDX-License-Identifier: GPL-3.0+ + +. $CCI_SRC/container/defconfig.sh + +docker_rm netdata-slave + +DIR=$(dirname $(realpath $0)) + +cmd=( + docker run -d + --restart=unless-stopped + --hostname=$HOSTNAME + --name netdata-slave + -p 19999:19999 + -v $DIR/config/netdata.conf:/etc/netdata/netdata.conf + -v $DIR/config/stream.conf:/etc/netdata/stream.conf + -v /etc/passwd:/host/etc/passwd:ro + -v /etc/group:/host/etc/group:ro + -v /proc:/host/proc:ro + -v /sys:/host/sys:ro + -v /etc/localtime:/etc/localtime:ro + -v /etc/os-release:/host//etc/os-release:ro + -v /var/run/docker.sock:/var/run/docker.sock:ro + -v /srv/cache/netdata_cache:/var/cache/netdata + -v /srv/cache/netdata_lib:/var/lib/netdata + --log-driver json-file + --log-opt max-size=1g + --cap-add SYS_PTRACE + --security-opt apparmor=unconfined + netdata/netdata-slave +) +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: change git-repo of upstream-repos
by Li Yuanchao 30 Nov '20

30 Nov '20

Change git_repo of upstream-repos from 'upstream-repos/upstream-repos' to 'u/upstream-repos/upstream-repos'. It's to keep style the same with other repos. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5b6b889..edee331 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -174,7 +174,7 @@ class MirrorMain update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - return reload_fork_info if git_repo == 'upstream-repos/upstream-repos' + return reload_fork_info if git_repo == 'u/upstream-repos/upstream-repos' new_refs = check_new_refs(git_repo) return if new_refs[:heads].empty? @@ -254,7 +254,7 @@ class MirrorMain end def reload_fork_info - upstream_repos = 'upstream-repos/upstream-repos' + upstream_repos = 'u/upstream-repos/upstream-repos' if @git_info[upstream_repos][:cur_refs].empty? @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) else -- 2.23.0

1 0

[PATCH compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 30 Nov '20

30 Nov '20

Add a document for introducing the arch, nr_node, nr_cpu, memory.md fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..2a4336c --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,16 @@ +# arch, nr_node, nr_cpu, memory + +Meaning: +- Each test machine has some basic information, such as arch, memory, nr_cpu, nr_node + these fields mean: + arch: the architecture of the machine. + nr_node: number of NUMA nodes. + nr_cpu: number of logical CPUs. + memory: the storage space of the machine. +- These fields are from the file which in $LAB_SRC/hosts. +- Here is an example of these fields in the host file: + arch: aarch64 + nr_node: 4 + nr_cpu: 96 + memory: 256G + ... -- 2.23.0

1 0