- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] doc/job/: add os_fields.md
by Zhang Yu 25 Nov '20

25 Nov '20

Add a document for introducing the os, os_version, os_arch fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/os_fields.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 doc/job/os_fields.md diff --git a/doc/job/os_fields.md b/doc/job/os_fields.md new file mode 100644 index 0000000..1e6baae --- /dev/null +++ b/doc/job/os_fields.md @@ -0,0 +1,15 @@ +# os, os_version, os_arch + +Meaning: +- Each test tells which os it needs, and the os_version, os_arch is also need to be specified, then the machine will run an os according to those three fields. +- Os, os_version, os_arch is the key for users to specify the os-related parameters. +- If os, os_version, os_arch is not given by users, it will use the default openeuler os-related parameters. +- Here are the os, os_version, os_arch that we support: + os os_version os_arch + openeuler 20.03, 1.0 aarch64, x86_64 + debian sid, 10 aarch64, x86_64 + centos 7.6, 7.8 aarch64, x86_64 + ... + +Usage example: +- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 -- 2.23.0

4 6

[PATCH compass-ci] block_helper.cr: change if block into one line
by Xu Xijian 25 Nov '20

25 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/block_helper.cr | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/lib/block_helper.cr b/src/lib/block_helper.cr index 37ad367..596ef7e 100644 --- a/src/lib/block_helper.cr +++ b/src/lib/block_helper.cr @@ -28,9 +28,7 @@ class BlockHelper @block_helper[uuid] = fiber end - if yield == true - spawn fiber.run - end + spawn fiber.run if yield == true until fiber.dead? Fiber.yield -- 2.23.0

2 2

[PATCH compass-ci] lib/compare_matrixes.rb: fix rubocop offenses
by Zhang Yu 25 Nov '20

25 Nov '20

Inspecting 1 file Offenses: compare_matrixes.rb:469:5: C: Layout/EmptyLineAfterGuardClause: Add empty line after guard clause. next if values['average'].size < 2 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/compare_matrixes.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 0b49e59..1550435 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -467,6 +467,7 @@ end def assign_metric_change(metrics_values) metrics_values.each do |metric, values| next if values['average'].size < 2 + metrics_values[metric]['change'] = {} dimension_list = values['average'].keys -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/submit: submit merge_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 4 ++ .../template/template-defaults.yaml | 71 +++++++++++++++++++ .../merge_yaml/yaml_merge_included_files.yaml | 1 + .../yaml_merge_included_files-defaults.yaml | 71 +++++++++++++++++++ spec/submit_spec.rb | 4 ++ 5 files changed, 151 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/template/template-defaults.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..6d6fb37c --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,4 @@ +suite: test +category: benchmark + +test: diff --git a/spec/submit/merge_yaml/template/template-defaults.yaml b/spec/submit/merge_yaml/template/template-defaults.yaml new file mode 100644 index 00000000..f1d13d04 --- /dev/null +++ b/spec/submit/merge_yaml/template/template-defaults.yaml @@ -0,0 +1,71 @@ +--- +suite: test +category: benchmark +test: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml diff --git a/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml new file mode 100644 index 00000000..f1d13d04 --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml @@ -0,0 +1,71 @@ +--- +suite: test +category: benchmark +test: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..b56a860c 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH v3 lkp-tests] daemon/sshd: fix pub_key to my_ssh_pubkey
by Zhang Yale 25 Nov '20

25 Nov '20

[why] 1. "my_ssh_pubkey" field is a collection of all pub_keys submitted by users. 2. We can save all the pub_keys submitted by users in the ES database and password-free login. 3. Therefore, users can sometimes borrow a machine to test without providing a pub_key. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index a3fdf954..c7d7b60c 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,5 +1,5 @@ #!/bin/sh -# pub_key +# my_ssh_pubkey # my_email # sshr_port # sshr_port_len @@ -14,10 +14,10 @@ sshr_ip=$LKP_SERVER run_ssh() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh - echo "$pub_key" > /root/.ssh/authorized_keys + echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys systemctl start sshd } @@ -73,11 +73,11 @@ compose_email() set_port() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return for i in $(seq 1 10) do - port=$(($(date +%s%N)%${sshr_port_len}+${sshr_port_base})) + port=$(($(date +%s%N)%$sshr_port_len+$sshr_port_base)) ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null [ $? -eq 0 ] && return -- 2.23.0

2 2

[PATCH v2 lkp-tests] lkp-tests/*: update AUR Helper for archlinux
by Wang Yong 25 Nov '20

25 Nov '20

yaourt is unmaintained and yay also support native pacman operation, so only minor changes are needed Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 15 +++++++++++++++ distro/installer/archlinux | 15 +++++++++------ lib/install.sh | 2 +- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/distro/archlinux b/distro/archlinux index 408c5fa23..3f8d62ebc 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -109,3 +109,18 @@ pack_benchmark() ) done } + +add_aur_helper() +{ + echo "lkp ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers + + pacman -Q yay 2>/dev/null && return + + local user=lkp + local dir=/home/${user} + + mkdir -p ${dir} && chown ${user}:${user} ${dir} + pacman -S --needed --noconfirm git base-devel 2>/dev/null + su - ${user} -c "git clone https://aur.archlinux.org/yay.git" + su - ${user} -c "cd yay && makepkg -si --needed --noconfirm" +} diff --git a/distro/installer/archlinux b/distro/installer/archlinux index 27cdc63cf..af98e466f 100755 --- a/distro/installer/archlinux +++ b/distro/installer/archlinux @@ -2,12 +2,15 @@ # Archlinux package installation -# yaourt is a wrapper for pacman that adds automated access -# to the AUR. -# See: https://wiki.archlinux.org/index.php/Yaourt +# Yet Another Yogurt - An AUR Helper Written in Go +# Yay is based on the design of yaourt, apacman and pacaur +# See: https://github.com/Jguer/yay -# yaourt not support root to build package +# Yay not support root to build package # Use regular user to do this -regular_user=`grep 1000 /etc/passwd | awk -F':' '{ print $1 }'` +. $LKP_SRC/distro/${DISTRO} +add_aur_helper -sudo -u $regular_user yaourt -S --noconfirm $* +regular_user=`grep 1090 /etc/passwd | awk -F':' '{ print $1 }'` + +sudo -u $regular_user yay -S --needed --noconfirm $* diff --git a/lib/install.sh b/lib/install.sh index 881ade9f4..bf7bf21d7 100755 --- a/lib/install.sh +++ b/lib/install.sh @@ -17,7 +17,7 @@ sync_distro_sources() yum update fi ;; redhat) yum update ;; - archlinux) yaourt -Sy ;; + archlinux) yay -Sy ;; opensuse) zypper update ;; oracle) yum update ;; -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit: submit separate_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/separate_yaml/ftq.yaml | 22 +++++++++++++++ .../ftq/ftq-1000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-10000-2-100000ss-add.yaml | 28 +++++++++++++++++++ .../ftq/ftq-20-2-6000ss-cache.yaml | 28 +++++++++++++++++++ spec/submit_spec.rb | 4 +++ 5 files changed, 110 insertions(+) create mode 100644 spec/submit/separate_yaml/ftq.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml create mode 100644 spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml diff --git a/spec/submit/separate_yaml/ftq.yaml b/spec/submit/separate_yaml/ftq.yaml new file mode 100644 index 00000000..8564d443 --- /dev/null +++ b/spec/submit/separate_yaml/ftq.yaml @@ -0,0 +1,22 @@ +suite: ftq +category: noise-benchmark + +nr_task: 100% + +samples: 100000ss + +ftq: + test: add + freq: 10000 + +--- +ftq: + test: add + freq: 1000 + +--- + +samples: 6000ss +ftq: + test: cache + freq: 20 diff --git a/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml new file mode 100644 index 00000000..855e7210 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-1000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 1000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 1000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml new file mode 100644 index 00000000..78725424 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-10000-2-100000ss-add.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 100000ss +ftq: + test: add + freq: 10000 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 100000ss + freq: 10000 + test: add + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml new file mode 100644 index 00000000..40edfcb2 --- /dev/null +++ b/spec/submit/separate_yaml/ftq/ftq-20-2-6000ss-cache.yaml @@ -0,0 +1,28 @@ +--- +suite: ftq +category: noise-benchmark +nr_task: 2 +samples: 6000ss +ftq: + test: cache + freq: 20 +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + ftq: + nr_task: 2 + samples: 6000ss + freq: 20 + test: cache + cpufreq_governor: {} +monitors: {} diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..ce6fed25 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'separate yaml spec' do + submit_job('separate_yaml') + end end -- 2.23.0

2 1

[PATCH v3 compass-ci 2/2] job.cr: give kernel_version a default value
by Xu Xijian 25 Nov '20

25 Nov '20

If the kernel_version is not set by users, give it a default value as same as vmlinuz version. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b533665..b87b55f 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -358,11 +358,11 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? - self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") + self["kernel_version"] ||= File.basename(File.real_path "#{boot_dir}/vmlinuz").lchop("vmlinuz-") + self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz-#{kernel_version}") if "#{os_mount}" == "initramfs" - self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") - self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") + self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules-#{kernel_version}.cgz") + self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers-#{kernel_version}.cgz") end end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] block_helper.cr: change if block into one line
by Xu Xijian 25 Nov '20

25 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/block_helper.cr | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/src/lib/block_helper.cr b/src/lib/block_helper.cr index 37ad367..596ef7e 100644 --- a/src/lib/block_helper.cr +++ b/src/lib/block_helper.cr @@ -28,9 +28,7 @@ class BlockHelper @block_helper[uuid] = fiber end - if yield == true - spawn fiber.run - end + spawn fiber.run if yield == true until fiber.dead? Fiber.yield -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: submit merge_yaml/*.yaml
by Hu Xuejiao 25 Nov '20

25 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 3 + .../template/template-defaults.yaml | 71 +++++++++++++++++++ .../merge_yaml/yaml_merge_included_files.yaml | 1 + .../yaml_merge_included_files-defaults.yaml | 71 +++++++++++++++++++ spec/submit_spec.rb | 4 ++ 5 files changed, 150 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/template/template-defaults.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..60e3df16 --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,3 @@ +suite: test +category: benchmark +test: diff --git a/spec/submit/merge_yaml/template/template-defaults.yaml b/spec/submit/merge_yaml/template/template-defaults.yaml new file mode 100644 index 00000000..f1d13d04 --- /dev/null +++ b/spec/submit/merge_yaml/template/template-defaults.yaml @@ -0,0 +1,71 @@ +--- +suite: test +category: benchmark +test: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml diff --git a/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml new file mode 100644 index 00000000..f1d13d04 --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files/yaml_merge_included_files-defaults.yaml @@ -0,0 +1,71 @@ +--- +suite: test +category: benchmark +test: +lab: spec_lab +testbox: vm-2p8g--spec_submit +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kmsg: +boot-time: +uptime: +iostat: +heartbeat: +vmstat: +numa-numastat: +numa-vmstat: +numa-meminfo: +proc-vmstat: +proc-stat: +meminfo: +slabinfo: +interrupts: +lock_stat: +latency_stats: +softirqs: +bdi_dev_mapping: +diskstats: +nfsstat: +cpuidle: +cpufreq-stats: +sched_debug: +perf-stat: +mpstat: +perf-profile: +cpufreq_governor: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + cpufreq_governor: {} +monitors: + kmsg: + boot-time: + uptime: + iostat: + heartbeat: + vmstat: + numa-numastat: + numa-vmstat: + numa-meminfo: + proc-vmstat: + proc-stat: + meminfo: + slabinfo: + interrupts: + lock_stat: + latency_stats: + softirqs: + bdi_dev_mapping: + diskstats: + nfsstat: + cpuidle: + cpufreq-stats: + sched_debug: + perf-stat: + mpstat: + perf-profile: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..b56a860c 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

3 2

[PATCH compass-ci 2/2] sparrow/0-package: fix docker.service restart failed
by Liu Yinsi 25 Nov '20

25 Nov '20

[why] when execute systemctl restart docker, error message: localhost.localdomain systemd[1]:docker.service: Failed with result 'timeout'. localhost.localdomain systemd[1]:Failed to start Docker Application Container Engine. because docker restart default timeout is 90s, when run too much containers, restart docker may need several minutes, then restart failed. [how] set TimeoutSec=0 to avoid timeout. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 5eb9f76..2753b9d 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -40,4 +40,7 @@ cat > /etc/docker/daemon.json <<EOF } EOF +sed '/^Environment=/a\TimeoutSec=0' /usr/lib/systemd/system/docker.service + +systemctl daemon-reload systemctl restart docker -- 2.23.0

1 0

[PATCH compass-ci 1/2] sparrow/0-package: fix failed to start ssh-r
by Liu Yinsi 25 Nov '20

25 Nov '20

[why] when execute ssh-r/start, error message: docker: Error response from daemon: driver failed programming external connectivity on endpoint ssh_r (55638ba16aa1264dff0fdc1badc93176eec07d2b848d4a8d418d07011d84313d): Error starting userland proxy: listen tcp 0.0.0.0:51998: bind: address already in use. [how] because ssh-r will bind ports 50000-51999, set ip_local_port_range to prevent occupation. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 523438c..5eb9f76 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -12,6 +12,7 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 net.ipv6.bindv6only=1 vm.max_map_count=262144 +net.ipv4.ip_local_port_range = 1024 48999 EOF sysctl -p -- 2.23.0

1 0

[PATCH compass-ci] job.cr: give kernel_version a default value
by Xu Xijian 25 Nov '20

25 Nov '20

If the kernel_version is not set by users, give it a default value as same as vmlinuz version. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b533665..fa804e5 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -358,11 +358,11 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? - self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") + self["kernel_version"] ||= File.basename(File.real_path "#{boot_dir}/vmlinuz").gsub("vmlinuz-", "") + self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz-#{kernel_version}") if "#{os_mount}" == "initramfs" - self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") - self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") + self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules-#{kernel_version}.cgz") + self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers-#{kernel_version}.cgz") end end -- 2.23.0

1 1

[PATCH v2 compass-ci] job.cr: give kernel_version a default value
by Xu Xijian 25 Nov '20

25 Nov '20

If the kernel_version is not set by users, give it a default value as same as vmlinuz version. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b533665..a9923cc 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -358,11 +358,11 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? - self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") + self["kernel_version"] ||= File.basename(File.real_path "#{boot_dir}/vmlinuz").gsub("vmlinuz-", "") + self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz-#{kernel_version}") if "#{os_mount}" == "initramfs" - self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") - self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") + self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules-#{kernel_version}.cgz") + self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers-#{kernel_version}.cgz") end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] daemon/sshd: fix pub_key to my_ssh_pubkey
by Zhang Yale 25 Nov '20

25 Nov '20

[Why] - "my_ssh_pubkey" field is a set of all pub_keys that had been submitted by users. - We can save all the pub_keys submitted by users in the ES database and set the login mode to password-free. - Therefore, users can sometimes borrow a machine to test without providing a pub_key. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index a3fdf954..c7d7b60c 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,5 +1,5 @@ #!/bin/sh -# pub_key +# my_ssh_pubkey # my_email # sshr_port # sshr_port_len @@ -14,10 +14,10 @@ sshr_ip=$LKP_SERVER run_ssh() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh - echo "$pub_key" > /root/.ssh/authorized_keys + echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys systemctl start sshd } @@ -73,11 +73,11 @@ compose_email() set_port() { - [ -n "$pub_key" ] || return + [ -n "$my_ssh_pubkey" ] || return for i in $(seq 1 10) do - port=$(($(date +%s%N)%${sshr_port_len}+${sshr_port_base})) + port=$(($(date +%s%N)%$sshr_port_len+$sshr_port_base)) ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null [ $? -eq 0 ] && return -- 2.23.0

1 1

[PATCH lkp-tests] setup/sysctl: add parameters
by Zhang Yu 25 Nov '20

25 Nov '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- setup/sysctl | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/setup/sysctl b/setup/sysctl index 81374939..913b83d3 100755 --- a/setup/sysctl +++ b/setup/sysctl @@ -202,6 +202,9 @@ # - sc_netdev_max_backlog # - sc_rmem_max # - sc_wmem_max +# - sc_tcp_rmem +# - sc_tcp_wmem +# - sc_sched_features . $LKP_SRC/lib/reproduce-log.sh @@ -408,5 +411,8 @@ [ -n "$sc_netdev_max_backlog" ] && log_eval "echo '$sc_netdev_max_backlog' > '/proc/sys/net/core/netdev_max_backlog'" [ -n "$sc_rmem_max" ] && log_eval "echo '$sc_rmem_max' > '/proc/sys/net/core/rmem_max'" [ -n "$sc_wmem_max" ] && log_eval "echo '$sc_wmem_max' > '/proc/sys/net/core/wmem_max'" +[ -n "$sc_tcp_rmem" ] && log_eval "echo '$sc_tcp_rmem' > '/proc/sys/net/ipv4/tcp_rmem'" +[ -n "$sc_tcp_wmem" ] && log_eval "echo '$sc_tcp_wmem' > '/proc/sys/net/ipv4/tcp_wmem'" +[ -n "$sc_sched_features" ] && log_eval "echo '$sc_sched_features' > '/sys/kernel/debug/sched_features'" exit 0 -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: fix repeated setting job
by Wu Zhende 25 Nov '20

25 Nov '20

Only first time should full initialize the job. Next time function "initialized?" will return true, just to use job_content, don't need to set default values. But now, "kernel_version" will let "initialized?" return false. Fix this. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b533665..e56f248 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -67,6 +67,8 @@ class Job set_defaults() end + # Each key must exist after the job is initialized + # Otherwise "initialized?" will return false METHOD_KEYS = %w( id os @@ -95,7 +97,6 @@ class Job kernel_append_root kernel_params docker_image - kernel_version linux_vmlinuz_path linux_modules_initrd linux_headers_initrd @@ -358,7 +359,7 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? + suffix = "-#{self["kernel_version"]}" if self["kernel_version"]? self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") if "#{os_mount}" == "initramfs" self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] sbin/submit: add job into my queue
by Xiao Shenwei 25 Nov '20

25 Nov '20

add --my-queue parameter the job will be add into your own queue usage: submit --my-queue testbox=vm-2p8g ../jobs/borrow-1h.yaml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/submit | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/sbin/submit b/sbin/submit index 9bba1018..be588bed 100755 --- a/sbin/submit +++ b/sbin/submit @@ -16,6 +16,7 @@ opt_output_dir = nil opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} +opt_my_queue = false actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -54,6 +55,11 @@ options = OptionParser.new do |opts| ARGV.shift end end + + opts.on('--my-queue', 'add to my queue') do + opt_my_queue = true + end + end options.parse!(ARGV) @@ -93,6 +99,10 @@ def find_jobfile(jobfile) exit 1 end +def add_my_queue(job) + job['queue'] = "#{job['tbox_group']}~#{ENV['USER']}" +end + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -134,6 +144,7 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) + add_my_queue(job_hash) if opt_my_queue # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

2 1

[PATCH v2 compass-ci] doc/job/: add os_fields.md
by Zhang Yu 25 Nov '20

25 Nov '20

Add a document for introducing the os, os_version, os_arch fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/os_fields.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/os_fields.md diff --git a/doc/job/os_fields.md b/doc/job/os_fields.md new file mode 100644 index 0000000..942f4b4 --- /dev/null +++ b/doc/job/os_fields.md @@ -0,0 +1,16 @@ +# os, os_version, os_arch + +Meaning: +- Each test tells which os it needs, and the os_version, os_arch are also need to be specified, + then the machine will run an os according to these three fields. +- These os, os_version, os_arch are the key for users to specify the os related parameters. +- If os, os_version, os_arch are not given by users, it will use the default openeuler os related parameters. +- Here are some examples: + os os_version os_arch + openeuler 20.03, 1.0 aarch64, x86_64 + debian sid, 10 aarch64, x86_64 + centos 7.6, 7.8 aarch64, x86_64 + ... + +Usage example: +- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: add my_ssh_pubkey
by Wu Zhende 25 Nov '20

25 Nov '20

Because pub_key is in sshd, and sshd is a hash. Hash will skip in "sh_export_top_env", So job.sh can't get the array pub_key to do password-free login. Add a new key "my_ssh_pubkey" in job to solve this problem. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 785daea..b533665 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -206,6 +206,7 @@ class Job sshd["pub_key"] = @account_info["my_ssh_pubkey"] @hash["sshd"] = JSON.parse(sshd.to_json) + @hash["my_ssh_pubkey"] = @account_info["my_ssh_pubkey"] end private def update_account_my_pub_key(pub_key) -- 2.23.0

1 0

[PATCH v2 compass-ci] add-software: add reminders
by Wang Chenglong 25 Nov '20

25 Nov '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/add-software/run | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/add-software/run b/container/add-software/run index 089efd2..62d4eff 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -9,6 +9,8 @@ 2> ./run /os/centos/aarch64/7.6.1810 $(show-depends-packages centos) centos is an adaption file contain packages mapping from debian to centos, the whole path is '$LKP_SRC/distro/adaptation/centos'. + [Remind] + This tool also works for openEuler. EOF exit } -- 2.23.0

1 0

[PATCH compass-ci] add-software: add reminders
by Wang Chenglong 25 Nov '20

25 Nov '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/add-software/run | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/add-software/run b/container/add-software/run index 089efd2..dcbb8f6 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -9,6 +9,8 @@ 2> ./run /os/centos/aarch64/7.6.1810 $(show-depends-packages centos) centos is an adaption file contain packages mapping from debian to centos, the whole path is '$LKP_SRC/distro/adaptation/centos'. + [Remind] + This tool also works for Openeuler. EOF exit } -- 2.23.0

2 2

[PATCH compass-ci] remove empty changes for metric compare result
by Lu Weitao 25 Nov '20

25 Nov '20

[Why] With template compare mode, it's useless return an empty changes for a metric like: { ... "tables": { "vmstat.system.cs": { ... "change": { "dimensions": [ "compare_dimension" ], "source": [ [ "8G", "16G" ] ] } } } [How] Skip assign metric change value, if the metric only have one dimension, the empty change will disappear automatically Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 14c14cb..0b49e59 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -466,8 +466,8 @@ end def assign_metric_change(metrics_values) metrics_values.each do |metric, values| - metrics_values[metric]['change'] = {} next if values['average'].size < 2 + metrics_values[metric]['change'] = {} dimension_list = values['average'].keys dimension_groups = get_dimensions_combination(dimension_list) -- 2.23.0

1 0

[PATCH compass-ci] doc/job/: add os.md
by Zhang Yu 25 Nov '20

25 Nov '20

add a document for introducing the os field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/os.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 doc/job/os.md diff --git a/doc/job/os.md b/doc/job/os.md new file mode 100644 index 0000000..0d0102c --- /dev/null +++ b/doc/job/os.md @@ -0,0 +1,14 @@ +# os + +Meaning: +- Each test tells which os is to be testing on, the vm will run a os according to different need. +- Os is a key for users to specify a os type. +- If os is not given by users, it will use the default debian. +- Here are the os type that we support: + openeuler + debian + centos + ... + +Usage example: +- submit iperf.yaml testbox=vm-2p8g queue=vm-2p8g~$USER os=openeuler os_arch=aarch64 os_version=20.03 os_mount=cifs -- 2.23.0

2 2

Re: [PATCH lkp-tests] stats/iperf: fix rubocop offenses
by Wei Jihui 25 Nov '20

25 Nov '20

hi, everyone, some lkp-tests's scripts is not our create, it is better not to modify them. Thanks, Jihui Reply-To: In-Reply-To: <20201125015157.2194483-1-2134782174(a)qq.com> On Wed, Nov 25, 2020 at 09:51:57AM +0800, Zhang Yu wrote: >Inspecting 1 file > >iperf:19:86: C: Style/IfUnlessModifier: Modifier form of if makes the line too long. >puts 'tcp.sender.bps: ' + obj['end']['streams'][0]['sender']['bits_per_second'].to_s if obj['end']['streams'][0]['sender'] > ^^ > >iperf:20:90: C: Style/IfUnlessModifier: Modifier form of if makes the line too long. >puts 'tcp.receiver.bps: ' + obj['end']['streams'][0]['receiver']['bits_per_second'].to_s if obj['end']['streams'][0]['receiver' > ^^ > >Signed-off-by: Zhang Yu <2134782174(a)qq.com> >--- > stats/iperf | 12 +++++++++--- > 1 file changed, 9 insertions(+), 3 deletions(-) > >diff --git a/stats/iperf b/stats/iperf >index b5c6c403..f2521db6 100755 >--- a/stats/iperf >+++ b/stats/iperf >@@ -16,6 +16,12 @@ end > > exit if obj['end'].empty? > >-puts 'tcp.sender.bps: ' + obj['end']['streams'][0]['sender']['bits_per_second'].to_s if obj['end']['streams'][0]['sender'] >-puts 'tcp.receiver.bps: ' + obj['end']['streams'][0]['receiver']['bits_per_second'].to_s if obj['end']['streams'][0]['receiver'] >-puts 'udp.bps: ' + obj['end']['streams'][0]['udp']['bits_per_second'].to_s if obj['end']['streams'][0]['udp'] >+if obj['end']['streams'][0]['sender'] >+ puts 'tcp.sender.bps: ' + obj['end']['streams'][0]['sender']['bits_per_second'].to_s >+end >+if obj['end']['streams'][0]['receiver'] >+ puts 'tcp.receiver.bps: ' + obj['end']['streams'][0]['receiver']['bits_per_second'].to_s >+end >+if obj['end']['streams'][0]['udp'] >+ puts 'udp.bps: ' + obj['end']['streams'][0]['udp']['bits_per_second'].to_s >+end >-- >2.23.0 >

1 0