- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] stats/iperf: fix rubocop offenses
by Zhang Yu 25 Nov '20

25 Nov '20

Inspecting 1 file iperf:19:86: C: Style/IfUnlessModifier: Modifier form of if makes the line too long. puts 'tcp.sender.bps: ' + obj['end']['streams'][0]['sender']['bits_per_second'].to_s if obj['end']['streams'][0]['sender'] ^^ iperf:20:90: C: Style/IfUnlessModifier: Modifier form of if makes the line too long. puts 'tcp.receiver.bps: ' + obj['end']['streams'][0]['receiver']['bits_per_second'].to_s if obj['end']['streams'][0]['receiver' ^^ Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/iperf | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/stats/iperf b/stats/iperf index b5c6c403..f2521db6 100755 --- a/stats/iperf +++ b/stats/iperf @@ -16,6 +16,12 @@ end exit if obj['end'].empty? -puts 'tcp.sender.bps: ' + obj['end']['streams'][0]['sender']['bits_per_second'].to_s if obj['end']['streams'][0]['sender'] -puts 'tcp.receiver.bps: ' + obj['end']['streams'][0]['receiver']['bits_per_second'].to_s if obj['end']['streams'][0]['receiver'] -puts 'udp.bps: ' + obj['end']['streams'][0]['udp']['bits_per_second'].to_s if obj['end']['streams'][0]['udp'] +if obj['end']['streams'][0]['sender'] + puts 'tcp.sender.bps: ' + obj['end']['streams'][0]['sender']['bits_per_second'].to_s +end +if obj['end']['streams'][0]['receiver'] + puts 'tcp.receiver.bps: ' + obj['end']['streams'][0]['receiver']['bits_per_second'].to_s +end +if obj['end']['streams'][0]['udp'] + puts 'udp.bps: ' + obj['end']['streams'][0]['udp']['bits_per_second'].to_s +end -- 2.23.0

1 1

[PATCH compass-ci] doc/job/: add suite.md
by Bai Jing 25 Nov '20

25 Nov '20

add a document for introducing the suite field Signed-off-by: Bai Jing <799286817(a)qq.com> --- doc/job/suite.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 doc/job/suite.md diff --git a/doc/job/suite.md b/doc/job/suite.md new file mode 100644 index 0000000..aecf06f --- /dev/null +++ b/doc/job/suite.md @@ -0,0 +1,9 @@ +# suite + +Meaning: +- Test suite is a collection of test cases. The test case and test purposes +are also described in detail, system configuration information during the test, +and steps to be performed before the test. A series of test cases serving the +same test purpose or running environment are combined through the test suite. +- When you write a job yaml, you should better to assign the value of the +program to the suite. -- 2.23.0

2 2

[PATCH lkp-tests] lkp-tests/*: update mount option
by Wang Yong 25 Nov '20

25 Nov '20

[why] mount error(2): No such file or directory [how] add port=446 option to mount.cifs Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 2 +- tests/cci-makepkg | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/cci-depends b/tests/cci-depends index 4b885dc0c..d5c21541d 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -30,7 +30,7 @@ pack_arch=$os_arch . $LKP_SRC/distro/$DISTRO is_mount_point ${DEPS_MNT} && umount ${DEPS_MNT} -mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$DEPS_MNT $DEPS_MNT || die "Failed to run mount" +mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$DEPS_MNT $DEPS_MNT || die "Failed to run mount" umask 002 diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e49f66998..d08f23718 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -36,7 +36,7 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" [ -n "$LKP_SERVER" ] && { is_mount_point ${PKG_MNT} && umount ${PKG_MNT} - mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } curl_pkgfile() -- 2.23.0

3 4

[PATCH v4 lkp-tests 1/2] tests/iozone: explicit support test=all
by Lu Kaiyi 25 Nov '20

25 Nov '20

[example] iozone-bs.yaml ... suite: iozone category: benchmark file_size: 4g test: all block_size: - 64k ... [output] ... Run began: Sat Nov 21 16:48:22 2020 Record Size 64 kB File size set to 4194304 kB Command line used: iozone -r 64k -s 4g -f /fs/sdb/iozonetest Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 4194304 64 13650392 11529057 7878373 7763936 7166639 10774159 7520609 16188351 7362482 9751567 9453804 7981606 7932187 --- tests/iozone | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/iozone b/tests/iozone index b6a1e206..bafc0087 100755 --- a/tests/iozone +++ b/tests/iozone @@ -19,6 +19,8 @@ if [ -n "$block_size" ]; then for ele in ${array[@]} do case $ele in + "all") args+=" -a" + ;; "write") args+=" -i 0" ;; "read") args+=" -i 1" -- 2.23.0

2 1

[PATCH v2 lkp-tests] lkp-tests/*: update mount option
by Wang Yong 24 Nov '20

24 Nov '20

[why] refer-to: compass-ci: commit 96584b45e528a4bad1f9 [how] add port=446 option to mount.cifs Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 2 +- tests/cci-makepkg | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/cci-depends b/tests/cci-depends index 4b885dc0c..d5c21541d 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -30,7 +30,7 @@ pack_arch=$os_arch . $LKP_SRC/distro/$DISTRO is_mount_point ${DEPS_MNT} && umount ${DEPS_MNT} -mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$DEPS_MNT $DEPS_MNT || die "Failed to run mount" +mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$DEPS_MNT $DEPS_MNT || die "Failed to run mount" umask 002 diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e49f66998..d08f23718 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -36,7 +36,7 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" [ -n "$LKP_SERVER" ] && { is_mount_point ${PKG_MNT} && umount ${PKG_MNT} - mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr,port=446 //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } curl_pkgfile() -- 2.23.0

1 0

[PATCH lkp-tests] lkp-tests/*: update AUR Helper for archlinux
by Wang Yong 24 Nov '20

24 Nov '20

yaourt is unmaintained and yay also support native pacman operation, so only minor changes are needed Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/archlinux | 15 +++++++++++++++ distro/installer/archlinux | 15 +++++++++------ lib/install.sh | 2 +- 3 files changed, 25 insertions(+), 7 deletions(-) diff --git a/distro/archlinux b/distro/archlinux index 408c5fa23..8e3fbbce7 100755 --- a/distro/archlinux +++ b/distro/archlinux @@ -109,3 +109,18 @@ pack_benchmark() ) done } + +add_aur_helper() +{ + pacman -Q yay 2>/dev/null && return + + local user=lkp + local dir=/home/${user} + + mkdir -p ${dir} && chown ${user}:${user} ${dir} + echo "lkp ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers + + pacman -S --needed --noconfirm git base-devel 2>/dev/null + su - ${user} -c "git clone https://aur.archlinux.org/yay.git" + su - ${user} -c "cd yay && makepkg -si --needed --noconfirm" +} diff --git a/distro/installer/archlinux b/distro/installer/archlinux index 27cdc63cf..af98e466f 100755 --- a/distro/installer/archlinux +++ b/distro/installer/archlinux @@ -2,12 +2,15 @@ # Archlinux package installation -# yaourt is a wrapper for pacman that adds automated access -# to the AUR. -# See: https://wiki.archlinux.org/index.php/Yaourt +# Yet Another Yogurt - An AUR Helper Written in Go +# Yay is based on the design of yaourt, apacman and pacaur +# See: https://github.com/Jguer/yay -# yaourt not support root to build package +# Yay not support root to build package # Use regular user to do this -regular_user=`grep 1000 /etc/passwd | awk -F':' '{ print $1 }'` +. $LKP_SRC/distro/${DISTRO} +add_aur_helper -sudo -u $regular_user yaourt -S --noconfirm $* +regular_user=`grep 1090 /etc/passwd | awk -F':' '{ print $1 }'` + +sudo -u $regular_user yay -S --needed --noconfirm $* diff --git a/lib/install.sh b/lib/install.sh index 881ade9f4..bf7bf21d7 100755 --- a/lib/install.sh +++ b/lib/install.sh @@ -17,7 +17,7 @@ sync_distro_sources() yum update fi ;; redhat) yum update ;; - archlinux) yaourt -Sy ;; + archlinux) yay -Sy ;; opensuse) zypper update ;; oracle) yum update ;; -- 2.23.0

1 1

[PATCH v3 lkp-tests] tests/iozone: explicit support test=all and add invalid test option identification
by Lu Kaiyi 24 Nov '20

24 Nov '20

[example] iozone-bs.yaml ... suite: iozone category: benchmark file_size: 4g test: all block_size: - 64k ... [output] ... Run began: Sat Nov 21 16:48:22 2020 Record Size 64 kB File size set to 4194304 kB Command line used: iozone -r 64k -s 4g -f /fs/sdb/iozonetest Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 4194304 64 13650392 11529057 7878373 7763936 7166639 10774159 7520609 16188351 7362482 9751567 9453804 7981606 7932187 --- tests/iozone | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/tests/iozone b/tests/iozone index b6a1e206..5186766f 100755 --- a/tests/iozone +++ b/tests/iozone @@ -19,6 +19,8 @@ if [ -n "$block_size" ]; then for ele in ${array[@]} do case $ele in + "all") args+=" -a" + ;; "write") args+=" -i 0" ;; "read") args+=" -i 1" @@ -45,6 +47,9 @@ if [ -n "$block_size" ]; then ;; "preadv") args+=" -i 12" ;; + *) echo "invalid test option, please check again!" + exit 1 + ;; esac done else -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: avoid saving empty dictionary
by Wei Jihui 24 Nov '20

24 Nov '20

[why] when add a program to pp, it will declare { pp: program: {} } if the program does not has args, it will keep {} to pp. [how] give a nil to program, then add program args when the program adds args. it will: { pp: program: } Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 2e94316c..84ab5056 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -580,11 +580,14 @@ class Job program_name = hash_key_re_string(h, p_n, @job['pp'].keys) next if program_name.nil? - @job["pp"][program_name] = Hash.new() + @job['pp'][program_name] = nil options_array = options_hash.keys options_array.each do |option| option_value = get_program_option_value(@job, program_name, option) - @job["pp"][program_name][option] = option_value unless option_value.nil? + next if option_value.nil? + + @job['pp'][program_name] ||= {} + @job['pp'][program_name][option] = option_value end end end -- 2.23.0

1 0

[PATCH v3 compass-ci] optimize document for apply account
by Luan Shengde 24 Nov '20

24 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/apply-ssh-account.md | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/doc/apply-ssh-account.md b/doc/apply-ssh-account.md index d718d0d..a4902f5 100644 --- a/doc/apply-ssh-account.md +++ b/doc/apply-ssh-account.md @@ -36,20 +36,12 @@ for external user email subject: apply account email body: - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + example: + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… attachment: ssh pubkey file - it will return email with my_email, my_name, my_uuid if successfully applied account -2. prepare to submit job - execute on user's local host - - config default yaml file: - ~/.config/compass-ci/default/account.yaml - - prepare job.yaml - - submit job - -3. apply account email format - example: + - email example: To: compass-ci(a)qq.com Subject: apply account @@ -58,5 +50,19 @@ for external user # We'll validate whether the URL contains your email. # for example, my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… - # attach your ssh pub key to the email as attachment + +2. successfully applied account email + it will return an email with following information: + - my_email + - my_name + - my_uuid + - SCHED_HOST + - SCHED_PORT + +3. environment configuration + follow steps in the email to finish the following configuration + - setup default yaml + ~/.config/compass-ci/default/account.yaml + - download and install lkp-tests + - prepare to submit jobs -- 2.23.0

1 0

[PATCH v10 compass-ci 2/2] lib: fix group compare no output when dimension number less than 2
by Lu Kaiyi 24 Nov '20

24 Nov '20

[why] When do group compare, if returned groups of auto_group like below (the dimension number less than 2 for group_key): groups = { group_key_1 => { dimension_1 => matrix_1 }, group_key_2 => {} } the group_key will be deleted, then, group compare will have no output. [how] Reduce dimension number limit from 2 to 1. [example] lukaiyi@z9 ~% compare "suite=iperf" -d "os" in z9 [before] <null> [after] os_arch=aarch64/os_version=20/pp.iperf.protocol=tcp/pp.iperf.runtime=300/tbox_group=vm-hi1620-2p8g--liangt openeuler metric -------------------- ------------------------------ fails:runs | 18:18 stderr.EOF' 18:18 stderr.has_stderr openeuler metric -------------------- ------------------------------ %stddev \ 3.496630e+10 ± 19% iperf.tcp.receiver.bps 3.497117e+10 ± 19% iperf.tcp.sender.bps 300.23 iperf.time.elapsed_time 300.23 iperf.time.elapsed_time.max ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 2 +- lib/params_group.rb | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2065039..82cf033 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,7 +116,7 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| - if value.size < 2 + if value.empty? groups.delete(group_key) next end diff --git a/lib/params_group.rb b/lib/params_group.rb index 40191a9..48d651b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -34,8 +34,7 @@ def auto_group(jobs_list, dimensions) return [] if dimensions.empty? jobs_list = extract_jobs_list(jobs_list) - groups = group(jobs_list, dimensions) - return remove_singleton(groups) + group(jobs_list, dimensions) end def extract_jobs_list(jobs_list) -- 2.23.0

1 0

[PATCH v10 compass-ci 1/2] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 24 Nov '20

24 Nov '20

[why] Avoid to execute some unnecessary operations and improve performance. [how] Move the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ff4ea31..2065039 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

1 0

[PATCH v2 lkp-tests] spec/submit: add job yaml for multi-args test
by Bai Jing 24 Nov '20

24 Nov '20

[test cmd] respec $LKP_SRC/spec/submit_spec.rb [output] lkp-tests/spec/submit/matrices-args/unixbench-spec.yaml => /lkp-tests/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml lkp-tests/spec/submit/matrices-args/unixbench-spec.yaml => /lkp-tests/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrices-args/unixbench-spec.yaml | 10 +++ ...unixbench-spec-1-300-whetstone-double.yaml | 78 +++++++++++++++++++ .../unixbench-spec-3-300-shell1.yaml | 78 +++++++++++++++++++ spec/submit_spec.rb | 4 + 4 files changed, 170 insertions(+) create mode 100644 spec/submit/matrices-args/unixbench-spec.yaml create mode 100644 spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml diff --git a/spec/submit/matrices-args/unixbench-spec.yaml b/spec/submit/matrices-args/unixbench-spec.yaml new file mode 100644 index 00000000..39447528 --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec.yaml @@ -0,0 +1,10 @@ +suite: unixbench +category: benchmark + +runtime: 300s + +nr_task test: + - 1 whetstone-double + - 3 shell1 + +unixbench: diff --git a/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml new file mode 100644 index 00000000..abc7fd53 --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml @@ -0,0 +1,78 @@ +--- +suite: unixbench +category: benchmark +runtime: 300 +unixbenc: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kms: +boot-tim: +uptim: +iosta: +heartbea: +vmsta: +numa-numasta: +numa-vmsta: +numa-meminf: +proc-vmsta: +proc-sta: +meminf: +slabinf: +interrupt: +lock_sta: +latency_stat: +softirq: +bdi_dev_mappin: +diskstat: +nfssta: +cpuidl: +cpufreq-stat: +sched_debu: +perf-sta: +mpsta: +perf-profil: +cpufreq_governo: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: {} +monitors: + kms: + boot-tim: + uptim: + iosta: + heartbea: + vmsta: + numa-numasta: + numa-vmsta: + numa-meminf: + proc-vmsta: + proc-sta: + meminf: + slabinf: + interrupt: + lock_sta: + latency_stat: + softirq: + bdi_dev_mappin: + diskstat: + nfssta: + cpuidl: + cpufreq-stat: + sched_debu: + perf-sta: + mpsta: + perf-profil: diff --git a/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml new file mode 100644 index 00000000..559a04d2 --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml @@ -0,0 +1,78 @@ +--- +suite: unixbench +category: benchmark +runtime: 300 +unixbenc: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kms: +boot-tim: +uptim: +iosta: +heartbea: +vmsta: +numa-numasta: +numa-vmsta: +numa-meminf: +proc-vmsta: +proc-sta: +meminf: +slabinf: +interrupt: +lock_sta: +latency_stat: +softirq: +bdi_dev_mappin: +diskstat: +nfssta: +cpuidl: +cpufreq-stat: +sched_debu: +perf-sta: +mpsta: +perf-profil: +cpufreq_governo: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: {} +monitors: + kms: + boot-tim: + uptim: + iosta: + heartbea: + vmsta: + numa-numasta: + numa-vmsta: + numa-meminf: + proc-vmsta: + proc-sta: + meminf: + slabinf: + interrupt: + lock_sta: + latency_stat: + softirq: + bdi_dev_mappin: + diskstat: + nfssta: + cpuidl: + cpufreq-stat: + sched_debu: + perf-sta: + mpsta: + perf-profil: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..ebebad5d 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'matrices-args spec' do + submit_job('matrices-args') + end end -- 2.23.0

1 0

[PATCH v2 compass-ci] optimize document for apply account
by Luan Shengde 24 Nov '20

24 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/apply-ssh-account.md | 30 ++++++++++++++++++------------ 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/doc/apply-ssh-account.md b/doc/apply-ssh-account.md index d718d0d..1844c86 100644 --- a/doc/apply-ssh-account.md +++ b/doc/apply-ssh-account.md @@ -36,20 +36,12 @@ for external user email subject: apply account email body: - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + example: + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… attachment: ssh pubkey file - it will return email with my_email, my_name, my_uuid if successfully applied account -2. prepare to submit job - execute on user's local host - - config default yaml file: - ~/.config/compass-ci/default/account.yaml - - prepare job.yaml - - submit job - -3. apply account email format - example: + - email example: To: compass-ci(a)qq.com Subject: apply account @@ -58,5 +50,19 @@ for external user # We'll validate whether the URL contains your email. # for example, my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… - # attach your ssh pub key to the email as attachment + +2. successfully applied account email + it will return an email with following infomation: + - my_email + - my_name + - my_uuid + - SCHED_HOST + - SCHED_PORT + +3. environment configuration + follow steps in the email to finish the following configuration + - setup default yaml + ~/.config/compass-ci/default/account.yaml + - download lkp-tests and dependencies + - prepare to submit jobs -- 2.23.0

2 2

[PATCH v4 compass-ci 2/2] lib/job.cr: verify account information
by Wu Zhende 24 Nov '20

24 Nov '20

Verify account information when submitting a job. Must key:my_email, my_name, my_uuid. Check whether the information is the same as that stored in the ES database. Also need update pub_key. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 51 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 48 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..785daea 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "../scheduler/elasticsearch_client" struct JSON::Any def []=(key : String, value : String) @@ -58,8 +59,11 @@ class Job return if @hash["id"] == "#{id}" end + @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") + @account_info = Hash(String, JSON::Any).new check_required_keys() + check_account_info() set_defaults() end @@ -152,7 +156,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() - set_sshr_port() + set_sshr_info() set_queue() end @@ -183,12 +187,34 @@ class Job end end - private def set_sshr_port + private def set_sshr_info return unless self["sshd"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] self["sshr_port_len"] = ENV["SSHR_PORT_LEN"] + + return if @account_info["found"]? == false + + update_sshd + end + + private def update_sshd + sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() + pub_key = sshd["pub_key"]?.to_s + update_account_my_pub_key(pub_key) + + sshd["pub_key"] = @account_info["my_ssh_pubkey"] + @hash["sshd"] = JSON.parse(sshd.to_json) + end + + private def update_account_my_pub_key(pub_key) + my_ssh_pubkey = @account_info["my_ssh_pubkey"].as_a + return if pub_key.empty? || my_ssh_pubkey.includes?(pub_key) + + my_ssh_pubkey << JSON::Any.new(pub_key) + @account_info["my_ssh_pubkey"] = JSON.parse(my_ssh_pubkey.to_json) + @es.update_account(JSON.parse(@account_info.to_json), self["my_email"].to_s) end private def set_os_dir @@ -271,6 +297,9 @@ class Job id suite testbox + my_email + my_name + my_uuid ] private def check_required_keys @@ -281,6 +310,20 @@ class Job end end + private def check_account_info + error_msg = "Failed to verify the account. Please check your configuration" + account_info = @es.get_account(self["my_email"]) + raise account_info unless account_info.is_a?(JSON::Any) + + @account_info = account_info.as_h + + raise error_msg if @account_info["found"]? == false + raise error_msg unless self["my_name"] == @account_info["my_name"].to_s + raise error_msg unless self["my_uuid"] == @account_info["my_uuid"] + + @hash.delete("my_uuid") + end + private def initialized? initialized_keys = [] of String @@ -302,6 +345,8 @@ class Job "SCHED_HOST", "SCHED_PORT"] + initialized_keys -= ["my_uuid"] + initialized_keys.each do |key| return false unless @hash.has_key?(key) end @@ -441,7 +486,7 @@ class Job else program_version = "latest" end - + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] scheduler/elasticsearch_client.cr: add two account functions
by Wu Zhende 24 Nov '20

24 Nov '20

add two functions about account 1. get_account 2. update_account Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index ec6de0d..7c92204 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -69,6 +69,24 @@ class Elasticsearch::Client return job end + def get_account(my_email : String) + query = {:index => "accounts", :type => "_doc", :id => my_email} + response = JSON.parse({"_id" => my_email, "found" => false}.to_json) + return response unless @client.exists(query) + + @client.get_source(query) + end + + def update_account(account_content : JSON::Any, my_email : String) + return @client.update( + { + :index => "accounts", :type => "_doc", + :id => my_email, + :body => {:doc => account_content} + } + ) + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

1 0

[PATCH v6 lkp-tests] pkg: add mysql
by Lin Jiaxin 24 Nov '20

24 Nov '20

sysbench-mysql need to test with mysql8.0.17 Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...00-mysql-add-fstack-protector-strong.patch | 20 ++++++ pkg/mysql/PKGBUILD | 60 +++++++++++++++++ pkg/mysql/my.cnf | 67 +++++++++++++++++++ 3 files changed, 147 insertions(+) create mode 100644 pkg/mysql/0000-mysql-add-fstack-protector-strong.patch create mode 100644 pkg/mysql/PKGBUILD create mode 100644 pkg/mysql/my.cnf diff --git a/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch new file mode 100644 index 00000000..23c07c87 --- /dev/null +++ b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch @@ -0,0 +1,20 @@ +diff --git a/cmake/build_configurations/compiler_options.cmake b/cmake/build_configurations/compiler_options.cmake +index fecf2a2..f3e012f 100644 +--- a/cmake/build_configurations/compiler_options.cmake ++++ b/cmake/build_configurations/compiler_options.cmake +@@ -43,6 +43,7 @@ IF(UNIX) + # Default GCC flags + IF(CMAKE_COMPILER_IS_GNUCC) + SET(COMMON_C_FLAGS "-fno-omit-frame-pointer") ++ SET(COMMON_C_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_C_FLAGS "-fno-inline ") +@@ -57,6 +58,7 @@ IF(UNIX) + ENDIF() + IF(CMAKE_COMPILER_IS_GNUCXX) + SET(COMMON_CXX_FLAGS "-std=c++14 -fno-omit-frame-pointer") ++ SET(COMMON_CXX_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_CXX_FLAGS "-fno-inline ") diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD new file mode 100644 index 00000000..8869e62b --- /dev/null +++ b/pkg/mysql/PKGBUILD @@ -0,0 +1,60 @@ +pkgname=mysql +pkgver=8.0.17 +pkgrel=3 +arch=('aarch64' 'x86_64') +url="http://www.mysql.com" +license=('GPLv2') +source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" + "0000-mysql-add-fstack-protector-strong.patch" + "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" + "my.cnf") +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' '7bb0372ec4e9281813a9afc67c6cfc20') + +prepare() +{ + cd $srcdir/$pkgname-$pkgver + patch -p1 < ../0000-mysql-add-fstack-protector-strong.patch +} + +build() +{ + build_getcpu + build_mysql +} + +build_mysql() +{ + cd $srcdir/$pkgname-$pkgver + + CMAKE=cmake + grep -sqF "CentOS Linux release 7" /etc/centos-release && CMAKE=cmake3 + + $CMAKE . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc \ + -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_PARTITION_STORAGE_ENGINE=1 -DWITH_FEDERATED_STORAGE_ENGINE=1 \ + -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_MYISAM_STORAGE_ENGINE=1 \ + -DENABLED_LOCAL_INFILE=1 -DENABLE_DTRACE=0 -DDEFAULT_CHARSET=utf8mb4 -DDEFAULT_COLLATION=utf8mb4_general_ci \ + -DWITH_EMBEDDED_SERVER=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=./boost -DFORCE_INSOURCE_BUILD=1 + make -j8 +} + +build_getcpu() +{ + cd $srcdir + tar -zxvf getcpu.tgz + + cd getcpu || exit + make +} + + +package() +{ + mkdir -p $pkgdir/getcpu + cp getcpu/getcpu-hi.ko $pkgdir/getcpu/ + + mkdir -p $pkgdir/etc + cp my.cnf $pkgdir/etc/ + + cd $srcdir/$pkgname-$pkgver + make DESTDIR=$pkgdir install +} diff --git a/pkg/mysql/my.cnf b/pkg/mysql/my.cnf new file mode 100644 index 00000000..9732af40 --- /dev/null +++ b/pkg/mysql/my.cnf @@ -0,0 +1,67 @@ +[mysqld_safe] +log-error=/data/mysql/log/mysql.log +pid-file=/data/mysql/run/mysqld.pid +[mysqldump] +quick +[mysql] +[client] +default-character-set=utf8mb4 +socket=/data/mysql/run/mysql.sock + +[mysqld] +socket=/data/mysql/run/mysql.sock +tmpdir=/data/mysql/tmp +datadir=/data/mysql/data +default_authentication_plugin=mysql_native_password +port=3309 +user=mysql +memlock=on +bind-address=0.0.0.0 +skip-grant-tables + +lc-messages=en_US +lc-messages-dir=/usr/local/mysql/share + +innodb_page_size=64k +max_connections=4000 +back_log=2048 +performance_schema=OFF +max_prepared_stmt_count=128000 + +#file +innodb_file_per_table +innodb_log_file_size=2048M +innodb_log_files_in_group=32 +innodb_open_files=10000 +table_open_cache_instances=64 + +#buffers +innodb_buffer_pool_size=128G +innodb_buffer_pool_instances=64 +innodb_log_buffer_size=2048M + +#tune +sync_binlog=0 +innodb_flush_log_at_trx_commit=0 +innodb_use_native_aio=1 +innodb_spin_wait_delay=160 +innodb_sync_spin_loops=20 +innodb_flush_method=O_DIRECT +innodb_io_capacity=30000 +innodb_io_capacity_max=40000 +innodb_lru_scan_depth=9000 +innodb_page_cleaners=1 +innodb_spin_wait_pause_multiplier=25 + +#perf special +innodb_flush_neighbors=0 +innodb_write_io_threads=1 +innodb_read_io_threads=1 +innodb_purge_threads=1 + +sql_mode=STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_AUTO_VALUE_ON_ZERO,STRICT_ALL_TABLES + +#skop_log_bin +ssl=0 + +table_open_cache=15000 -- 2.23.0

1 0

[PATCH compass-ci] container/os-cifs: add the container to share /srv/os
by Xiao Shenwei 24 Nov '20

24 Nov '20

[why] our os-cifs container will mount three dir, /srv/os, /srv/initrd, /srv/result in order to reduce the coupling, need split them [how] one folder mount by one container [usage] mount -t cifs o guest //ip/os /tmp/os mount -t cifs o guest //ip/os-rw /tmp/os-rw Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/os-cifs/Dockerfile | 8 ++++---- container/os-cifs/smb.conf | 26 ++------------------------ container/os-cifs/start | 7 +++---- 3 files changed, 9 insertions(+), 32 deletions(-) diff --git a/container/os-cifs/Dockerfile b/container/os-cifs/Dockerfile index 66f978d..c22ce35 100644 --- a/container/os-cifs/Dockerfile +++ b/container/os-cifs/Dockerfile @@ -4,16 +4,16 @@ FROM alpine:edge -RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN adduser -u 1090 -D lkp - RUN apk add --update \ samba-common-tools \ samba-client \ samba-server \ - bash \ - && rm -rf /var/cache/apk/* + bash +RUN rm -rf /var/cache/apk/* COPY ./smb.conf /etc/samba/ diff --git a/container/os-cifs/smb.conf b/container/os-cifs/smb.conf index 5ff6841..5e454db 100644 --- a/container/os-cifs/smb.conf +++ b/container/os-cifs/smb.conf @@ -18,6 +18,7 @@ server min protocol = NT1 unix extensions = yes mangled names = no + [os] path = /srv/os/ comment = os @@ -28,6 +29,7 @@ # that can only be accessed by root. force user = root force group = root + [os-rw] path = /srv/os/ comment = os-rw @@ -36,27 +38,3 @@ public = yes force user = root force group = root -[initrd] - path = /srv/initrd/ - comment = initrd - browseable = yes - writable = yes - public = yes - force user = lkp - force group = lkp -[osimage] - path = /srv/initrd/ - comment = initrd - browseable = yes - writable = yes - public = yes - force user = lkp - force group = lkp -[result] - path = /srv/result/ - comment = result - browseable = yes - writable = yes - public = yes - force user = lkp - force group = lkp diff --git a/container/os-cifs/start b/container/os-cifs/start index f42c71c..5e08238 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -11,12 +11,11 @@ lsmod | grep -q "^cifs\s" || { docker_rm os-cifs cmd=( - docker run -dt + docker run + -d -p 445:445 - -v /srv/os:/srv/os -v /etc/localtime:/etc/localtime:ro - -v /srv/initrd:/srv/initrd - -v /srv/result:/srv/result + -v /srv/os:/srv/os --name os-cifs --restart=always os-cifs -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] scheduler/elasticsearch_client.cr: add two account functions
by Wu Zhende 24 Nov '20

24 Nov '20

add two functions about account 1. get_account 2. update_account Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index ec6de0d..7c92204 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -69,6 +69,24 @@ class Elasticsearch::Client return job end + def get_account(my_email : String) + query = {:index => "accounts", :type => "_doc", :id => my_email} + response = JSON.parse({"_id" => my_email, "found" => false}.to_json) + return response unless @client.exists(query) + + @client.get_source(query) + end + + def update_account(account_content : JSON::Any, my_email : String) + return @client.update( + { + :index => "accounts", :type => "_doc", + :id => my_email, + :body => {:doc => account_content} + } + ) + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

2 1

[PATCH compass-ci] srv-http/start: add os images data download directory
by Cui Yili 24 Nov '20

24 Nov '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index b56fe4d..8c8ca0b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -14,6 +14,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/os:/srv/os:ro -d srv-http ) -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] lib/job.cr: check account info when submit job
by Wu Zhende 24 Nov '20

24 Nov '20

When a job submit, check it's account info. Must key:my_email, my_name, my_uuid. Check whether the information is the same as that stored in the ES database. Also need update pub_key. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 51 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 48 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..785daea 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "../scheduler/elasticsearch_client" struct JSON::Any def []=(key : String, value : String) @@ -58,8 +59,11 @@ class Job return if @hash["id"] == "#{id}" end + @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") + @account_info = Hash(String, JSON::Any).new check_required_keys() + check_account_info() set_defaults() end @@ -152,7 +156,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() - set_sshr_port() + set_sshr_info() set_queue() end @@ -183,12 +187,34 @@ class Job end end - private def set_sshr_port + private def set_sshr_info return unless self["sshd"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] self["sshr_port_len"] = ENV["SSHR_PORT_LEN"] + + return if @account_info["found"]? == false + + update_sshd + end + + private def update_sshd + sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() + pub_key = sshd["pub_key"]?.to_s + update_account_my_pub_key(pub_key) + + sshd["pub_key"] = @account_info["my_ssh_pubkey"] + @hash["sshd"] = JSON.parse(sshd.to_json) + end + + private def update_account_my_pub_key(pub_key) + my_ssh_pubkey = @account_info["my_ssh_pubkey"].as_a + return if pub_key.empty? || my_ssh_pubkey.includes?(pub_key) + + my_ssh_pubkey << JSON::Any.new(pub_key) + @account_info["my_ssh_pubkey"] = JSON.parse(my_ssh_pubkey.to_json) + @es.update_account(JSON.parse(@account_info.to_json), self["my_email"].to_s) end private def set_os_dir @@ -271,6 +297,9 @@ class Job id suite testbox + my_email + my_name + my_uuid ] private def check_required_keys @@ -281,6 +310,20 @@ class Job end end + private def check_account_info + error_msg = "Failed to verify the account. Please check your configuration" + account_info = @es.get_account(self["my_email"]) + raise account_info unless account_info.is_a?(JSON::Any) + + @account_info = account_info.as_h + + raise error_msg if @account_info["found"]? == false + raise error_msg unless self["my_name"] == @account_info["my_name"].to_s + raise error_msg unless self["my_uuid"] == @account_info["my_uuid"] + + @hash.delete("my_uuid") + end + private def initialized? initialized_keys = [] of String @@ -302,6 +345,8 @@ class Job "SCHED_HOST", "SCHED_PORT"] + initialized_keys -= ["my_uuid"] + initialized_keys.each do |key| return false unless @hash.has_key?(key) end @@ -441,7 +486,7 @@ class Job else program_version = "latest" end - + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] stats/mysql-server: remove redundant whitespace
by Lu Kaiyi 24 Nov '20

24 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/mysql-server | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/stats/mysql-server b/stats/mysql-server index d3f3fd79..776f3f2e 100755 --- a/stats/mysql-server +++ b/stats/mysql-server @@ -16,9 +16,9 @@ while (line = STDIN.gets) # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 when /\[\s*\d+\%\]/ - results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') - results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') - results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') + results << line.split(']')[1].split[0] + '.pass: 1' if line.include?('[ pass ]') + results << line.split(']')[1].split[0] + '.fail: 1' if line.include?('[ fail ]') + results << line.split(']')[1].split[0] + '.skip: 1' if line.include?('[ skipped ]') end end -- 2.23.0

2 1

[PATCH v2 lkp-tests 2/2] stats/crystal: mysql-server to mysql-server.cr
by Lu Kaiyi 24 Nov '20

24 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/mysql-server.cr | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 stats/crystal/mysql-server.cr diff --git a/stats/crystal/mysql-server.cr b/stats/crystal/mysql-server.cr new file mode 100644 index 00000000..e2ab4b8e --- /dev/null +++ b/stats/crystal/mysql-server.cr @@ -0,0 +1,21 @@ +#!/usr/bin/env crystal +# frozen_string_literal: true + +results = [] of String + +while (line = STDIN.gets) + case line + # The output is as below: + # 200706 14:35:48 [ 99%] main.subquery_sj_innodb_all w8 [ pass ] 140 + # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 + # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 + when /\[\s*\d+\%\]/ + results << line.split(']')[1].split[0] + ".pass: 1" if line.includes?("[ pass ]") + results << line.split(']')[1].split[0] + ".fail: 1" if line.includes?("[ fail ]") + results << line.split(']')[1].split[0] + ".skip: 1" if line.includes?("[ skipped ]") + end +end + +results.each do |item| + puts item +end -- 2.23.0

2 1

[PATCH compass-ci] optimize document for apply account
by Luan Shengde 24 Nov '20

24 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/apply-ssh-account.md | 31 +++++++++++++++++++------------ 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/doc/apply-ssh-account.md b/doc/apply-ssh-account.md index d718d0d..b8dd4bb 100644 --- a/doc/apply-ssh-account.md +++ b/doc/apply-ssh-account.md @@ -36,20 +36,12 @@ for external user email subject: apply account email body: - my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + example: + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… attachment: ssh pubkey file - it will return email with my_email, my_name, my_uuid if successfully applied account -2. prepare to submit job - execute on user's local host - - config default yaml file: - ~/.config/compass-ci/default/account.yaml - - prepare job.yaml - - submit job - -3. apply account email format - example: + - email example: To: compass-ci(a)qq.com Subject: apply account @@ -58,5 +50,20 @@ for external user # We'll validate whether the URL contains your email. # for example, my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… - # attach your ssh pub key to the email as attachment + +2. successfully applied account email + it will returns email with following info: + - my_email + - my_name + - my_uuid + - sched_host + - schec_port + +3. environment configuration + you will get email after successfully applied account + follow steps in the email to finish the following configuration + - setup default yaml + ~/.config/compass-ci/default/account.yaml + - download lkp-tests and dependencies + - prepare to submit jobs -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] scheduler/elasticsearch_client.cr: add two account function
by Wu Zhende 24 Nov '20

24 Nov '20

add two function about account Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index ec6de0d..efe304c 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -69,6 +69,28 @@ class Elasticsearch::Client return job end + def get_account(my_email : String) + if @client.exists({:index => "accounts", :type => "_doc", :id => my_email}) + response = @client.get_source({:index => "accounts", :type => "_doc", :id => my_email}) + else + response = "" + end + + response = JSON.parse({"_id" => my_email, "found" => false}.to_json) if response.is_a?(String) + + return response + end + + def update_account(account_content : JSON::Any, my_email : String) + return @client.update( + { + :index => "accounts", :type => "_doc", + :id => my_email, + :body => {:doc => account_content} + } + ) + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

3 3

[PATCH lkp-tests] stats/crystal: mysql-server to mysql-server.cr
by Lu Kaiyi 24 Nov '20

24 Nov '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/mysql-server.cr | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 stats/crystal/mysql-server.cr diff --git a/stats/crystal/mysql-server.cr b/stats/crystal/mysql-server.cr new file mode 100644 index 00000000..ce3903ef --- /dev/null +++ b/stats/crystal/mysql-server.cr @@ -0,0 +1,21 @@ +#!/usr/bin/env crystal +# frozen_string_literal: true + +results = [] of String + +while (line = STDIN.gets) + case line + # The output is as below: + # 200706 14:35:48 [ 99%] main.subquery_sj_innodb_all w8 [ pass ] 140 + # 200706 14:35:52 [ 99%] main.ssl_dynamic_persisted w1 [ fail ] 5953 + # 200706 14:35:59 [ 99%] main.mysql_upgrade_grant w7 [ skipped ] 80805 + when /\[\s*\d+\%\]/ + results << line.split(']')[1].split[0] + ".pass: 1" if line.includes?("[ pass ]") + results << line.split(']')[1].split[0] + ".fail: 1" if line.includes?("[ fail ]") + results << line.split(']')[1].split[0] + ".skip: 1" if line.includes?("[ skipped ]") + end +end + +results.each do |item| + puts item +end -- 2.23.0

1 0

[PATCH v6 compass-ci 1/2] result-webdav/start: add share dir to upload file
by Cui Yili 24 Nov '20

24 Nov '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index e0b6676..01141c3 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -14,6 +14,7 @@ cmd=( -p 3080:3080 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf -v /srv/result/:/srv/result/ + -v /srv/initrd/:/srv/initrd/ result-webdav ) -- 2.23.0

2 1

[PATCH compass-ci] about multi-device test
by Ren Wen 24 Nov '20

24 Nov '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- doc/multi-device-test.md | 108 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 doc/multi-device-test.md diff --git a/doc/multi-device-test.md b/doc/multi-device-test.md new file mode 100644 index 0000000..226287f --- /dev/null +++ b/doc/multi-device-test.md @@ -0,0 +1,108 @@ +# Multi-device Test # + +��, ��. + +## When using multi-device ## + + * �� + * �� + +## How to run a multi-device test ## + +��job.yaml��, ��**cluster**��, �� +��, ��: *lab-z9/cluster/*. +��job��job, �� +��. + +��? +��job.yaml��: + +``` yaml +if role <role01> + <test_script01> + <test_script02> + +if role <role02> + <test_script03> + <test_script04> +... +``` + +��, ��. �� +job��, ��. + +��: ��*role01*, �� +*test_script01*, *test_script02*, ��. + +### how to find test script ### + +��lkp��**daemon**, **tests**��. +��**daemon**��*test_script01*, ��`start_daemon`�� +��. ��**tests**��*test_script01*, ��`run_test`�� +��. ��, ��. + +### about `start_daemon` and `run_test` ### + +��2��**$LKP_SRC/lib/job.sh**. +��, ��. + +#### the processes of `start_damon` #### + + 1. ��**daemon**��. ��, ��: + `started`, ��: `start`; ��, ��: + `failed`, ��: `abort`, ��. ��redis�� + key: `HGETALL sched/cluster_state`. + 2. ��: `write_state`, ��: + * `node_roles`: �� + * `ip`: ��`hostname -I`�� + * `direct_macs`: ��**macs**�� + * `direct_ips`: ��**ip0**��ip, �� + ��**macs**�� + ��, ��redis��key: + **sched/cluster_state**. + 3. ��**direct_ips**��**direct_macs**��. + 4. ��: `wait_ready`, ��: `ready`; �� + ��, ��. �� + ��, ��`ready`; ��`abort`, �� + `abort`, ��; ��, ��, ��2��. + 5. ��: `roles_ip`, ��. + ��: `ip=ip01`��, ��: `export ip=ip01`. + 6. ��: `wait_finish`, ��: `finish`; �� + ��, ��`finish`. �� + ��`finish`, ��`finish`; ��`abort`, �� + `abort`, ��; ��, ��, ��2��. + 7. ��. + +#### the processes of `run_test` #### + +��1,2,3,4��2,3,4,5. + + 1. ��: `write_state`, ��: + * `node_roles`: �� + * `ip`: ��`hostname -I`�� + * `direct_macs`: ��**macs**�� + * `direct_ips`: ��**ip0**��ip, �� + ��**macs**�� + ��, ��redis��key: + **sched/cluster_state**. + 2. ��**direct_ips**��**direct_macs**��. + 3. ��: `wait_ready`, ��: `ready`; �� + ��, ��. �� + ��, ��`ready`; ��`abort`, �� + `abort`, ��; ��, ��, ��2��. + 4. ��: `roles_ip`, ��. + ��: `ip=ip01`��, ��: `export ip=ip01` + 5. ��**tests**��. ��, ��: + `finished`, ��: `finish`; ��, ��: `failed`, �� + ��: `abort`, ��. + 6. ��. + +#### other #### + +`start_daemon`��`run_test`��`abort`��: ��100�� +��, ��`ready`��`finish`, �� +`retry`, ��, ��. + +��, ��(��)��, �� +��`ready`(`finish`)��`abort`. ��100��, �� +��`abort`��. -- 2.23.0

2 4

[PATCH v2 compass-ci 2/2] lib/job.cr: check account info when submit job
by Wu Zhende 24 Nov '20

24 Nov '20

[Why] When a job submit, check it's account info. Must key:my_email, my_name, my_uuid Check whether the information is the same as that stored in the ES database Also need update pub_key. [Why comment code] Because now many people don't have an account. If the verification is mandatory, many users can't use the service. Therefore, use a gentle approach to promote accounts first. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 46 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 43 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..cfd52d7 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "../scheduler/elasticsearch_client" struct JSON::Any def []=(key : String, value : String) @@ -58,8 +59,11 @@ class Job return if @hash["id"] == "#{id}" end + @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") + @account_info = Hash(String, JSON::Any).new check_required_keys() + check_account_info() set_defaults() end @@ -152,7 +156,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() - set_sshr_port() + set_sshr_info() set_queue() end @@ -183,12 +187,35 @@ class Job end end - private def set_sshr_port + private def set_sshr_info return unless self["sshd"]? self["sshr_port"] = ENV["SSHR_PORT"] self["sshr_port_base"] = ENV["SSHR_PORT_BASE"] self["sshr_port_len"] = ENV["SSHR_PORT_LEN"] + + return if @account_info["found"]? == false + + sshd = update_my_pub_key + @hash["sshd"] = JSON.parse(sshd.to_json) + end + + private def update_my_pub_key + sshd = @hash["sshd"].as_h? || Hash(String, JSON::Any).new() + pub_key = sshd["pub_key"]?.to_s + my_ssh_pubkey = @account_info["my_ssh_pubkey"].as_a + + if pub_key.empty? || my_ssh_pubkey.includes?(pub_key) + sshd["pub_key"] = @account_info["my_ssh_pubkey"] + return sshd + end + + my_ssh_pubkey << JSON::Any.new(pub_key) + @account_info["my_ssh_pubkey"] = JSON.parse(my_ssh_pubkey.to_json) + @es.update_account(JSON.parse(@account_info.to_json), self["my_email"].to_s) + + sshd["pub_key"] = @account_info["my_ssh_pubkey"] + return sshd end private def set_os_dir @@ -281,6 +308,17 @@ class Job end end + private def check_account_info + # error_msg = "Failed to verify the account. Please check your configuration" + @account_info = @es.get_account(self["my_email"]).as_h + + # raise error_msg if @account_info["found"]? == false + # raise error_msg unless self["my_name"] == @account_info["my_name"].to_s + # raise error_msg unless self["my_uuid"] == @account_info["my_uuid"] + + @hash.delete("my_uuid") + end + private def initialized? initialized_keys = [] of String @@ -302,6 +340,8 @@ class Job "SCHED_HOST", "SCHED_PORT"] + initialized_keys -= ["my_uuid"] + initialized_keys.each do |key| return false unless @hash.has_key?(key) end @@ -441,7 +481,7 @@ class Job else program_version = "latest" end - + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" -- 2.23.0

2 1

[PATCH lkp-tests] spec/submit: add job yaml for multi-args test
by Bai Jing 24 Nov '20

24 Nov '20

[test cmd] respec $LKP_SRC/spec/submit_spec.rb [output] lkp-tests/spec/submit/matrices-args/unixbench-spec.yaml => /lkp-tests/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml lkp-tests/spec/submit/matrices-args/unixbench-spec.yaml => /lkp-tests/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml Signed-off-by: Bai Jing <799286817(a)qq.com> --- spec/submit/matrices-args/unixbench-spec.yaml | 10 +++ ...unixbench-spec-1-300-whetstone-double.yaml | 78 +++++++++++++++++++ .../unixbench-spec-3-300-shell1.yaml | 78 +++++++++++++++++++ spec/submit_spec.rb | 4 + 4 files changed, 170 insertions(+) create mode 100644 spec/submit/matrices-args/unixbench-spec.yaml create mode 100644 spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml create mode 100644 spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml diff --git a/spec/submit/matrices-args/unixbench-spec.yaml b/spec/submit/matrices-args/unixbench-spec.yaml new file mode 100644 index 00000000..39447528 --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec.yaml @@ -0,0 +1,10 @@ +suite: unixbench +category: benchmark + +runtime: 300s + +nr_task test: + - 1 whetstone-double + - 3 shell1 + +unixbench: diff --git a/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml new file mode 100644 index 00000000..abc7fd53 --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-1-300-whetstone-double.yaml @@ -0,0 +1,78 @@ +--- +suite: unixbench +category: benchmark +runtime: 300 +unixbenc: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '1' +test: whetstone-double +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kms: +boot-tim: +uptim: +iosta: +heartbea: +vmsta: +numa-numasta: +numa-vmsta: +numa-meminf: +proc-vmsta: +proc-sta: +meminf: +slabinf: +interrupt: +lock_sta: +latency_stat: +softirq: +bdi_dev_mappin: +diskstat: +nfssta: +cpuidl: +cpufreq-stat: +sched_debu: +perf-sta: +mpsta: +perf-profil: +cpufreq_governo: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: whetstone-double + nr_task: '1' + cpufreq_governor: {} +monitors: + kms: + boot-tim: + uptim: + iosta: + heartbea: + vmsta: + numa-numasta: + numa-vmsta: + numa-meminf: + proc-vmsta: + proc-sta: + meminf: + slabinf: + interrupt: + lock_sta: + latency_stat: + softirq: + bdi_dev_mappin: + diskstat: + nfssta: + cpuidl: + cpufreq-stat: + sched_debu: + perf-sta: + mpsta: + perf-profil: diff --git a/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml new file mode 100644 index 00000000..559a04d2 --- /dev/null +++ b/spec/submit/matrices-args/unixbench-spec/unixbench-spec-3-300-shell1.yaml @@ -0,0 +1,78 @@ +--- +suite: unixbench +category: benchmark +runtime: 300 +unixbenc: +lab: spec_lab +testbox: vm-2p8g--spec_submit +nr_task: '3' +test: shell1 +tbox_group: vm-2p8g--spec_submit +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G +arch: aarch64 +kms: +boot-tim: +uptim: +iosta: +heartbea: +vmsta: +numa-numasta: +numa-vmsta: +numa-meminf: +proc-vmsta: +proc-sta: +meminf: +slabinf: +interrupt: +lock_sta: +latency_stat: +softirq: +bdi_dev_mappin: +diskstat: +nfssta: +cpuidl: +cpufreq-stat: +sched_debu: +perf-sta: +mpsta: +perf-profil: +cpufreq_governo: +need_kconfig: +- CONFIG_KVM_GUEST=y +pp: + unixbench: + runtime: 300 + test: shell1 + nr_task: '3' + cpufreq_governor: {} +monitors: + kms: + boot-tim: + uptim: + iosta: + heartbea: + vmsta: + numa-numasta: + numa-vmsta: + numa-meminf: + proc-vmsta: + proc-sta: + meminf: + slabinf: + interrupt: + lock_sta: + latency_stat: + softirq: + bdi_dev_mappin: + diskstat: + nfssta: + cpuidl: + cpufreq-stat: + sched_debu: + perf-sta: + mpsta: + perf-profil: diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..fc34f58d 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + it 'matrices-args spec' do + submit_job('matrices-args') + end + end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/json_logger.cr: add a new class for json log
by Wu Zhende 24 Nov '20

24 Nov '20

[Why] The built-in log system prints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example1] use: log.info("test") output: {"level":"INFO","level_num":1,"time":"2020-11-19 16:57:17+0800","progname":"test","message":"test"} [Example2] use: log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":"test", "job_id":"1","job_state":"submit","caller":["./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 src/lib/json_logger.cr diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr new file mode 100755 index 0000000..371c042 --- /dev/null +++ b/src/lib/json_logger.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "logger" +require "json" +require "any_merge" + +class JSONLogger < Logger + FORMATTER = Logger::Formatter.new do | severity, datetime, progname, msg, io| + level_num = severity.to_i32 + logger_hash = JSON.parse(%({"level_num": #{level_num}, + "level": "#{severity}", + "time": "#{datetime}" + })).as_h + + logger_hash.any_merge!({"progname" => progname}) unless progname.empty? + logger_hash.merge!(JSON.parse(%({"caller": #{caller}})).as_h) if level_num >= 2 + + begin + message = JSON.parse(msg).as_h + rescue + message = {"message" => msg} + end + logger_hash.any_merge!(message) + + io << logger_hash.to_json + end + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/json_logger.rb: refactor log.rb
by Wu Zhende 24 Nov '20

24 Nov '20

1. change log.rb to json_logger.rb 2. change Log to JSONLogger 3. change datetime to time 4. don't output empty values 5. msg use hash or string(originally json or string) Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/{log.rb => json_logger.rb} | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) rename lib/{log.rb => json_logger.rb} (65%) diff --git a/lib/log.rb b/lib/json_logger.rb similarity index 65% rename from lib/log.rb rename to lib/json_logger.rb index aad00ca..6d1173e 100644 --- a/lib/log.rb +++ b/lib/json_logger.rb @@ -6,7 +6,7 @@ require 'logger' require 'json' # print logs in JSON format -class Log < Logger +class JSONLogger < Logger LEVEL_INFO = { 'DEBUG' => 0, 'INFO' => 1, @@ -18,21 +18,19 @@ class Log < Logger FORMATTER = proc { |severity, datetime, progname, msg| level_num = LEVEL_INFO[severity] - begin - message = JSON.parse(msg) - rescue JSON::ParserError - message = { 'message' => msg } - end - h = { + logger_hash = { 'level' => severity.to_s, 'level_num' => level_num, - 'datetime' => datetime, - 'progname' => progname, - 'message' => '' + 'time' => datetime } - h.merge!(message) - h.merge!({ 'caller' => caller }) if level_num >= 2 - h.to_json + + logger_hash['progname'] = progname if progname + logger_hash['caller'] = caller if level_num >= 2 + + msg = { 'message' => msg } unless msg.is_a?(Hash) + logger_hash.merge!(msg) + + logger_hash.to_json } def initialize(logdev = STDOUT, formatter = FORMATTER) -- 2.23.0

1 0

[PATCH v6 compass-ci 2/2] container/result-webdav: add symbolic link after uploading file
by Cui Yili 24 Nov '20

24 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..c880522 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -13,8 +13,10 @@ http { server_name "result-webdav"; server_tokens off; client_max_body_size 100m; + access_log /tmp/access.log; + error_log /tmp/error.log; - location / { + location /result { allow all; root /srv/; @@ -39,6 +41,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + upload_file = string.match(uri, ".+/([^/]*%.%w+)$") + second_dir = string.match(uri, "/.+/(.*)/.+") + + if (second_dir == "pkg") + then + link_name = "latest.cgz" + else + link_name = string.match(upload_file, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..upload_file.." "..link_name) + } + } } } -- 2.23.0

1 0

[PATCH v5 compass-ci] container/result-webdav: add symbolic link after uploading file
by Cui Yili 24 Nov '20

24 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..1085492 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -14,7 +14,7 @@ http { server_tokens off; client_max_body_size 100m; - location / { + location /result { allow all; root /srv/; @@ -39,6 +39,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + second_dir = string.match(uri, "/.+/(.*)/.+") + + if (second_dir == "pkg") + then + link_name = "latest.cgz" + else + link_name = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..link_name) + } + } } } -- 2.23.0

3 2

[PATCH lkp-tests] makepkg.conf: mv -Wl from LDFLAGS to C(CXX)FLAGS
by Xu Xijian 24 Nov '20

24 Nov '20

-Wl parameter should be used by CFLAGS and CXXFLAGS instead of LDFLAGS. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- etc/makepkg.conf | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/etc/makepkg.conf b/etc/makepkg.conf index 093789f5..1fffdc2d 100644 --- a/etc/makepkg.conf +++ b/etc/makepkg.conf @@ -36,9 +36,8 @@ VCSCLIENTS=('bzr::bzr' # -march (or -mcpu) builds exclusively for an architecture # -mtune optimizes for an architecture, but builds for whole processor family CPPFLAGS="-D_FORTIFY_SOURCE=2" -CFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt" -CXXFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt" -LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" +CFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" +CXXFLAGS="-march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now" #-- Make Flags: change this for DistCC/SMP systems #MAKEFLAGS="-j2" #-- Debugging flags -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/2] job: fix each_jobs to fit multi-args
by Bai Jing 24 Nov '20

24 Nov '20

[why] if you add job field like this : os: openeuler os_version: 20.03 os_arch: aarch64 nr_threads: - 1 - 2 then the @jobs=[hash], hash={"nr_threads"=>[1,2],"os"=>"openeuler","os_arch"=>"aarch64","os_version"=>"20.03"} if you add job field like this: os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 nr_threads: - 1 - 2 then the @jobs=[h1, h2], h1={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20.03","os_arch"=>"aarch64"} h2={"nr_threads"=>[1, 2],"os"=>"openeuler", "os_version"=>"20","os_arch"=>"aarch64"} h2 will split to 2 jobs by function "each_job" Should: job1={"nr_threads"=>1} job2={"nr_threads"=>2} But if "@job2=hash",will get: job1={"nr_threads"=>[1, 2]} job2={"nr_threads"=>[1, 2]} Because "@job2=hash" will merge the h2 to @job after "each_job". We need to merge the h2 to @job before "each_job". So do the merge advance and set @job2={} Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index bc6c91ef..2062ba58 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -727,7 +727,8 @@ class Job @jobs.each do |hash| @load_hosts_done = false @job = deepcopy(job) - @job2 = hash + @job.merge!(hash) + @job2 = {} load_defaults load_hosts_config each_job_init -- 2.23.0

1 0

[PATCH v4 lkp-tests 1/2] lib: add multi-args in job yaml
by Bai Jing 24 Nov '20

24 Nov '20

[original style] os: openeuler os_version: 20.03 os_arch: aarch64 os: openeuler os_version: 20 os_arch: aarch64 [now style] os os_version os_arch: - openeuler 20.03 aarch64 - openeuler 20 aarch64 [point] Currently, only one key of this style(xx xxx xx) can be contained in YAML files. Signed-off-by: Bai Jing <799286817(a)qq.com> --- lib/job.rb | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 2e94316c..bc6c91ef 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -250,7 +250,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) - @jobs << hash + @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e log_error "#{jobfile}: " + e.message @@ -280,6 +280,30 @@ class Job @jobfile = jobfile end + def multi_args(hash) + jobs_array = [] + hash.each { |key, value| + next unless key =~ /^\w.*\s+.*\w$/ + + next unless value.class == Array + + key_array = key.split + value.each do |v| + v_array = v.split + next unless key_array.size == v_array.size + + hash_job = hash.clone + key_array.size.times do |i| + hash_job[key_array[i]] = v_array[i] + end + hash_job.delete(key) + jobs_array << hash_job + end + } + jobs_array = [hash] if jobs_array.empty? + return jobs_array + end + def delete_keys_from_spec(spec_file) return unless File.exist? spec_file -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/build-pkg: fix syntax errors caused by replacement
by Lin Jiaxin 24 Nov '20

24 Nov '20

error: //PKGBUILD: line 13: syntax error near unexpected token `)' //PKGBUILD: line 13: ` "$_pkgname.desktop")' The original logic only adapts to the situation where source has only one value. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index c5d55eda..65c83e22 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -90,6 +90,11 @@ create_softlink() ln -sf "${bm_name}" "${soft_path}/latest.cgz" } +replace_source() +{ + sed -i "s|^source=[^)]*|$1|g" "$2" +} + build_source_pkg() { local repo_dir="" @@ -103,11 +108,9 @@ build_source_pkg() repo_dir="${repo_dir%.git*}::" fi - upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\")" - sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD - [ -n "$PKGBUILD_TAG" ] && { - sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" - } + upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + replace_source "${upstream_source}" PKGBUILD + [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" } cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" -- 2.23.0

1 0

[PATCH lkp-tests] lib/job2sh.rb: the job value obtaining logic is modified
by Hu Xuejiao 24 Nov '20

24 Nov '20

[why] the original logic is too round Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 14bf7159..7f9cf562 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -287,9 +287,7 @@ class Job2sh < Job @setups = available_programs(:setup) @programs = available_programs(:workload_elements) - if !job - job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job - end + job ||= (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job @cur_func = :extract_stats @@ -317,7 +315,7 @@ class Job2sh < Job @setups = available_programs(:setup) @programs = available_programs(:workload_elements) - if !job + unless job job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job job.delete_if { |key, val| parse_one([], key, val, :PASS_EXPORT_ENV) } end -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/wrapper support user define file to parse result
by Lu Weitao 23 Nov '20

23 Nov '20

[Why] User need debug their script which haven't commit to lkp-tests/stats [How] lkp-tests/stats/wrapper calls the program-script in result_root if lkp-tests/stats/ have no program-script [Dependence] Write job['define_files'] to file in result_root, job can query from ES DB Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/wrapper | 3 +++ 1 file changed, 3 insertions(+) diff --git a/stats/wrapper b/stats/wrapper index a44e9f31..d13249d2 100755 --- a/stats/wrapper +++ b/stats/wrapper @@ -140,6 +140,9 @@ check_empty_output check_binary_output tmpfile=$(mktemp -p /tmp -t lkp-stats.XXXXXXXX) + +[[ -f "$RESULT_ROOT/stats/$program" ]] && program_dir="$RESULT_ROOT/stats" + if [[ -f $log ]]; then $program_dir/$program $log < $log > $tmpfile || { log_error "$program_dir/$program $log < $log exit code $?, check $tmpfile"; exit 1; } else -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] doc/job/: add define_files.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing the define_files field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/define_files.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/define_files.md diff --git a/doc/job/define_files.md b/doc/job/define_files.md new file mode 100644 index 0000000..2c3457d --- /dev/null +++ b/doc/job/define_files.md @@ -0,0 +1,16 @@ +# define_files + +Meaning: +- We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: + distro/depends/$testcase + distro/depends/$testcase-dev + pkg/$testcase/PKGBUILD + stats/$testcase + tests/$testcase + jobs/$testcase.yaml +- When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. +- The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. + +Usage example: +- % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 +- % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] doc/job/: add initrd_deps.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing initrd_deps field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/initrd_deps.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/job/initrd_deps.md diff --git a/doc/job/initrd_deps.md b/doc/job/initrd_deps.md new file mode 100644 index 0000000..0c3d79c --- /dev/null +++ b/doc/job/initrd_deps.md @@ -0,0 +1,5 @@ +# initrd_deps + +Meaning: +- Most testcases depend on many other package to run. If user has already packed all the other package for running testcase. Then initrd_deps field would give the location of the package. +- The initrd_deps field do not need to be specified by user in the $testcase yaml. It is filled in by the scheduler. User can get the package through the wget command to check the rpm in compressed package. -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] doc/job/: add initrd_pkg.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing the initrd_pkg field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/initrd_pkg.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/job/initrd_pkg.md diff --git a/doc/job/initrd_pkg.md b/doc/job/initrd_pkg.md new file mode 100644 index 0000000..2ef3d03 --- /dev/null +++ b/doc/job/initrd_pkg.md @@ -0,0 +1,5 @@ +# initrd_pkg + +Meaning: +- Most testcases rely on many other files such as configuration files to run. If user has already packed all the files for running testcase. Then initrd_pkg field would give the location of the packed compression file. +- The initrd_pkg field do not need to be specified by user in the $testcase yaml. It is filled in by the scheduler. User can get the package through the wget command to check the files in compressed package. -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: return ips of all nodes
by Ren Wen 23 Nov '20

23 Nov '20

Return all nodes' ips when request state is "roles_ip". Example: If roles of all nodes are: "server", "client", ... ips of all nodes are: "<ip01>", "<ip02>", ... Then return: sever=<ip01> client=<ip02> ... All nodes will export the return content as environment variables line by line. --- src/scheduler/request_cluster_state.cr | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index 5bbbad1..72e3810 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -52,11 +52,16 @@ class Sched "direct_macs" => direct_macs} update_cluster_state(cluster_id, job_id, job_info) when "roles_ip" - role = "server" - role_state = get_role_state(cluster_id, role) - raise "Missing #{role} state in cluster state" unless role_state - return "server=#{role_state["ip"]}\n" \ - "direct_server_ips=#{role_state["direct_ips"]}" + cluster_state = get_cluster_state(cluster_id) + roles_ip = [] of String + + cluster_state.each_value do |host_state| + roles = host_state["roles"] + direct_ips = host_state["direct_ips"] + roles_ip << "#{roles}=#{direct_ips}" + end + + return roles_ip.join('\n') end # show cluster state -- 2.23.0

2 2

[PATCH compass-ci] lib/job.cr: fix repeated setting job
by Wu Zhende 23 Nov '20

23 Nov '20

Only first time should full initialize the job. Next time function "initialized?" will return true, just to use job_content, don't need to set default values. But now, "kernel_version" will let "initialized?" return false. Fix this. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..512f256 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -63,6 +63,8 @@ class Job set_defaults() end + # Each key must exist after the job is initialized + # Otherwise "initialized"? will return false METHOD_KEYS = %w( id os @@ -91,7 +93,6 @@ class Job kernel_append_root kernel_params docker_image - kernel_version linux_vmlinuz_path linux_modules_initrd linux_headers_initrd @@ -312,7 +313,7 @@ class Job private def set_kernel_version boot_dir = "#{SRV_OS}/#{os_dir}/boot" - suffix = "-#{kernel_version}" if self["kernel_version"]? + suffix = "-#{self["kernel_version"]}" if self["kernel_version"]? self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") if "#{os_mount}" == "initramfs" self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") -- 2.23.0

2 1

[PATCH compass-ci 1/3] doc/job/: add define_files.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing the define_files field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/define_files.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/define_files.md diff --git a/doc/job/define_files.md b/doc/job/define_files.md new file mode 100644 index 0000000..dec74f1 --- /dev/null +++ b/doc/job/define_files.md @@ -0,0 +1,16 @@ +# define_files + +Meaning: +- We use lkp-tests as client for compass-ci. There are many testcases in lkp-tests. most testcases contain the following files: + distro/depends/$testcase + distro/depends/$testcase-dev + pkg/$testcase/PKGBUILD + stats/$testcase + tests/$testcase + jobs/$testcase.yaml +- When user add new testcase files to lkp-tests or change existing testcase files in lkp-tests, these change files that related with test program would be added to define_files field. +- The define_files field do not need to be specified by user in the $testcase.yaml. It can be generated automatically by the scheduler when user submit with the option '-a'. User can confirm the define_files field in the job.yaml under result_root. + +Usage example: +- % submit -a $testcase.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 +- % cat /srv/result/$testcase/vm-2p8g--$USER/$date/$job_id/job.yaml -- 2.23.0

2 2

[PATCH compass-ci 2/2] lib/json_logger.cr: add a new class for json log
by Wu Zhende 23 Nov '20

23 Nov '20

[Why] The built-in log system prints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example1] use: log.info("test") output: {"level":"INFO","level_num":1,"time":"2020-11-19 16:57:17+0800","progname":"test","message":"test"} [Example2] use: log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":"test", "job_id":"1","job_state":"submit","caller":["./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 src/lib/json_logger.cr diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr new file mode 100755 index 0000000..371c042 --- /dev/null +++ b/src/lib/json_logger.cr @@ -0,0 +1,32 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "logger" +require "json" +require "any_merge" + +class JSONLogger < Logger + FORMATTER = Logger::Formatter.new do | severity, datetime, progname, msg, io| + level_num = severity.to_i32 + logger_hash = JSON.parse(%({"level_num": #{level_num}, + "level": "#{severity}", + "time": "#{datetime}" + })).as_h + + logger_hash.any_merge!({"progname" => progname}) unless progname.empty? + logger_hash.merge!(JSON.parse(%({"caller": #{caller}})).as_h) if level_num >= 2 + + begin + message = JSON.parse(msg).as_h + rescue + message = {"message" => msg} + end + logger_hash.any_merge!(message) + + io << logger_hash.to_json + end + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

2 1

[PATCH v10 compass-ci 3/3] lib: fix group compare no output when dimension number less than 2
by Lu Kaiyi 23 Nov '20

23 Nov '20

[why] When do group compare, if returned groups of auto_group like below (the dimension number less than 2 for group_key): groups = { group_key_1 => { dimension_1 => matrix_1 }, group_key_2 => {} } the group_key will be deleted, then, group compare will have no output. [how] Reduce dimension number limit from 2 to 1. [example] lukaiyi@z9 ~% compare "suite=iperf" -d "os" in z9 [before] <null> [after] os_arch=aarch64/os_version=20/pp.iperf.protocol=tcp/pp.iperf.runtime=300/tbox_group=vm-hi1620-2p8g--liangt openeuler metric -------------------- ------------------------------ fails:runs | 18:18 stderr.EOF' 18:18 stderr.has_stderr openeuler metric -------------------- ------------------------------ %stddev \ 3.496630e+10 ± 19% iperf.tcp.receiver.bps 3.497117e+10 ± 19% iperf.tcp.sender.bps 300.23 iperf.time.elapsed_time 300.23 iperf.time.elapsed_time.max ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 2 +- lib/params_group.rb | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2065039..82cf033 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,7 +116,7 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| - if value.size < 2 + if value.empty? groups.delete(group_key) next end diff --git a/lib/params_group.rb b/lib/params_group.rb index 40191a9..48d651b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -34,8 +34,7 @@ def auto_group(jobs_list, dimensions) return [] if dimensions.empty? jobs_list = extract_jobs_list(jobs_list) - groups = group(jobs_list, dimensions) - return remove_singleton(groups) + group(jobs_list, dimensions) end def extract_jobs_list(jobs_list) -- 2.23.0

1 0

[PATCH v10 compass-ci 2/3] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 23 Nov '20

23 Nov '20

[why] Avoid to execute some unnecessary operations and improve performance. [how] Move the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ff4ea31..2065039 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

1 0

[PATCH v10 compass-ci 1/3] lib: keep suite result in top for compare output
by Lu Kaiyi 23 Nov '20

23 Nov '20

[why] Currently, the compare output result default ordered by alphabet. If there are too many lines in result, it will be difficult for user to find the lines that start with $suite in metric column. So, need reorder compare output result with keeping $suite item on the top of the table. [how] Divide the lines into two parts before printing output according to whether the lines start with $suite, then combine output result into a whole to highlight the important info. [example] lukaiyi@z9 ~% compare "id=z9.153094" "id=z9.153725" in z9 suites_list:["sysbench-mysql"] [before] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 58.80 +0.0% 58.82 boot-time.boot 5522.36 -0.1% 5515.17 boot-time.idle 21.22 -0.1% 21.19 boot-time.kernel_boot 9913.00 -2.1% 9703.00 interrupts.228:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 ... 396953.00 -0.6% 394693.00 softirqs.SCHED 26445.00 -5.8% 24902.00 softirqs.TASKLET 1364789.00 -0.2% 1362689.00 softirqs.TIMER 24925.67 +1.3% 25240.95 sysbench-mysql.transactions 2585.92 +0.2% 2592.20 uptime.boot 239274.99 +0.2% 239786.47 uptime.idle 9062055.00 -0.0% 9059563.00 vmstat.memory.cache 243316968.00 +0.0% 243352799.00 vmstat.memory.free ... [after] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 24925.67 +1.3% 25240.95 sysbench-mysql.transactions 58.80 +0.0% 58.82 boot-time.boot 5522.36 -0.1% 5515.17 boot-time.idle 21.22 -0.1% 21.19 boot-time.kernel_boot 9913.00 -2.1% 9703.00 interrupts.228:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 3526557.00 +5.1% 3707673.00 interrupts.4:GICv3.26.Level.arch_timer ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 16 ++++++++++------ lib/compare_matrixes.rb | 41 ++++++++++++++++++++++++++++------------- lib/matrix2.rb | 20 +++++++++++++++----- 3 files changed, 53 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..f34e239 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suite_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suite_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,16 @@ end def create_matrices_list(conditions) matrices_list = [] + suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suite_list.concat(suites) end - matrices_list + + return matrices_list, suite_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 762c087..14c14cb 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -128,7 +128,7 @@ def set_compare_values(index, values, field, success, options) end def get_values_by_field(matrixes_list, field, matrixes_size, success, options) - # get values by field, values struce example: values[0][:average] + # get values by field, values struct example: values[0][:average] # values = {} matrixes_list.length.times do |index| @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suite_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -808,26 +809,40 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suite_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suite_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suite_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suite_list, matrixes_titles, success, theme) return '' if values.empty? + suite_set = Set.new(suite_list) result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + ranked_str = get_ranked_str(values, suite_set, success, theme) + result_str += ranked_str + result_str +end + +def get_ranked_str(values, suite_set, success, theme) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suite_set.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end - result_str + suite_str + common_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..ff4ea31 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,23 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end - groups.delete(group_key) if value.size < 2 + if value.size < 2 + groups.delete(group_key) + next + end + suites_list << suite_list end - groups + + return groups, suites_list end # input: @@ -139,7 +149,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], _ = create_matrix(jobs) end end -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] sbin/my-submit: this script is used together with the my-qemu
by Xiao Shenwei 23 Nov '20

23 Nov '20

you can ignore the queue field, it will add default value which is "$tbox_group~$USER" usage: my-submit testbox=vm-2p8g jobs/borrow-1h.yaml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/my-submit | 3 +++ 1 file changed, 3 insertions(+) create mode 100755 sbin/my-submit diff --git a/sbin/my-submit b/sbin/my-submit new file mode 100755 index 00000000..57489948 --- /dev/null +++ b/sbin/my-submit @@ -0,0 +1,3 @@ +#!/bin/bash + +submit --my-queue $@ -- 2.23.0

3 2

[PATCH compass-ci 1/2] lib/json_logger.rb: refactor log.rb
by Wu Zhende 23 Nov '20

23 Nov '20

1.change log.rb to json_logger.rb 2.change Log to JSONLogger 3.change datetime to time 4.don't output empty values 5.msg use hash or string(originally json or string) Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/{log.rb => json_logger.rb} | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) rename lib/{log.rb => json_logger.rb} (65%) diff --git a/lib/log.rb b/lib/json_logger.rb similarity index 65% rename from lib/log.rb rename to lib/json_logger.rb index aad00ca..6d1173e 100644 --- a/lib/log.rb +++ b/lib/json_logger.rb @@ -6,7 +6,7 @@ require 'logger' require 'json' # print logs in JSON format -class Log < Logger +class JSONLogger < Logger LEVEL_INFO = { 'DEBUG' => 0, 'INFO' => 1, @@ -18,21 +18,19 @@ class Log < Logger FORMATTER = proc { |severity, datetime, progname, msg| level_num = LEVEL_INFO[severity] - begin - message = JSON.parse(msg) - rescue JSON::ParserError - message = { 'message' => msg } - end - h = { + logger_hash = { 'level' => severity.to_s, 'level_num' => level_num, - 'datetime' => datetime, - 'progname' => progname, - 'message' => '' + 'time' => datetime } - h.merge!(message) - h.merge!({ 'caller' => caller }) if level_num >= 2 - h.to_json + + logger_hash['progname'] = progname if progname + logger_hash['caller'] = caller if level_num >= 2 + + msg = { 'message' => msg } unless msg.is_a?(Hash) + logger_hash.merge!(msg) + + logger_hash.to_json } def initialize(logdev = STDOUT, formatter = FORMATTER) -- 2.23.0

2 1

[PATCH compass-ci] sched: save the upload dirs in redis
by Cao Xueliang 23 Nov '20

23 Nov '20

Before: Use access_key_file to limit job upload, need to create the file in upload dirs, that's not flexible enough. After: Set $job_id => "$dir1,$dir2,..." in "id2upload_dirs" hash of the redis. The dir of job upload must be equal or deeper than one of "$dir1,$dir2,...". Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 13 ++++--------- src/scheduler/close_job.cr | 6 ------ src/scheduler/find_job_boot.cr | 10 +++++----- 3 files changed, 9 insertions(+), 20 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..cb187a4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -83,8 +83,7 @@ class Job pp_params submit_date result_root - access_key - access_key_file + upload_dirs lkp_initrd_user user_lkp_src kernel_uri @@ -206,15 +205,11 @@ class Job def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group self["result_root"] = File.join("/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}") - - # access_key has information based on "result_root" - # so when set result_root, we need redo set_ to update it - set_access_key() + set_upload_dirs() end - private def set_access_key - self["access_key"] = "#{Random::Secure.hex(10)}-#{id}" - self["access_key_file"] = File.join("/srv/", "#{result_root}", ".#{access_key}") + def set_upload_dirs + self["upload_dirs"] = "#{result_root}" end private def set_result_service diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d071d69..b5aaa63 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -5,8 +5,6 @@ class Sched def close_job(job_id : String) job = @redis.get_job(job_id) - delete_access_key_file(job) if job - response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -24,8 +22,4 @@ class Sched return %({"job_id": "#{job_id}", "job_state": "complete"}) end - - def delete_access_key_file(job : Job) - File.delete(job.access_key_file) if File.exists?(job.access_key_file) - end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b5a23c5..968b9ed 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -128,10 +128,6 @@ class Sched return response end - def touch_access_key_file(job : Job) - FileUtils.touch(job.access_key_file) - end - private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" response += job.initrds_uri @@ -142,8 +138,12 @@ class Sched return response end + def set_id2upload_dirs(job) + @redis.hash_set("sched/id2upload_dirs",job.id, job.upload_dirs) + end + def boot_content(job : Job | Nil, boot_type : String) - touch_access_key_file(job) if job + set_id2upload_dirs(job) if job case boot_type when "ipxe" -- 2.23.0

1 1

[PATCH v3 compass-ci] sbin/es-jobs-mapping.sh: Support more fields in "jobs" index
by Zhang Yuhang 23 Nov '20

23 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..8e19662 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,9 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' \ + -d '{"index.mapping.total_fields.limit": 10000}' fi fi -- 2.23.0

1 0

[PATCH v1 compass-ci] sched: set the upload dirs to redis
by Cao Xueliang 23 Nov '20

23 Nov '20

Before: Use access_key_file to limit job upload, need to create the file in upload dirs, that's not flexible enough. After: Set $job_id => "$dir1,$dir2,..." in "id2upload_dirs" hash of the redis. The dir of job upload must be equal or deeper than one of "$dir1,$dir2,...". Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 13 ++++--------- src/scheduler/close_job.cr | 6 ------ src/scheduler/find_job_boot.cr | 10 +++++----- 3 files changed, 9 insertions(+), 20 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..cb187a4 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -83,8 +83,7 @@ class Job pp_params submit_date result_root - access_key - access_key_file + upload_dirs lkp_initrd_user user_lkp_src kernel_uri @@ -206,15 +205,11 @@ class Job def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group self["result_root"] = File.join("/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}") - - # access_key has information based on "result_root" - # so when set result_root, we need redo set_ to update it - set_access_key() + set_upload_dirs() end - private def set_access_key - self["access_key"] = "#{Random::Secure.hex(10)}-#{id}" - self["access_key_file"] = File.join("/srv/", "#{result_root}", ".#{access_key}") + def set_upload_dirs + self["upload_dirs"] = "#{result_root}" end private def set_result_service diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index d071d69..b5aaa63 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -5,8 +5,6 @@ class Sched def close_job(job_id : String) job = @redis.get_job(job_id) - delete_access_key_file(job) if job - response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -24,8 +22,4 @@ class Sched return %({"job_id": "#{job_id}", "job_state": "complete"}) end - - def delete_access_key_file(job : Job) - File.delete(job.access_key_file) if File.exists?(job.access_key_file) - end end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index b5a23c5..968b9ed 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -128,10 +128,6 @@ class Sched return response end - def touch_access_key_file(job : Job) - FileUtils.touch(job.access_key_file) - end - private def get_boot_ipxe(job : Job) response = "#!ipxe\n\n" response += job.initrds_uri @@ -142,8 +138,12 @@ class Sched return response end + def set_id2upload_dirs(job) + @redis.hash_set("sched/id2upload_dirs",job.id, job.upload_dirs) + end + def boot_content(job : Job | Nil, boot_type : String) - touch_access_key_file(job) if job + set_id2upload_dirs(job) if job case boot_type when "ipxe" -- 2.23.0

1 0

[PATCH v3 compass-ci] container/result-webdav: add symbolic link after uploading file
by Cui Yili 23 Nov '20

23 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..c9199ed 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -14,7 +14,7 @@ http { server_tokens off; client_max_body_size 100m; - location / { + location /result { allow all; root /srv/; @@ -39,6 +39,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + seconddir = string.match(uri, "/.+/(.*)/.+") + + if (seconddir == "pkg") + then + linkname = "latest.cgz" + else + linkname = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..linkname) + } + } } } -- 2.23.0

2 1

[PATCH compass-ci 2/2] container/result-webdav: add symbolic link after uploading file
by Cui Yili 23 Nov '20

23 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..0cb786c 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -13,8 +13,10 @@ http { server_name "result-webdav"; server_tokens off; client_max_body_size 100m; + access_log /tmp/access.log; + error_log /tmp/error.log; - location / { + location /result { allow all; root /srv/; @@ -39,6 +41,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + seconddir = string.match(uri, "/.+/(.*)/.+") + + if (seconddir == "pkg") + then + linkname = "latest.cgz" + else + linkname = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..linkname) + } + } } } -- 2.23.0

2 1

[PATCH v2 compass-ci] container/result-webdav: add symbolic link after uploading file
by Cui Yili 23 Nov '20

23 Nov '20

[why] Use curl to upload files, cannot add symbolic links. [how] nginx adds a symbolic link locally after receiving the file. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 5ef3647..34df004 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -14,7 +14,7 @@ http { server_tokens off; client_max_body_size 100m; - location / { + location /result { allow all; root /srv/; @@ -39,6 +39,32 @@ http { io.close(f) } } + location /initrd { + allow all; + root /srv/; + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + + access_by_lua_block { + uri = ngx.var.request_uri + + path = string.match(uri,"%g*/") + uploadfile = string.match(uri, ".+/([^/]*%.%w+)$") + seconddir = string.match(uri, "/.+/(.*)/.+") + + if (seconddir == "pkg") + then + linkname = "latest.cgz" + else + linkname = string.match(uploadfile, "(.*)_%d+%.cgz")..".cgz" + end + + root_patch = "/srv/"..path + io.popen("cd "..root_patch.."&& ln -sf "..uploadfile.." "..linkname) + } + } } } -- 2.23.0

1 0

[PATCH compass-ci 1/2] result-webdav/start: add share dir to upload file
by Cui Yili 23 Nov '20

23 Nov '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index e0b6676..01141c3 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -14,6 +14,7 @@ cmd=( -p 3080:3080 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf -v /srv/result/:/srv/result/ + -v /srv/initrd/:/srv/initrd/ result-webdav ) -- 2.23.0

1 0

[PATCH v2 compass-ci] add api /compare_template
by Lu Weitao 23 Nov '20

23 Nov '20

[Why] user need to compare the specific metrics [how] add new api: /compare_template that can be access like: curl -X POST 'localhost:32767/compare_template' -H 'Content-Type: Application/json' -d '{ "compare_metrics": ["fio.write_iops", "fio.read_iops"], "filter": {"suite": ["fio-basic"], "os_arch": ["aarch64", "x86"]}, "compare_dimensions": [ {"os": "debian", "os_version": "sid"}, {"os": "openeuler", "os_version": 20.03} ], "x_params": ["bs", "test_size"], "title": "Hackbench Performance Testing", "unit": "KB/s"}' Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 32 +++++++++++++++++++++++++++++++ src/lib/web_backend.rb | 18 +++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 7db82ac..d68e9f2 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -71,3 +71,35 @@ end get '/get_repos' do get_repos(params) end + +# POST /compare_template +# - header: "Content-Type: Application/json" +# - body: json +# eg: +# { +# "compare_metrics": ["fio.write_iops", "fio.read_iops"], +# "filter":{ +# "suite": ["fio-basic"], +# "os_arch": ["aarch64", "x86"] +# }, +# "compare_dimensions": [ +# { +# "os": "debian", +# "os_version": "sid" +# }, +# { +# "os": "openeuler", +# "os_version: 20.03 +# } +# ], +# "x_params": ["bs", "test_size"], +# "title": "Hackbench Performance Testing", +# "unit": "KB/s" +# } +# +# Response: +# - compare result +post '/compare_template' do + request.body.rewind # in case someone already read it + compare_template(request.body.read) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 1f7224e..23716cb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -413,3 +414,20 @@ def get_repos(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def compare_template(data) + begin + body = template_body(JSON.parse(data)) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'compare error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def template_body(request_body) + groups_matrices = create_groups_matrices(request_body) + compare_results = compare_metrics_values(groups_matrices) + formatter = FormatEchartData.new(compare_results, request_body) + formatter.format_for_echart.to_json +end -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix syntax errors caused by replacement
by Lin Jiaxin 23 Nov '20

23 Nov '20

error: //PKGBUILD: line 13: syntax error near unexpected token `)' //PKGBUILD: line 13: ` "$_pkgname.desktop")' The original logic only adapts to the situation where source has only one value. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index c5d55eda..6ad825d7 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -103,10 +103,10 @@ build_source_pkg() repo_dir="${repo_dir%.git*}::" fi - upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\")" - sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD + upstream_source="source=(\"${repo_dir}git://${LKP_SERVER}/${upstream_repo}#commit=${upstream_commit}\"" + sed -i "s|^source=[^)]*|${upstream_source}|g" PKGBUILD [ -n "$PKGBUILD_TAG" ] && { - sed -i "s|^source=.*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" + sed -i "s|^source=[^)]*|${upstream_source}|g" "PKGBUILD-$PKGBUILD_TAG" } } -- 2.23.0

2 1

[PATCH lkp-tests] stats/wrapper support user define file to parse result
by Lu Weitao 23 Nov '20

23 Nov '20

[Why] User need debug their script which haven't commit to lkp-tests/stats [How] lkp-tests/stats/wrapper calls the program-script in result_root if lkp-tests/stats/ have no program-script [Dependence] Write job['define_files'] to file in result_root, job can query from ES DB Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/wrapper | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/stats/wrapper b/stats/wrapper index a44e9f31..03db5975 100755 --- a/stats/wrapper +++ b/stats/wrapper @@ -140,6 +140,13 @@ check_empty_output check_binary_output tmpfile=$(mktemp -p /tmp -t lkp-stats.XXXXXXXX) + +if [[ ! -f $program_dir/$program ]]; then + if [[ -f $RESULT_ROOT/"stats"/$program ]]; then + program_dir=$RESULT_ROOT/"stats" + fi +fi + if [[ -f $log ]]; then $program_dir/$program $log < $log > $tmpfile || { log_error "$program_dir/$program $log < $log exit code $?, check $tmpfile"; exit 1; } else -- 2.23.0

2 2

[PATCH compass-ci] create define_files in result_root
by Lu Weitao 23 Nov '20

23 Nov '20

[Why] User need debug their result-parse-script which haven't commit in lkp-tests/stats. [How] We already support user write script as "define_file" => {"stats/$script_name" => "$script_content"} in job.yaml, the job.yaml will be save as job to ES DB, write the job['define_file']['stats/$script_name'](value) at $result_root/stats/$script, extract-service will parse result by $result_root/stats/$script(user's result-parse-script) firstly Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index bcf6e52..ae5d703 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -22,6 +22,8 @@ class StatsWorker if job_id # will consume the job by post-processing job = @es.get_job_content(job_id) result_root = job["result_root"]? + define_files = job["define_files"]? + write_user_stats(result_root.to_s, define_files.as_h) if define_files.is_a?(JSON::Any) result_post_processing(job_id, result_root.to_s, queue_path) @tq.delete_task(queue_path + "/in_process", "#{job_id}") end @@ -139,4 +141,17 @@ class StatsWorker STDERR.puts e.message end end + + def write_user_stats(result_root : String, define_files : Hash(String, JSON::Any)) + return nil unless result_root && File.exists?(result_root) + + define_files.each do |k, v| + if k.match (/^stats/) + Dir.mkdir("#{result_root}/stats") unless File.exists?("#{result_root}/stats") + File.open("#{result_root}/#{k}", "w+", File::Permissions.new(0o775)) do |file| + file.puts v + end + end + end + end end -- 2.23.0

3 4

[PATCH lkp-tests] iso2rootfs: fix the os_version of test job.yaml is not dailybuild
by Yu Chuan 23 Nov '20

23 Nov '20

[Why] The target iso version we want to generate: {iso_os}, {iso_arch}, {iso_version} The target rootfs version we want to test: {iso_os}, {iso_arch}, {iso_version}-dailybuild But now, we test {iso_version} instead of {iso_version}-dailybuild. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 0353c045a4bc..fcaadaa1ba46 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -454,7 +454,7 @@ generate_global_yaml() generate_submit_append_str() { [ "$#" -eq 2 ] || die "generate_submit_append_str: please give 2 parameters." - echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version} os_mount=$1 testbox=$2" + echo "os=${iso_os} os_arch=${iso_arch} os_version=${iso_version}-dailybuild os_mount=$1 testbox=$2" } prepare_submit() -- 2.23.0

1 0

[PATCH v9 compass-ci 1/3] lib: keep suite result in top for compare output
by Lu Kaiyi 23 Nov '20

23 Nov '20

[why] Currently, the compare output result default ordered by alphabet. If there are too many lines in result, it will be difficult for user to find the lines that start with $suite in metric column. So, need reorder compare output result with keeping $suite item on the top of the table. [how] Divide the lines into two parts before printing output according to whether the lines start with $suite, then combine output result into a whole to highlight the important info. [example] lukaiyi@z9 ~% compare "id=z9.148096" "id=z9.148143" in z9 suites_list:["sysbench-mysql"] [before] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 ... 1266042.00 -18.3% 1034206.00 softirqs.RCU 547849.00 -27.4% 397593.00 softirqs.SCHED 159990.00 -80.5% 31181.00 softirqs.TASKLET 1617967.00 -23.0% 1245331.00 softirqs.TIMER 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 2149.23 -100.0% 0.00 time.elapsed_time 2149.23 -100.0% 0.00 time.elapsed_time.max 214458.00 -100.0% 0.00 time.involuntary_context_switches 2.00 -100.0% 0.00 time.major_page_faults 192832.00 -100.0% 0.00 time.maximum_resident_set_size ... [after] 0 1 metric -------------------- ------------------------------ ------------------------------ %stddev change %stddev \ | \ 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time 2149.23 -100.0% 0.00 sysbench-mysql.time.elapsed_time.max 214458.00 -100.0% 0.00 sysbench-mysql.time.involuntary_context_switches 2.00 -100.0% 0.00 sysbench-mysql.time.major_page_faults 192832.00 -100.0% 0.00 sysbench-mysql.time.maximum_resident_set_size 90746.00 -100.0% 0.00 sysbench-mysql.time.minor_page_faults 65536.00 -100.0% 0.00 sysbench-mysql.time.page_size 201.00 -100.0% 0.00 sysbench-mysql.time.percent_of_cpu_this_job_got 2796.60 -100.0% 0.00 sysbench-mysql.time.system_time 1535.98 -100.0% 0.00 sysbench-mysql.time.user_time 282616989.00 -100.0% 0.00 sysbench-mysql.time.voluntary_context_switches 224.42 -74.1% 58.14 boot-time.boot 20400.95 -73.5% 5397.13 boot-time.idle 11.56 +83.3% 21.19 boot-time.kernel_boot 29424.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 4100080.00 -99.8% 10196.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 2894304.00 +12.8% 3265730.00 interrupts.4:GICv3.26.Level.arch_timer 11546998.00 -100.0% 0.00 interrupts.746:ITS-MSI.99092481.Edge.hns3-0000:bd:00.1-TxRx-0 7476064.00 -100.0% 0.00 interrupts.747:ITS-MSI.99092482.Edge.hns3-0000:bd:00.1-TxRx-2 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 16 ++++++++++------ lib/compare_matrixes.rb | 41 ++++++++++++++++++++++++++++------------- lib/matrix2.rb | 20 +++++++++++++++----- 3 files changed, 53 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..f34e239 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suite_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suite_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,16 @@ end def create_matrices_list(conditions) matrices_list = [] + suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suite_list.concat(suites) end - matrices_list + + return matrices_list, suite_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 762c087..14c14cb 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -128,7 +128,7 @@ def set_compare_values(index, values, field, success, options) end def get_values_by_field(matrixes_list, field, matrixes_size, success, options) - # get values by field, values struce example: values[0][:average] + # get values by field, values struct example: values[0][:average] # values = {} matrixes_list.length.times do |index| @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suite_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -808,26 +809,40 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suite_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suite_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suite_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suite_list, matrixes_titles, success, theme) return '' if values.empty? + suite_set = Set.new(suite_list) result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + ranked_str = get_ranked_str(values, suite_set, success, theme) + result_str += ranked_str + result_str +end + +def get_ranked_str(values, suite_set, success, theme) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suite_set.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end - result_str + suite_str + common_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..ff4ea31 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,23 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end - groups.delete(group_key) if value.size < 2 + if value.size < 2 + groups.delete(group_key) + next + end + suites_list << suite_list end - groups + + return groups, suites_list end # input: @@ -139,7 +149,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], _ = create_matrix(jobs) end end -- 2.23.0

2 1

[PATCH compass-ci] result-webdav/start: add share dir to upload file
by Cui Yili 23 Nov '20

23 Nov '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index e0b6676..01141c3 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -14,6 +14,7 @@ cmd=( -p 3080:3080 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf -v /srv/result/:/srv/result/ + -v /srv/initrd/:/srv/initrd/ result-webdav ) -- 2.23.0

1 0

[PATCH v9 compass-ci 3/3] lib: fix group compare no output when dimension number less than 2
by Lu Kaiyi 23 Nov '20

23 Nov '20

[why] When do group compare, if returned groups of auto_group like below (the dimension number less than 2 for group_key): groups = { group_key_1 => { dimension_1 => matrix_1 }, group_key_2 => {} } the group_key will be deleted, then, group compare will have no output. [how] Reduce dimension number limit from 2 to 1. [example] lukaiyi@z9 ~% compare "suite=iperf" -d "os" in z9 [before] <null> [after] os_arch=aarch64/os_version=20/pp.iperf.protocol=tcp/pp.iperf.runtime=300/tbox_group=vm-hi1620-2p8g--liangt openeuler metric -------------------- ------------------------------ fails:runs | 18:18 stderr.EOF' 18:18 stderr.has_stderr openeuler metric -------------------- ------------------------------ %stddev \ 3.496630e+10 ± 19% iperf.tcp.receiver.bps 3.497117e+10 ± 19% iperf.tcp.sender.bps 300.23 iperf.time.elapsed_time 300.23 iperf.time.elapsed_time.max ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 2 +- lib/params_group.rb | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2065039..9753a6f 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,7 +116,7 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| - if value.size < 2 + if value.size < 1 groups.delete(group_key) next end diff --git a/lib/params_group.rb b/lib/params_group.rb index 40191a9..48d651b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -34,8 +34,7 @@ def auto_group(jobs_list, dimensions) return [] if dimensions.empty? jobs_list = extract_jobs_list(jobs_list) - groups = group(jobs_list, dimensions) - return remove_singleton(groups) + group(jobs_list, dimensions) end def extract_jobs_list(jobs_list) -- 2.23.0

2 2

[PATCH lkp-tests] create merge_yaml and submit merge_yaml/*.yaml
by Hu Xuejiao 23 Nov '20

23 Nov '20

[why] We will add more spec cases, so it should give each type of cases for different input and output directories Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- spec/submit/merge_yaml/template.yaml | 3 +++ spec/submit/merge_yaml/yaml_merge_included_files.yaml | 1 + spec/submit_spec.rb | 4 ++++ 3 files changed, 8 insertions(+) create mode 100644 spec/submit/merge_yaml/template.yaml create mode 100644 spec/submit/merge_yaml/yaml_merge_included_files.yaml diff --git a/spec/submit/merge_yaml/template.yaml b/spec/submit/merge_yaml/template.yaml new file mode 100644 index 00000000..60e3df16 --- /dev/null +++ b/spec/submit/merge_yaml/template.yaml @@ -0,0 +1,3 @@ +suite: test +category: benchmark +test: diff --git a/spec/submit/merge_yaml/yaml_merge_included_files.yaml b/spec/submit/merge_yaml/yaml_merge_included_files.yaml new file mode 100644 index 00000000..c291508d --- /dev/null +++ b/spec/submit/merge_yaml/yaml_merge_included_files.yaml @@ -0,0 +1 @@ +<<: template.yaml diff --git a/spec/submit_spec.rb b/spec/submit_spec.rb index f365f35b..b56a860c 100644 --- a/spec/submit_spec.rb +++ b/spec/submit_spec.rb @@ -40,4 +40,8 @@ describe 'submit job spec' do it 'link jobs spec' do submit_job('link_jobs') end + + it 'merge yaml spec' do + submit_job('merge_yaml') + end end -- 2.23.0

1 0

[PATCH compass-ci 3/3] doc/job/: add initrd_deps.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing initrd_deps field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/initrd_deps.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/job/initrd_deps.md diff --git a/doc/job/initrd_deps.md b/doc/job/initrd_deps.md new file mode 100644 index 0000000..0c3d79c --- /dev/null +++ b/doc/job/initrd_deps.md @@ -0,0 +1,5 @@ +# initrd_deps + +Meaning: +- Most testcases depend on many other package to run. If user has already packed all the other package for running testcase. Then initrd_deps field would give the location of the package. +- The initrd_deps field do not need to be specified by user in the $testcase yaml. It is filled in by the scheduler. User can get the package through the wget command to check the rpm in compressed package. -- 2.23.0

1 0

[PATCH compass-ci 2/3] doc/job/: add initrd_pkg.md
by Li Ping 23 Nov '20

23 Nov '20

add a document for introducing the initrd_pkg field. Signed-off-by: Li Ping <15396232681(a)163.com> --- doc/job/initrd_pkg.md | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 doc/job/initrd_pkg.md diff --git a/doc/job/initrd_pkg.md b/doc/job/initrd_pkg.md new file mode 100644 index 0000000..2ef3d03 --- /dev/null +++ b/doc/job/initrd_pkg.md @@ -0,0 +1,5 @@ +# initrd_pkg + +Meaning: +- Most testcases rely on many other files such as configuration files to run. If user has already packed all the files for running testcase. Then initrd_pkg field would give the location of the packed compression file. +- The initrd_pkg field do not need to be specified by user in the $testcase yaml. It is filled in by the scheduler. User can get the package through the wget command to check the files in compressed package. -- 2.23.0

1 0

[PATCH lkp-tests 2/2] bin/my-submit: this script is used together with the my-qemu
by Xiao Shenwei 23 Nov '20

23 Nov '20

ignore the queue field, it will add default value which is "$tbox_group~$USER" usage: my-submit testbox=vm-2p8g jobs/borrow-1h.yaml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/my-submit | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 sbin/my-submit diff --git a/sbin/my-submit b/sbin/my-submit new file mode 100755 index 00000000..3a7c741e --- /dev/null +++ b/sbin/my-submit @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +submit --my-queue $@ -- 2.23.0

1 1

[PATCH lkp-tests 1/2] sbin/submit: add job into my queue
by Xiao Shenwei 23 Nov '20

23 Nov '20

add --my-queue parameter the job will be add into your own queue usage: submit --my-queue testbox=vm-2p8g ../jobs/borrow-1h.yaml Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/submit | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/submit b/sbin/submit index 9bba1018..6af86496 100755 --- a/sbin/submit +++ b/sbin/submit @@ -16,6 +16,7 @@ opt_output_dir = nil opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} +opt_my_queue = false actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -54,6 +55,11 @@ options = OptionParser.new do |opts| ARGV.shift end end + + opts.on('--my-queue', 'add to my queue') do + opt_my_queue = true + end + end options.parse!(ARGV) @@ -93,6 +99,11 @@ def find_jobfile(jobfile) exit 1 end +def add_my_queue(job) + queue = "#{job['tbox_group']}~#{ENV['USER']}" + job['queue'] = queue +end + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -134,6 +145,7 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) + add_my_queue(job_hash) if opt_my_queue # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/job.cr: check the account
by Wu Zhende 23 Nov '20

23 Nov '20

When a job submit, check it's account info. Must key:my_email, my_name, my_uuid Check whether the information is the same as that stored in the ES database Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4bef329..05c5462 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -9,6 +9,7 @@ require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" +require "../scheduler/elasticsearch_client" struct JSON::Any def []=(key : String, value : String) @@ -58,8 +59,10 @@ class Job return if @hash["id"] == "#{id}" end + @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") check_required_keys() + check_account_info() set_defaults() end @@ -271,6 +274,9 @@ class Job id suite testbox + my_email + my_uuid + my_name ] private def check_required_keys @@ -281,6 +287,17 @@ class Job end end + private def check_account_info + error_msg = "Failed to verify the account. Please check your configuration" + account_info = @es.get_account(self["my_email"]) + + raise error_msg if account_info["found"]? == false + raise error_msg unless self["my_name"] == account_info["my_name"].to_s + raise error_msg unless self["my_uuid"] == account_info["my_uuid"] + + @hash.delete("my_uuid") + end + private def initialized? initialized_keys = [] of String @@ -302,6 +319,8 @@ class Job "SCHED_HOST", "SCHED_PORT"] + initialized_keys -= ["my_uuid"] + initialized_keys.each do |key| return false unless @hash.has_key?(key) end @@ -441,7 +460,7 @@ class Job else program_version = "latest" end - + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" -- 2.23.0

2 1

[PATCH v3 compass-ci 1/3] git_mirror: fix that README.md in upstream-repos cause a load error
by Li Yuanchao 23 Nov '20

23 Nov '20

[why] As README.md or other docs in upstream-repos, and they are not yaml file. But git-mirror load all files in upstream-repos as yaml file, so there comes out a load error. Now files whose dirname is 'xxx/upstream-repos' will not be loaded. [errmsg] ...in `load_repo_file': undefined method `[]=' for x:Integer (NoMethodError) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9648063..72ca328 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -135,6 +135,8 @@ class MirrorMain traverse_repodir("#{repodir}/#{entry}") end else + return if File.dirname(repodir) == REPO_DIR + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") fork_name = File.basename(repodir) load_repo_file(repodir, project, fork_name) -- 2.23.0

2 1

[PATCH v3 compass-ci 3/3] container/git-mirror: fix that can't get group id of committer in z9
by Li Yuanchao 23 Nov '20

23 Nov '20

In z9, 'committer' is only a group, not a user, so command 'id -g committer' can't get id of committer group Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/start | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/git-mirror/start b/container/git-mirror/start index e1d7d8f..a618ec3 100755 --- a/container/git-mirror/start +++ b/container/git-mirror/start @@ -10,13 +10,13 @@ DOCKER_REPO = '/c/upstream-repos' docker_rm "git-mirror" lkp = %x(id -u lkp) -committer = %x(id -g committer) +group_id = %x(cat /etc/group | grep '^committer' | awk -F ':' '{print $3}') cmd = %W[ docker run --restart=always --name git-mirror - -u #{lkp.to_i}:#{committer.to_i} + -u #{lkp.to_i}:#{group_id.to_i} -d -e CCI_SRC=#{DOCKER_CCI} -e REPO_SRC=#{DOCKER_REPO} -- 2.23.0

2 1

[PATCH lkp-tests 2/2] sbin/my-submit: submit a task to personal queue
by Xiao Shenwei 23 Nov '20

23 Nov '20

usage: my-submit testbox=vm-2p8g jobs/borrow-1h.yaml use this script to submit job, your job will be add to persoanl queue which is $tbox_group~$USER Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/my-submit | 3 +++ 1 file changed, 3 insertions(+) create mode 100755 sbin/my-submit diff --git a/sbin/my-submit b/sbin/my-submit new file mode 100755 index 00000000..57489948 --- /dev/null +++ b/sbin/my-submit @@ -0,0 +1,3 @@ +#!/bin/bash + +submit --my-queue $@ -- 2.23.0

2 2

[PATCH v3 compass-ci 2/3] git_mirror: fix that too few repos in upstream-repos cause memory leak
by Li Yuanchao 21 Nov '20

21 Nov '20

When there are only a few repo files in upstream-repos, that means the same repo will be called very frequently. As the main loop is too fast, it seems the new flow would be called before the last flow of the same repo end completely. That cause a problem. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 72ca328..5b6b889 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,6 +203,7 @@ class MirrorMain push_git_queue handle_feedback Signal.trap(:SIGCHLD, 'SIG_IGN') + sleep(0.1) end end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] scheduler/elasticsearch_client.cr: add function get_account
by Wu Zhende 21 Nov '20

21 Nov '20

Add a new function "get_account" to get account info by "my_email". Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index ec6de0d..6ea8e2d 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -69,6 +69,16 @@ class Elasticsearch::Client return job end + def get_account(my_email : String) + if @client.exists({:index => "accounts", :type => "_doc", :id => my_email}) + response = @client.get_source({:index => "accounts", :type => "_doc", :id => my_email}) + else + response = {"_id" => my_email, "found" => false} + end + + return response + end + private def create(job_content : JSON::Any, job_id : String) return @client.create( { -- 2.23.0

1 0

[PATCH lkp-tests] lkp-tests/*: add archlinux install pp support
by Wang Yong 21 Nov '20

21 Nov '20

archlinux use pacman to install pp Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- bin/lkp-setup-rootfs | 1 + lib/bootstrap.sh | 7 +++++++ 2 files changed, 8 insertions(+) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 98c63bae9..0eabea755 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -261,6 +261,7 @@ while true; do fi install_rpms install_deb + install_pkgs echo $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-lkp $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-lkp $job diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index a7776d687..ce34a6329 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -705,3 +705,10 @@ boot_init() mount_rootfs } + +install_pkgs() +{ + [ -d /opt/pkgs ] || return + echo "install $(ls /opt/pkgs/*.pkg.tar*)" + pacman -U --needed --noconfirm /opt/pkgs/*.pkg.tar* +} -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/iozone: explicit support test=all for iozone
by Lu Kaiyi 21 Nov '20

21 Nov '20

[example] iozone-bs.yaml ... suite: iozone category: benchmark file_size: 4g test: all block_size: - 64k ... [output] ... Run began: Sat Nov 21 16:48:22 2020 Record Size 64 kB File size set to 4194304 kB Command line used: iozone -r 64k -s 4g -f /fs/sdb/iozonetest Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 4194304 64 13650392 11529057 7878373 7763936 7166639 10774159 7520609 16188351 7362482 9751567 9453804 7981606 7932187 iozone test complete. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/iozone b/tests/iozone index b6a1e206..bafc0087 100755 --- a/tests/iozone +++ b/tests/iozone @@ -19,6 +19,8 @@ if [ -n "$block_size" ]; then for ele in ${array[@]} do case $ele in + "all") args+=" -a" + ;; "write") args+=" -i 0" ;; "read") args+=" -i 1" -- 2.23.0

1 0

[PATCH lkp-tests] tests/iozone: explicit support test=all for iozone
by Lu Kaiyi 21 Nov '20

21 Nov '20

[example] iozone-bs.yaml ... suite: iozone category: benchmark file_size: 4g test: all block_size: - 64k ... [output] ... Run began: Sat Nov 21 16:10:59 2020 Record Size 64 kB File size set to 4194304 kB Command line used: iozone -r 64k -s 4g -f /fs/sda/iozonetest Output is in kBytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 kBytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride kB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 4194304 64 13566379 14323483 8305849 8185793 7770078 13206535 4562592 15656107 4448821 4868781 4908219 4736366 4720413 iozone test complete. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/iozone b/tests/iozone index b6a1e206..48688ebb 100755 --- a/tests/iozone +++ b/tests/iozone @@ -19,6 +19,8 @@ if [ -n "$block_size" ]; then for ele in ${array[@]} do case $ele in + "all") args+= " -a" + ;; "write") args+=" -i 0" ;; "read") args+=" -i 1" -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] container/git-mirror: fix that can't get group id of committer in z9
by Li Yuanchao 21 Nov '20

21 Nov '20

[why] In z9, 'committer' is only a group, not a user, so command 'id -g committer' can't get id of committer group Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/git-mirror/start b/container/git-mirror/start index e1d7d8f..f5321a1 100755 --- a/container/git-mirror/start +++ b/container/git-mirror/start @@ -10,7 +10,7 @@ DOCKER_REPO = '/c/upstream-repos' docker_rm "git-mirror" lkp = %x(id -u lkp) -committer = %x(id -g committer) +committer = %x(cat /etc/group | grep '^committer' | awk -F ':' '{print $3}') cmd = %W[ docker run -- 2.23.0

3 4

[PATCH v9 compass-ci 2/3] lib/matrix2.rb: refactor performance optimization
by Lu Kaiyi 21 Nov '20

21 Nov '20

[why] Avoid to execute some unnecessary operations and improve performance. [how] Move the value.size condition judgement to the front of block. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index ff4ea31..2065039 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -116,15 +116,15 @@ def combine_group_query_data(query_data, dims) job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + if value.size < 2 + groups.delete(group_key) + next + end suite_list = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - if value.size < 2 - groups.delete(group_key) - next - end suites_list << suite_list end -- 2.23.0

1 0

[PATCH v6 lkp-tests] add: sysbench-threads-git result parsing file
by Hu Xuejiao 21 Nov '20

21 Nov '20

[input] General statistics: total time: 0.3987s total number of events: 10000 total time taken by event execution: 0.3971 response time: min: 0.04ms [output] total time: 0.3987s Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- stats/sysbench-threads-git | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 stats/sysbench-threads-git diff --git a/stats/sysbench-threads-git b/stats/sysbench-threads-git new file mode 100755 index 00000000..afe5d664 --- /dev/null +++ b/stats/sysbench-threads-git @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby + +# General statistics: +# total time: 0.2671s +# total number of events: 10000 +# total time taken by event execution: 0.3971s +# response time: +# min: 0.04ms + +$stdin.each_line do |line| + next unless line =~ /^\s+total time:\s+([0-9.]+)/ + + puts "total time: #{$1.to_f}s" +end -- 2.23.0

2 1

[PATCH v2 lkp-tests] stats/lmbench3: add new parsing parameters for performance test
by Bai Jing 21 Nov '20

21 Nov '20

[input] Memory read bandwidth 0.000512 16353.73 0.001024 16577.89 0.002048 16752.28 [output] L1_$: 1.539 L2_$: 0.539 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/lmbench3 | 62 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 2 deletions(-) diff --git a/stats/lmbench3 b/stats/lmbench3 index 211c3abc..750e8bfc 100755 --- a/stats/lmbench3 +++ b/stats/lmbench3 @@ -21,6 +21,27 @@ def largest_bandwidth bandwidth end +def designated_file_size + designated_hash = {} + result_str = '' + + $stdin.each_line do |line| + result_str += line + break if line =~ /(^\s|^\t)/ + end + + result_str.split("\n").each do |e| + designated_hash['L1_$'] = e.split[1] if e =~ /0.01758/ + designated_hash['Rand_mem'] = e.split[1] if e =~ /512.0000/ + designated_hash['Mmap_Latency'] = e.split[1] if e =~ /536.870912/ + designated_hash['L2_$'] = e.split[1] if e =~ /0.05078/ + end + + designated_hash +end + +null_io_array = [] + while (line = STDIN.gets) line = line.resolve_invalid_bytes case line @@ -32,14 +53,15 @@ while (line = STDIN.gets) # Simple fstat: 0.3517 microseconds # Simple open/close: 1.3696 microseconds when /^Simple (\S+): (\d+.\d+) microseconds$/ + null_io_array << $2.to_f if $1.to_s == 'read' || $1.to_s == 'write' puts "syscall.#{$1}.latency.us: #{$2}" # Extract select test result. # Select on 100 fd's: 1.2293 microseconds # Select on 100 tcp fd's: 5.0377 microseconds - when /^Select on 100 fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 fd's: (\d+.\d+) microseconds$/ puts "Select.100fd.latency.us: #{$1}" - when /^Select on 100 tcp fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 tcp fd's: (\d+.\d+) microseconds$/ puts "Select.100tcp.latency.us: #{$1}" # Extract proc test result. @@ -198,5 +220,41 @@ while (line = STDIN.gets) # ... when /^"Mmap read open2close bandwidth$/ puts "MMAP.read_open2close.bandwidth.MB/sec: #{largest_bandwidth}" + + # "Random Memory load latency" + # 416.00000 92.824 + # 448.00000 92.772 + # 512.00000 92.640 + # ... + when /^Random load latency$/ + result_rand_mem = designated_file_size + puts "Rand_mem: #{result_rand_mem['Rand_mem']}" + + # "Memory load latency" + # 0.00049 1.539 + # 0.00098 1.539 + # 512.00000 74.421 + # ... + when /^Memory load latency$/ + result_rand_mem = designated_file_size + puts "L1_$: #{result_rand_mem['L1_$']}" + puts "L2_$: #{result_rand_mem['L2_$']}" + + # "mappings" + # 67.108864 82 + # 134.217728 151 + # 268.435456 309 + # 536.870912 615 + # ... + when /^"mappings$/ + result_map = designated_file_size + puts "Mmap_Latency: #{result_map['Mmap_Latency']}" end end + +num = 0 +null_io_array.each do |v| + num += v +end +null_num = format('%.4f', num / 2).to_f +puts "null_io: #{null_num}" -- 2.23.0

3 6

git count
by Wu Fengguang 20 Nov '20

20 Nov '20

Shengde is our code star, thanks Shengde! wfg@crystal ~/todo% bin/git-count 21 Luan Shengde 13 Sun Yukui 10 Li Yuanchao 8 Hu Xuejiao 8 Xiao Shenwei 8 Zhang Yale 8 Yu Chuan 8 Lin Jiaxin 7 Bai Jing 7 Liu Yinsi 6 Liu Shaofei 6 Zhang Yu 5 Cao Xueliang 5 Cui Yili 5 Wang Yong 4 Xu Xijian 4 Wei Jihui 4 Lu Kaiyi 3 Wu Zhende 2 Lu Weitao 2 Wang Chenglong 2 Zhang Yuhang 1 Wu Fengguang 1 Ren Wen 1 Li Ping

1 0

review star
by Wu Fengguang 20 Nov '20

20 Nov '20

Xueliang is our review star, thanks Xueliang! wfg@crystal /c/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 51 44 107 34 Wu Fengguang 118 37 48 24 Cao Xueliang 99 35 50 20 Wei Jihui 29 18 22 14 Xu Xijian 33 6 18 9 Xiao Shenwei 75 23 54 9 Luan Shengde 47 19 21 7 Liu Shaofei 61 14 13 7 Lu Kaiyi 31 13 17 6 Zhang Yu 16 6 11 6 Zhang Yuhang 52 9 11 5 Bai Jing 18 5 10 4 Wang Yong 34 6 11 4 Wang Chenglong 22 14 17 4 Cui Yili 56 7 10 4 Lin Jiaxin 13 3 5 4 Lu Weitao 19 6 8 3 Wu Zhende 61 7 9 3 Hu Xuejiao 47 3 8 2 Yu Chuan 31 7 9 1 Liu Yinsi 8 0 0 0 Li Ping 1 1 1 0 Zhang Dewan 12 2 3 0 Ren Wen 28 9 9 -2 Zhang Yale 23 5 9 -3 Li Yuanchao 41 15 26 -3 Sun Yukui

1 0

help star
by Wu Fengguang 20 Nov '20

20 Nov '20

Yuchuan and Jihui is our help star, thanks Yuchuan and Jihui! wfg@crystal ~/todo% bin/help-star people/*/credit/2020-11-20.md 18.8 Wu Fengguang 17.2 Yu Chuan 17.2 Wei Jihui 16.3 Cao Xueliang 13.7 Xiao Shenwei 12.0 Wu Zhende 11.6 Xu Xijian 10.1 Wang Chenglong 9.8 Liu Shaofei 9.7 Lu Weitao 9.5 Hu Xuejiao 9.0 Luan Shengde 8.5 Cui Yili 8.4 Zhang Yuhang 7.9 Liu Yinsi 7.2 Wang Yong 7.0 Zhang Dewan 6.9 Zhang Yu 6.4 Lin Jiaxin 6.0 Ren Wen 5.8 Li Yuanchao 5.0 Bai Jing 4.3 Du Kaitian 4.2 Zhang Yale 3.3 Lu Kaiyi 2.0 Sun Yukui 1.4 Tong Qunfeng 0.0 Wang Shuai 0.0 Li Ping 0.0 Cui Xucui

1 0

[PATCH compass-ci] doc: add Linux article
by Wu Fengguang 20 Nov '20

20 Nov '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/learning-resources.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/learning-resources.md b/doc/learning-resources.md index 62716cf..e631f4a 100644 --- a/doc/learning-resources.md +++ b/doc/learning-resources.md @@ -55,6 +55,9 @@ https://github.com/DocSpring/ruby_crystal_codemod shell ===== +Linux 的概念与体系 +https://www.cnblogs.com/vamei/archive/2012/10/10/2718229.html + man bash https://devhints.io/bash https://github.com/denysdovhan/bash-handbook/blob/master/translations/zh-CN… -- 2.23.0

1 1

[PATCH v2 compass-ci] scheduler: fix: alter processes of request_state
by Ren Wen 20 Nov '20

20 Nov '20

There're 2 "finished" states before, one is *daemon* request, the other is *test* request. Now change *daemon*'s to "started", means daemon started. This state will save to redis key "sched/cluster_state". Why move "finished" to another "when condition"? When all nodes in cluster do *run_test*, no *start_daemon*, the node will restart immediately after its tests finished. Now need to wait other nodes, so move "finished" to another "when condition" to wait all nodes finished. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/request_cluster_state.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index ac6cb8e..5bbbad1 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -13,13 +13,14 @@ class Sched cluster_state = "" states = {"abort" => "abort", + "started" => "start", "finished" => "finish", "failed" => "abort", "wait_ready" => "ready", "wait_finish" => "finish"} case request_state - when "abort", "finished", "failed" + when "abort", "started", "failed" # update node state only update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" @@ -30,7 +31,7 @@ class Sched } return cluster_state - when "wait_finish" + when "wait_finish", "finished" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: alter processes of request_state
by Ren Wen 20 Nov '20

20 Nov '20

There're 2 "finished" states before, one is *daemon* request, the other is *test* request. Now change *daemon*'s to "started", means daemon started. This state will save to redis key "sched/cluster_state". Why move "finished" to another "when condition"? When all nodes in cluster do *run_test*, no *start_daemon*, the node will restart immediately after its tests finished. Now need to wait other nodes, so move "finished" to another "when condition" to wait all nodes finished. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/request_cluster_state.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index ac6cb8e..fffa4cf 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -13,13 +13,14 @@ class Sched cluster_state = "" states = {"abort" => "abort", + "started" => "started", "finished" => "finish", "failed" => "abort", "wait_ready" => "ready", "wait_finish" => "finish"} case request_state - when "abort", "finished", "failed" + when "abort", "started", "failed" # update node state only update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" @@ -30,7 +31,7 @@ class Sched } return cluster_state - when "wait_finish" + when "wait_finish", "finished" update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) -- 2.23.0

2 3

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: Support more fields in "jobs" index
by Zhang Yuhang 20 Nov '20

20 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..018f57e 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,10 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' -d ' + {"index.mapping.total_fields.limit": 10000} + ' fi fi -- 2.23.0

3 3

[PATCH compass-ci] sbin/es-jobs-mapping.sh: set "index.mapping.total_fields.limit: 10000" when create "jobs" mapping.
by Zhang Yuhang 20 Nov '20

20 Nov '20

[why] "index.mapping.total_fields.limit": The maximum number of fields in an index. The default value is 1000. 1000 is small for "jobs" index. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-jobs-mapping.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 076bdd3..018f57e 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -149,5 +149,10 @@ else if [ $? -ne 0 ] then echo "create jobs index failed." + else + echo "set index.mapping.total_fields.limit: 10000" + curl -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' -d ' + {"index.mapping.total_fields.limit": 10000} + ' fi fi -- 2.23.0

3 4

[PATCH compass-ci] container/ssh-r: fix address already in use
by Liu Yinsi 20 Nov '20

20 Nov '20

[why] when execute ssh-r/start, error message: docker: Error response from daemon: driver failed programming external connectivity on endpoint ssh_r (55638ba16aa1264dff0fdc1badc93176eec07d2b848d4a8d418d07011d84313d): Error starting userland proxy: listen tcp 0.0.0.0:51998: bind: address already in use. [how] because ssh-r will bind ports 50000-51999, set ip_local_port_range to prevent occupation. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 523438c..c327707 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -12,6 +12,8 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 net.ipv6.bindv6only=1 vm.max_map_count=262144 +net.ipv4.ip_local_port_range = 1024 48999 + EOF sysctl -p -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/0-package: fix failed to start ssh-r
by Liu Yinsi 20 Nov '20

20 Nov '20

[why] when execute ssh-r/start, error message: docker: Error response from daemon: driver failed programming external connectivity on endpoint ssh_r (55638ba16aa1264dff0fdc1badc93176eec07d2b848d4a8d418d07011d84313d): Error starting userland proxy: listen tcp 0.0.0.0:51998: bind: address already in use. [how] because ssh-r will bind ports 50000-51999, set ip_local_port_range to prevent occupation. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 523438c..5eb9f76 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -12,6 +12,7 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 net.ipv6.bindv6only=1 vm.max_map_count=262144 +net.ipv4.ip_local_port_range = 1024 48999 EOF sysctl -p -- 2.23.0

1 0

[PATCH v5 compass-ci] lib/log.rb: add new class for json log
by Wu Zhende 20 Nov '20

20 Nov '20

[Why] The built-in log system prints logs in string format by default. The logs should be formatted in json format for subsequent analysis and processing. [Example1] use: log.info("test") output: {"level":"INFO","level_num":1,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"test"} [Example2] use: log.error({"job_id" => "1", "job_state" => "submit"}.to_json) output: {"level":"ERROR","level_num":3,"datetime":"2020-11-19 16:57:17+0800","progname":null,"message":"", "job_id":"1","job_state":"submit","caller":["/usr/share/ruby/logger.rb:545:in `error'","./test.rb:6:in`<main>'"]} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/log.rb | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 lib/log.rb diff --git a/lib/log.rb b/lib/log.rb new file mode 100644 index 0000000..aad00ca --- /dev/null +++ b/lib/log.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'logger' +require 'json' + +# print logs in JSON format +class Log < Logger + LEVEL_INFO = { + 'DEBUG' => 0, + 'INFO' => 1, + 'WARN' => 2, + 'ERROR' => 3, + 'FATAL' => 4, + 'UNKNOWN' => 5 + }.freeze + + FORMATTER = proc { |severity, datetime, progname, msg| + level_num = LEVEL_INFO[severity] + begin + message = JSON.parse(msg) + rescue JSON::ParserError + message = { 'message' => msg } + end + h = { + 'level' => severity.to_s, + 'level_num' => level_num, + 'datetime' => datetime, + 'progname' => progname, + 'message' => '' + } + h.merge!(message) + h.merge!({ 'caller' => caller }) if level_num >= 2 + h.to_json + } + + def initialize(logdev = STDOUT, formatter = FORMATTER) + super(logdev, formatter: formatter) + end +end -- 2.23.0

2 2

[PATCH lkp-tests 1/2] submit: support add job to personal queue
by Xiao Shenwei 20 Nov '20

20 Nov '20

usage: submit --my-queue testbox=vm-2p8g jobs/borrow-1h.yaml if use this parameter, your job will be add to the queue which is $tbox_group~$USER Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sbin/submit | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/sbin/submit b/sbin/submit index 9bba1018..53ea5bf0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -16,6 +16,7 @@ opt_output_dir = nil opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} +personal_job = false actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -54,6 +55,11 @@ options = OptionParser.new do |opts| ARGV.shift end end + + opts.on('--my-queue', 'add to personal queue') do + personal_job = true + end + end options.parse!(ARGV) @@ -93,6 +99,11 @@ def find_jobfile(jobfile) exit 1 end +def add_personal_queue(job) + queue = "#{job['tbox_group']}~#{ENV['USER']}" + job['queue'] = queue +end + ARGV.each do |jobfile| jobfile = find_jobfile(jobfile) jobs = Job2sh.new @@ -134,6 +145,7 @@ ARGV.each do |jobfile| job_hash_list << job_hash job_hash = job_hash.merge(sh_hash) + add_personal_queue(job_hash) if personal_job # save job to yaml if opt_output_dir prefix = File.join(opt_output_dir, File.basename(jobfile, '.yaml')) -- 2.23.0

1 0

[PATCH v3 lkp-tests] stats/lmbench3: add new parsing parameters for performance test
by Bai Jing 20 Nov '20

20 Nov '20

[input] Memory read bandwidth 0.000512 16353.73 0.001024 16577.89 0.002048 16752.28 [output] L1_$: 1.539 L2_$: 0.539 Signed-off-by: Bai Jing <799286817(a)qq.com> --- stats/lmbench3 | 62 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 60 insertions(+), 2 deletions(-) diff --git a/stats/lmbench3 b/stats/lmbench3 index 211c3abc..ba36dbb6 100755 --- a/stats/lmbench3 +++ b/stats/lmbench3 @@ -21,6 +21,27 @@ def largest_bandwidth bandwidth end +def designated_file_size + designated_hash = {} + result_str = '' + + $stdin.each_line do |line| + result_str += line + break if line =~ /(^\s|^\t)/ + end + + result_str.split("\n").each do |e| + designated_hash['L1_$'] = e.split[1] if e =~ /0.01758/ + designated_hash['Rand_mem'] = e.split[1] if e =~ /512.0000/ + designated_hash['Mmap_Latency'] = e.split[1] if e =~ /536.870912/ + designated_hash['L2_$'] = e.split[1] if e =~ /0.05078/ + end + + designated_hash +end + +null_io_array = [] + while (line = STDIN.gets) line = line.resolve_invalid_bytes case line @@ -32,14 +53,15 @@ while (line = STDIN.gets) # Simple fstat: 0.3517 microseconds # Simple open/close: 1.3696 microseconds when /^Simple (\S+): (\d+.\d+) microseconds$/ + null_io_array << $2.to_f if $1.to_s == 'read' || $1.to_s == 'write' puts "syscall.#{$1}.latency.us: #{$2}" # Extract select test result. # Select on 100 fd's: 1.2293 microseconds # Select on 100 tcp fd's: 5.0377 microseconds - when /^Select on 100 fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 fd's: (\d+.\d+) microseconds$/ puts "Select.100fd.latency.us: #{$1}" - when /^Select on 100 tcp fd\'s: (\d+.\d+) microseconds$/ + when /^Select on 100 tcp fd's: (\d+.\d+) microseconds$/ puts "Select.100tcp.latency.us: #{$1}" # Extract proc test result. @@ -198,5 +220,41 @@ while (line = STDIN.gets) # ... when /^"Mmap read open2close bandwidth$/ puts "MMAP.read_open2close.bandwidth.MB/sec: #{largest_bandwidth}" + + # Random load latency + # 416.00000 92.824 + # 448.00000 92.772 + # 512.00000 92.640 + # ... + when /^Random load latency$/ + result_rand_mem = designated_file_size + puts "Rand_mem: #{result_rand_mem['Rand_mem']}" + + # Memory load latency + # 0.00049 1.539 + # 0.00098 1.539 + # 512.00000 74.421 + # ... + when /^Memory load latency$/ + result_rand_mem = designated_file_size + puts "L1_$: #{result_rand_mem['L1_$']}" + puts "L2_$: #{result_rand_mem['L2_$']}" + + # "mappings + # 67.108864 82 + # 134.217728 151 + # 268.435456 309 + # 536.870912 615 + # ... + when /^"mappings$/ + result_map = designated_file_size + puts "Mmap_Latency: #{result_map['Mmap_Latency']}" end end + +num = 0 +null_io_array.each do |v| + num += v +end +null_num = format('%.4f', num / 2).to_f +puts "null_io: #{null_num}" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] git_mirror: fix that too few repos in upstream-repos cause memory leak
by Li Yuanchao 20 Nov '20

20 Nov '20

[why] when there is only a few repo files in upstream-repos, that means the same repo will be called very frequently. As the main loop is too fast, it seems the new flow would be called before the last flow of the same repo end completely. That cause a problem. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 72ca328..5b6b889 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -203,6 +203,7 @@ class MirrorMain push_git_queue handle_feedback Signal.trap(:SIGCHLD, 'SIG_IGN') + sleep(0.1) end end end -- 2.23.0

2 2

[PATCH v2 compass-ci 1/3] git_mirror: fix that README.md in upstream-repos cause a load error
by Li Yuanchao 20 Nov '20

20 Nov '20

[why] As README.md or other docs in upstream-repos, and they are not yaml file. But git-mirror load all files in upstream-repos as yaml file, so there comes out a load error. Now files whose dirname is 'xxx/upstream-repos' will not be loaded. [errmsg] in `load_repo_file': undefined method `[]=' for x:Integer (NoMethodError) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9648063..72ca328 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -135,6 +135,8 @@ class MirrorMain traverse_repodir("#{repodir}/#{entry}") end else + return if File.dirname(repodir) == REPO_DIR + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") fork_name = File.basename(repodir) load_repo_file(repodir, project, fork_name) -- 2.23.0

2 2

[PATCH lkp-tests] daemon/multi-qemu: add a default value for the queues field
by Zhang Yu 20 Nov '20

20 Nov '20

[why] For multi-qemu.yaml, if the queues field is not set, a default value is required. [how] If the multi-qemu.yaml not set queues field, there are two queues by default: "$tbox_group.$HOSTNAME, $tbox_group.$(arch)". Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- daemon/multi-qemu | 1 + 1 file changed, 1 insertion(+) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index 8d9dd38d..7cc625f4 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -6,6 +6,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +: ${queues:="$tbox_group.$HOSTNAME, $tbox_group.$(arch)"} multi_qemu() { export CCI_SRC=/c/compass-ci -- 2.23.0

3 3