- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] container: expose/report error when check modules exist
by Liu Yinsi 28 Jan '21

28 Jan '21

[why] sparrow/2-network/nfs: modprobe nfs modprobe nfsd sparrow/2-network/cifs: modprobe cifs when locally deploy compass-ci, the above codes have modprobe nfs, nfsd, cifs, no need to do again when execute sparrow/4-docker/buildall(deploy containers), exit and report error message to function callers is better. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 4 +++- container/os-cifs/start | 4 +++- container/os-nfs/start | 7 +++++-- container/result-cifs/start | 4 +++- 4 files changed, 14 insertions(+), 5 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 77558a1..119d1b1 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -3,9 +3,11 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh +. $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + exit 1 } docker_rm initrd-cifs diff --git a/container/os-cifs/start b/container/os-cifs/start index 5e08238..47da260 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -3,9 +3,11 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh +. $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + exit 1 } docker_rm os-cifs diff --git a/container/os-nfs/start b/container/os-nfs/start index 8b8ff5f..8c08db8 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -3,12 +3,15 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh +. $LKP_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { - sudo modprobe nfs + log_error "[ERROR] kernel module nfs is missing (try: sudo modprobe nfs)" + exit 1 } lsmod | grep -q "^nfsd\s" || { - sudo modprobe nfsd + log_error "[ERROR] kernel module nfsd is missing (try: sudo modprobe nfsd)" + exit 1 } # Several nfs servers running will cause conflict, so rm an old before running a new nfs server docker. diff --git a/container/result-cifs/start b/container/result-cifs/start index 2d6acba..4f8c2b9 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -3,9 +3,11 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh +. $LKP_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - sudo modprobe cifs + log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + exit 1 } docker_rm result-cifs -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] replace log.sh
by Liu Yinsi 28 Jan '21

28 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/initrd-cifs/start | 4 ++-- container/os-cifs/start | 4 ++-- container/os-nfs/start | 4 ++-- container/result-cifs/start | 4 ++-- providers/my-qemu.sh | 4 +--- providers/qemu/kvm.sh | 3 +-- providers/qemu/pxe.sh | 4 +--- sparrow/4-docker/buildall | 8 ++++---- 8 files changed, 15 insertions(+), 20 deletions(-) diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start index 119d1b1..6a91b37 100755 --- a/container/initrd-cifs/start +++ b/container/initrd-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-cifs/start b/container/os-cifs/start index 47da260..85017cc 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/container/os-nfs/start b/container/os-nfs/start index 8c08db8..7577647 100755 --- a/container/os-nfs/start +++ b/container/os-nfs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^nfs\s" || { - log_error "[ERROR] kernel module nfs is missing (try: sudo modprobe nfs)" + log_error "kernel module nfs is missing (try: sudo modprobe nfs)" exit 1 } lsmod | grep -q "^nfsd\s" || { diff --git a/container/result-cifs/start b/container/result-cifs/start index 4f8c2b9..7ce066a 100755 --- a/container/result-cifs/start +++ b/container/result-cifs/start @@ -3,10 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. . $CCI_SRC/container/defconfig.sh -. $LKP_SRC/lib/log.sh +. $CCI_SRC/lib/log.sh lsmod | grep -q "^cifs\s" || { - log_error "[ERROR] kernel module cifs is missing (try: sudo modprobe cifs)" + log_error "kernel module cifs is missing (try: sudo modprobe cifs)" exit 1 } diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 113b3ef..3e3b2f4 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,9 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh help() { diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..d394681 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,9 +6,8 @@ : ${nr_cpu:=1} : ${memory:=1G} -: ${LKP_SRC:="/c/lkp-tests"} -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh check_logfile() { diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 0ba052c..3ab3c2b 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -4,9 +4,7 @@ # - nr_cpu # - memory -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh : ${nr_cpu:=1} : ${memory:=1G} diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a60067f..9d6c977 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -2,11 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +source ${CCI_SRC}/lib/log.sh + CONTAINER_PATH="$CCI_SRC/container" export action=$1 -[ -n "$LKP_SRC" ] && source "${LKP_SRC}/lib/log.sh" - build_depends() { local container=$1 @@ -27,7 +27,7 @@ do_one() lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock echo "$started_containers" | grep -w -q "$container_name" && return 0 - [ -n "$LKP_SRC" ] && log_info "start build $container." + log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" @@ -40,7 +40,7 @@ do_one() [ -x first-run ] && ./first-run [ -x start ] && ./start ) - [ -n "$LKP_SRC" ] && log_info "finish build $container." + log_info "finish build $container." started_containers+="$container_name" lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib: add log.sh
by Liu Yinsi 28 Jan '21

28 Jan '21

Records the run logs of the shell program Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/log.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 0000000..83c330c --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +log_info() +{ + date +"%F %T $* [INFO]" +} + +log_debug() +{ + date +"%F %T $* [DEBUG]" +} + +log_warn() +{ + date +"%F %T $* [WARN]" >&2 +} + +log_error() +{ + date +"%F %T $* [ERROR]" >&2 +} -- 2.23.0

1 1

[PATCH compass-ci 2/2] replace log.sh
by Liu Yinsi 28 Jan '21

28 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/my-qemu.sh | 4 +--- providers/qemu/kvm.sh | 3 +-- providers/qemu/pxe.sh | 4 +--- sparrow/4-docker/buildall | 8 ++++---- 4 files changed, 7 insertions(+), 12 deletions(-) diff --git a/providers/my-qemu.sh b/providers/my-qemu.sh index 113b3ef..3e3b2f4 100755 --- a/providers/my-qemu.sh +++ b/providers/my-qemu.sh @@ -2,9 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh help() { diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 66099ee..d394681 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -6,9 +6,8 @@ : ${nr_cpu:=1} : ${memory:=1G} -: ${LKP_SRC:="/c/lkp-tests"} -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh check_logfile() { diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 0ba052c..3ab3c2b 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -4,9 +4,7 @@ # - nr_cpu # - memory -: ${LKP_SRC:="/c/lkp-tests"} - -source ${LKP_SRC}/lib/log.sh +source ${CCI_SRC}/lib/log.sh : ${nr_cpu:=1} : ${memory:=1G} diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a60067f..9d6c977 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -2,11 +2,11 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +source ${CCI_SRC}/lib/log.sh + CONTAINER_PATH="$CCI_SRC/container" export action=$1 -[ -n "$LKP_SRC" ] && source "${LKP_SRC}/lib/log.sh" - build_depends() { local container=$1 @@ -27,7 +27,7 @@ do_one() lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock echo "$started_containers" | grep -w -q "$container_name" && return 0 - [ -n "$LKP_SRC" ] && log_info "start build $container." + log_info "start build $container." mkdir $tmpdir/$container_name 2>/dev/null && ( cd "$container" @@ -40,7 +40,7 @@ do_one() [ -x first-run ] && ./first-run [ -x start ] && ./start ) - [ -n "$LKP_SRC" ] && log_info "finish build $container." + log_info "finish build $container." started_containers+="$container_name" lockfile-remove --lock-name "$container_name".lock } -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: show the ssh connect with red background
by Cao Xueliang 28 Jan '21

28 Jan '21

The ssh connect information is very important, attract users' attention by the red background. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/monitor.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 4785477ec..9091c92a9 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -89,7 +89,7 @@ class Monitor web_socket.close cmd = "ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" - puts cmd + puts "\033[41m#{cmd}\033[0m" cmd = "ssh-keygen -R #{ssh_host};" + cmd exec cmd -- 2.23.0

2 2

[PATCH v2 lkp-tests] jobs: add send email yaml
by Wei Jihui 28 Jan '21

28 Jan '21

[usage] submit job.yaml -i send-email.yaml send_email.subject=$subject submit job.yaml -i send-email-on-fail.yaml on_fail.send_email.subject=$subject values of subject are limited to: job_failed/job_debug/borrow_success/borrow_failed Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/send-email-on-fail.yaml | 7 +++++++ jobs/send-email.yaml | 6 ++++++ 2 files changed, 13 insertions(+) create mode 100644 jobs/send-email-on-fail.yaml create mode 100644 jobs/send-email.yaml diff --git a/jobs/send-email-on-fail.yaml b/jobs/send-email-on-fail.yaml new file mode 100644 index 00000000..a38eb52d --- /dev/null +++ b/jobs/send-email-on-fail.yaml @@ -0,0 +1,7 @@ +# use this send-email-on-fail.yaml by: +# submit job.yaml -i send-email-on-fail.yaml on_fail.send_email.subject=$subject +# values of $subject are limited to job_failed/job_debug/borrow_success/borrow_failed. + +on_fail: + send_email: + subject: job_failed diff --git a/jobs/send-email.yaml b/jobs/send-email.yaml new file mode 100644 index 00000000..8f8b48b9 --- /dev/null +++ b/jobs/send-email.yaml @@ -0,0 +1,6 @@ +# use this send_email.yaml by: +# submit job.yaml -i send-email.yaml send_email.subject=$subject +# values of $subject are limited to job_failed/job_debug/borrow_success/borrow_failed. + +send_email: + subject: job_failed -- 2.23.0

1 0

[PATCH lkp-tests] jobs: add send email yaml
by Wei Jihui 28 Jan '21

28 Jan '21

[usage] submit job.yaml -i send-email.yaml send_email.subject=xx submit job.yaml -i send-email-on-fail.yaml on_fail.send_email.subject=xx subject value limited at: job_failed/job_debug/borrow_success/borrow_failed Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/send-email-on-fail.yaml | 7 +++++++ jobs/send-email.yaml | 6 ++++++ 2 files changed, 13 insertions(+) create mode 100644 jobs/send-email-on-fail.yaml create mode 100644 jobs/send-email.yaml diff --git a/jobs/send-email-on-fail.yaml b/jobs/send-email-on-fail.yaml new file mode 100644 index 00000000..c81f756f --- /dev/null +++ b/jobs/send-email-on-fail.yaml @@ -0,0 +1,7 @@ +# use this job yaml by: +# submit job.yaml -i send-email-on-fail.yaml on_fail.send_email.subject=xx ... +# subject value limited at job_failed/job_debug/borrow_success/borrow_failed. + +on_fail: + send_email: + subject: job_failed diff --git a/jobs/send-email.yaml b/jobs/send-email.yaml new file mode 100644 index 00000000..e6e767ba --- /dev/null +++ b/jobs/send-email.yaml @@ -0,0 +1,6 @@ +# use this job yaml by: +# submit job.yaml -i send-email.yaml send_email.subject=xx ... +# subject value limited at job_failed/job_debug/borrow_success/borrow_failed. + +send_email: + subject: job_failed -- 2.23.0

2 1

[PATCH v2 compass-ci] lib/stats_wrapper.rb support handle $result_root/$script.yaml
by Lu Weitao 28 Jan '21

28 Jan '21

[Example] iperf.yaml: { tcp.sender.bps: 34804801216.197174 tcp.receiver.bps: 34804762215.18231 } iperf.json: { "iperf.tcp.sender.bps": [ 34804801216.197174 ], "iperf.tcp.receiver.bps": [ 34804762215.18231 ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index 5bde4ab..c0406ad 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -24,7 +24,7 @@ module StatsWrapper @stats_group = program_time || program @log = "#{RESULT_ROOT}/#{@stats_group}" - return unless pretreatment + return unless File.exist?("#{(a)log}.yaml") || pretreatment return unless create_tmpfile check_tmpfile @@ -180,6 +180,15 @@ module StatsWrapper system "xzcat #{(a)dmesg_log}.xz > #{@dmesg_log}" if File.exist?("#{(a)dmesg_log}.xz") end + def self.create_yaml_tmpfile(file) + File.readlines(file).each do |line| + next if line =~ /{|}/ + + line.lstrip! + File.write(@tmpfile, line, mode: 'a') + end + end + def self.create_tmpfile tmp = Tempfile.new('lkp-stats.', '/tmp') @tmpfile = tmp.path @@ -191,6 +200,8 @@ module StatsWrapper check #{@tmpfile}") create_status = false end + elsif File.exist?("#{(a)log}.yaml") + create_yaml_tmpfile("#{(a)log}.yaml") else %x(#{PROGRAM_DIR}/#{@program} < /dev/null > #{@tmpfile}) unless $CHILD_STATUS.exitstatus.zero? -- 2.23.0

1 0

[PATCH compass-ci 3/3] lib/stats_wrapper.rb support handle $result_root/$script.yaml
by Lu Weitao 28 Jan '21

28 Jan '21

[Example] iperf.yaml: { tcp.sender.bps: 34804801216.197174 tcp.receiver.bps: 34804762215.18231 } iperf.json: { "iperf.tcp.sender.bps": [ 34804801216.197174 ], "iperf.tcp.receiver.bps": [ 34804762215.18231 ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index 5bde4ab..40647c4 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -24,7 +24,7 @@ module StatsWrapper @stats_group = program_time || program @log = "#{RESULT_ROOT}/#{@stats_group}" - return unless pretreatment + return unless File.exist?("#{(a)log}.yaml") || pretreatment return unless create_tmpfile check_tmpfile @@ -180,6 +180,13 @@ module StatsWrapper system "xzcat #{(a)dmesg_log}.xz > #{@dmesg_log}" if File.exist?("#{(a)dmesg_log}.xz") end + def self.create_yaml_tmpfile(file) + File.readlines(file).each do |line| + next if line =~ %r{({|})} + line.lstrip! + File.write(@tmpfile, line, mode: 'a') + end + end def self.create_tmpfile tmp = Tempfile.new('lkp-stats.', '/tmp') @tmpfile = tmp.path @@ -191,6 +198,8 @@ module StatsWrapper check #{@tmpfile}") create_status = false end + elsif File.exist?("#{(a)log}.yaml") + create_yaml_tmpfile("#{(a)log}.yaml") else %x(#{PROGRAM_DIR}/#{@program} < /dev/null > #{@tmpfile}) unless $CHILD_STATUS.exitstatus.zero? -- 2.23.0

1 1

[PATCH compass-ci 2/3] translate lkp-tests/stats/wrapper to compass-ci/lib/stats_wrapper.rb
by Lu Weitao 28 Jan '21

28 Jan '21

translate lkp-tests/stats/wrapper(shell) to compass-ci/lib/stats_wrapper.rb(ruby) [Why] we need reduce calls between processes to improve performance Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 252 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 252 insertions(+) create mode 100644 lib/stats_wrapper.rb diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb new file mode 100644 index 0000000..5bde4ab --- /dev/null +++ b/lib/stats_wrapper.rb @@ -0,0 +1,252 @@ +# SPDX-License-Identifier: GPL-2.0-only + +# frozen_string_literal: true + +LKP_SRC ||= ENV['LKP_SRC'] || '/c/lkp-tests' +require "#{LKP_SRC}/lib/log.rb" +require 'tempfile' +require 'English' + +PROGRAM_DIR = "#{LKP_SRC}/stats" + +# exit processing the stats if the program is not in the program_list. +# program_list is a file that records all the programs(setups, monitors, tests, +# daemons) being executed during the test. +# this is to solve the problem for cluster test jobs running on the server node +# where no program's log file is generated for running server daemons but the +# stats processing for the program running on client node will also be handled +# on server node due to the current Job2sh algorithm. + +# default stats are not in the program_list +module StatsWrapper + def self.wrapper(program, program_time = nil) + @program = program + @stats_group = program_time || program + @log = "#{RESULT_ROOT}/#{@stats_group}" + + return unless pretreatment + return unless create_tmpfile + + check_tmpfile + warn_empty_stats + return unless create_stat + + check_empty_json + delete_log_package + end + + def self.pretreatment + return unless available_program? + return unless unzip_log + + extract_kmsg_dmesg + check_incomplete_run(@log) + check_soft_timeout + return if check_empty_output + return if check_binary_output + + true + end + + def self.available_program? + yaml_data = File.read("#{LKP_SRC}/etc/default_stats.yaml") + return true if yaml_data =~ /^#{@program}:/ + + return unless File.exist?("#{RESULT_ROOT}/program_list") + + pro_list = File.read("#{RESULT_ROOT}/program_list") + pro_list =~ /#{@program}/ + end + + def self.check_incomplete_run(file) + return if File.size?(file) + return unless File.exist?("#{LKP_SRC}/tests/#{@program}") + + data = "# missing #{@program} #{file}\nis_incomplete_run: 1" + File.write("#{RESULT_ROOT}/last_state", data, mode: 'a') + end + + def self.check_soft_timeout + return unless File.exist?("#{RESULT_ROOT}/soft_timeout") + + last_state = File.readlines("#{RESULT_ROOT}/last_state") + last_state.map!(&:chomp!) + data = 'soft_timeout: 1' + return if last_state.indlude?(data) + + File.write("#{RESULT_ROOT}/last_state", data, mode: 'a') + end + + def self.delete_log + File.delete(@log) if File.exist?(@log) + File.delete("#{(a)log}.gz") if File.exist?("#{(a)log}.gz") + end + + def self.check_empty_output + return if File.size?(@log) + return if %w[tcrypt kernel-size perf-profile].include?(@program) + return if @program == 'dmesg' && File.size?("#{RESULT_ROOT}/kmsg") + return if @program == 'kmsg' && File.size?("#{RESULT_ROOT}/dmesg") + + delete_log + true + end + + def self.check_binary_output + return if %w[dmesg kmsg mpstat iostat].include?(@program) + + # kmsg may actually read the dmesg file + # refer to the exception cases in check_empty_output() + return unless File.exist?(@log) + return unless File.read(@log) =~ /\x0/ + + log_warn "skip binary file #{@log}" + + true + end + + def self.warn_empty_stats + return if File.size?(@tmpfile) + return if %w[dmesg ftrace turbostat perf-profile].include?(@program) + return unless File.size?("#{RESULT_ROOT}/time") + return if File.size?("#{RESULT_ROOT}/last_state") + return if File.read("#{LKP_SRC}/etc/failure") =~ /^#{(a)program}\./ + return if File.read("#{LKP_SRC}/etc/pass") =~ /^#{(a)program}\./ + + log_warn "empty stats for #{@log}" + end + + def self.check_exist_json + return if File.exist?("#{RESULT_ROOT}/#{(a)stats_group}.json") || File.exist?("#{RESULT_ROOT}/#{(a)stats_group}.json.gz") + return if File.exist?("#{RESULT_ROOT}/last_stat") && File.read("#{RESULT_ROOT}/last_state") =~ /is_incomplete_run/ + + log_warn "no generate json file for #{@stats_group}, check #{RESULT_ROOT}" + data = "# no json file for #{@stats_group}\nis_incomplete_run: 1" + File.write("#{RESULT_ROOT}/last_state", data, mode: 'a') + end + + def self.check_empty_json + testcase = RESULT_ROOT.split('/')[2] + testcase[4, testcase.length] if testcase =~ /^kvm:/ + + return if %w[borrow boot].include?(testcase) + return unless File.exist?("#{RESULT_ROOT}/job.sh") + + # testcase maybe different with run_case + # take fio-basic-1hdd-write.yaml job as example: + # - testcase will be fio-basc + # - run_case will be fio + + run_case = nil + job_data = File.readlines("#{RESULT_ROOT}/job.sh") + job_data.each do |line| + next unless line.chomp! =~ %r{LKP_SRC/tests/wrapper} + + parse = line.split(' ')[-1] + run_case = parse + break + end + + return unless run_case == @stats_group + + check_exist_json + end + + def self.unzip_log + if File.exist?("#{(a)log}.gz") + return unless File.size?("#{(a)log}.gz") + + system "zcat #{(a)log}.gz > #{@log}" + elsif File.exist?("#{(a)log}.xz") + return unless File.size?("#{(a)log}.xz") + + system "xzcat #{(a)log}.xz > #{@log}" + end + + true + end + + def self.extract_kmsg_dmesg + @kmsg_log = "#{RESULT_ROOT}/kmsg" + # extract kmsg for kmsg related stats + if @log =~ %r{^#{RESULT_ROOT}/(boot-memory|boot-time|tcrypt|dmesg)$} + system "xzcat #{(a)kmsg_log}.xz" if File.exist?("#{(a)kmsg_log}.xz") + end + + @dmesg_log = "#{RESULT_ROOT}/dmesg" + # extract dmesg for dmesg related stats + return unless @log =~ %r{^#{RESULT_ROOT}/kmsg$} + + system "xzcat #{(a)dmesg_log}.xz > #{@dmesg_log}" if File.exist?("#{(a)dmesg_log}.xz") + end + + def self.create_tmpfile + tmp = Tempfile.new('lkp-stats.', '/tmp') + @tmpfile = tmp.path + create_status = true + if File.exist?(@log) + %x(#{PROGRAM_DIR}/#{@program} #{@log} < #{@log} > #{@tmpfile}) + unless $CHILD_STATUS.exitstatus.zero? + log_error("#{PROGRAM_DIR}/#{@program} #{@log} < #{@log} exit code #{$CHILD_STATUS.exitstatus}, + check #{@tmpfile}") + create_status = false + end + else + %x(#{PROGRAM_DIR}/#{@program} < /dev/null > #{@tmpfile}) + unless $CHILD_STATUS.exitstatus.zero? + log_error("#{PROGRAM_DIR}/#{@program} < /dev/null exit code #{$CHILD_STATUS.exitstatus}, check #{@tmpfile}") + create_status = false + end + end + + create_status + end + + def self.check_tmpfile + str = File.read("#{LKP_SRC}/etc/failure") + str += File.read("#{LKP_SRC}/etc/pass") + + str =~ /^#{(a)program}\./ || check_incomplete_run(@tmpfile) + end + + def self.dump_stat(stats_group, file) + %x(#{LKP_SRC}/sbin/dump-stat #{stats_group} < #{file}) + unless $CHILD_STATUS.exitstatus.zero? + log_error("#{LKP_SRC}/sbin/dump-stat #{@program} exit code #{$CHILD_STATUS.exitstatus}, check #{file}") + return nil + end + + File.delete(file) + + true + end + + def self.create_stat + if @program == 'ftrace' + Dir.foreach(RESULT_ROOT) do |file| + if file =~ /^#{@program}\..*\.yaml/ + stats_group = File.basename(file, '.yaml') + dump_stat(stats_group, file) + end + end + else + dump_stat(@stats_group, @tmpfile) + end + end + + def self.delete_log_package + if File.exist?("#{(a)log}.gz") + File.delete(@log) + elsif File.exist?("#{(a)log}.xz") + File.delete(@log) + end + + # delete temporarily extracted kmsg above + File.delete(@kmsg_log) if File.exist?("#{(a)kmsg_log}.xz") + + # delete temporarily extracted dmesg above + File.delete(@dmesg_log) if File.exist?("#{(a)dmesg_log}.xz") + + File.delete(@tmpfile) if File.exist?(@tmpfile) + end +end -- 2.23.0

1 0