- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 1/3] monitoring/parse_serial_logs.cr: add docker patterns
by Wu Zhende 16 Jan '21

16 Jan '21

Add the start and end flags of the Docker startup mode. Use to generate docker's dmsg file. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/parse_serial_logs.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 5a54849..0589523 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -25,6 +25,7 @@ require "json" class SerialParser START_PATTERNS = [ "starting QEMU", + "starting DOCKER", "Start PXE over IPv4", "iPXE initialising devices", "Open Source Network Boot Firmware", @@ -34,6 +35,7 @@ class SerialParser END_PATTERNS = [ "Total QEMU duration: ", + "Total DOCKER duration: ", "No job now", "Restarting system", ] -- 2.23.0

1 0

[PATCH compass-ci] monitoring/parse_serial_logs.cr: add docker patterns
by Wu Zhende 16 Jan '21

16 Jan '21

Add the start and end flags of the Docker startup mode. Use to generate docker's dmsg file. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/parse_serial_logs.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 5a54849..0589523 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -25,6 +25,7 @@ require "json" class SerialParser START_PATTERNS = [ "starting QEMU", + "starting DOCKER", "Start PXE over IPv4", "iPXE initialising devices", "Open Source Network Boot Firmware", @@ -34,6 +35,7 @@ class SerialParser END_PATTERNS = [ "Total QEMU duration: ", + "Total DOCKER duration: ", "No job now", "Restarting system", ] -- 2.23.0

1 0

[PATCH v2 lkp-tests] daemon/sshd: refactor sshd
by Wei Jihui 16 Jan '21

16 Jan '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/sshd | 61 +++++++----------------------------------------- lib/email.sh | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+), 53 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index e9f38006..7ba471a8 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -10,6 +10,7 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/http.sh . $PROJECT_DIR/lib/env.sh +. $PROJECT_DIR/lib/email.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 @@ -36,56 +37,6 @@ run_ssh() fi } -data_success() -{ - data="To: $my_email -Subject: [NOTIFY Compass-CI] Machine application successful - -Dear $my_username, - - Your test machine has been provisioned. - - Login: - ssh root(a)api.compass-ci.openeuler.org -p $port - Due time: - $deadline - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch - -Regards -Compass-CI -" -} - -data_failure() -{ - data="To: $my_email -Subject: [NOTIFY Compass-ci] Machine application failed - -Dear $my_username: - - Sorry to inform you that your application failed. - You may need to wait a moment, or check whether your pub_key exists. - -Regards -Compass-Ci -" -} - -compose_email() -{ - deadline=$(date -d "$runtime seconds" +"%Y-%m-%d %H:%M:%S") - if [ -n "$port" ]; then - data_success - else - data_failure - fi -} - set_port() { [ -n "$my_ssh_pubkey" ] || return @@ -103,9 +54,13 @@ set_port() run_email() { [ -n "$my_email" ] || return 0 - my_username=$(echo "$my_email" | awk -F '@' '{print $1}') - compose_email - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-49000}/send_mail_text" -d "$data" + + deadline=$(date -d "$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + send_email "borrow_success" + else + send_email "borrow_failed" + fi } run_ssh diff --git a/lib/email.sh b/lib/email.sh index d7c2c44a..a0c99c24 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -15,6 +15,71 @@ Compass-CI " } +job_debug_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Test job debug: $id + +Dear $my_name, + + Your test job is ready to debug and test machine has been provisioned. + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-CI +" +} + +borrow_success_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Machine application successful: $id + +Dear $my_name, + + Your test machine has been provisioned. + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-CI +" +} + +borrow_failed_content() +{ + email_content="To: $my_email +Subject: [NOTIFY Compass-CI] Machine application failed: $id + +Dear $my_name: + + Sorry to inform you that your application failed. + You may need to wait a moment, or check whether your pub_key exists. + +Regards +Compass-CI +" +} + send_email() { local subject=$1 -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: replace IP with domain name
by Wu Fengguang 16 Jan '21

16 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- lib/monitor.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index eca70f196..4785477ec 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -107,7 +107,7 @@ class Monitor def lftp_mirror @result_root.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || '124.160.11.58' + srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' srv_http_port = job['SRV_HTTP_PORT'] || '11300' url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" system "lftp -c mirror #{url} >/dev/null 2>&1" -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] container/assist-cci: upload function
by Ren Wen 15 Jan '21

15 Jan '21

Upload an XML template of libvirt. Response is the modified filename. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/routes.rb | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/assist-cci/routes.rb diff --git a/container/assist-cci/routes.rb b/container/assist-cci/routes.rb new file mode 100755 index 0000000..0e71a75 --- /dev/null +++ b/container/assist-cci/routes.rb @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'digest/md5' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_CCI_PORT'] +end + +LIBVIRT_XML_PATH = ENV['LIBVIRT_XML_PATH'] +VERSION = '0.1' + +get '/' do + "version: #{VERSION}" +end + +post '/upload' do + if params[:file] && (tempfile = params[:file][:tempfile]) && (filename = params[:file][:filename]) + dirname = File.basename(filename, '.xml') + content = tempfile.read + md5_filename = Digest::MD5.hexdigest(content) + '.xml' + + dir_path = "#{LIBVIRT_XML_PATH}/#{dirname}" + Dir.exist?(dir_path) || Dir.mkdir(dir_path) + + target = "#{dir_path}/#{md5_filename}" + File.open(target, 'wb') { |f| f.write content } + + File.join(dirname, md5_filename) + else + '' + end +end -- 2.23.0

3 4

[PATCH compass-ci] sbin/auto_submit: fix invalid lab value
by Liu Yinsi 15 Jan '21

15 Jan '21

lab should set by dev env $lab Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index c121ae0..ecc4d70 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,7 +86,7 @@ end option = { my_name: 'Auto Submit Robot', my_email: 'autosubmit@localhost', - lab: 'autosubmit' + lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp } # fix verification problems when submit jobs by auto-submit service -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/0-package: add packages
by Liu Yinsi 15 Jan '21

15 Jan '21

[why] 1. providers/qemu/kvm.sh needs wget, curl, qemu command, so add it. 2. use qemu-system-ARCH packages to replace qemue in debian, refer: https://packages.debian.org/sid/qemu. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 2 ++ sparrow/0-package/os/debian | 6 +++++- sparrow/0-package/os/openEuler | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 8df36c0..b25dade 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -22,6 +22,8 @@ pkgs=( busybox iproute hostname + wget + curl ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index c45e8c2..4d13413 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -13,13 +13,17 @@ pkgs=( docker.io ruby-full ruby-dev - qemu dmidecode lockfile-progs bridge-utils busybox iproute2 hostname + wget + curl + qemu-system-aarch64 + qemu-system-x86 + qemu-system-riscv64 ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 37c74d0..04996f5 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -22,6 +22,7 @@ pkgs=( iproute hostname libvirt + curl ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v5 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archLinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archLinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 sum.stats.last_state.test.iperf.exit_code.127: 12 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0, nil, nil, nil, nil, nil, nil, nil, nil, 524288.0, 524288.0, nil, nil, nil, nil, nil, 516608.0, nil, 1048576.0, nil, nil, nil, nil, nil, nil, nil, 524288.0, nil, nil, nil, nil, 2104832.0, 1572864.0, nil, nil, 2097152.0, nil, nil, nil, nil, nil, nil, nil, nil, nil] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 106 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 99 insertions(+), 7 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..07533ce 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,107 @@ class ESJobs return 0 end - def output + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) result = { - 'stats.count' => @stats['stats.count'] + 'kvcount' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] + + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + end + end + + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = '---' + value.inspect + '---' + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + set_default_value(result, stats, metrics) + kvcount(result, job) end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @result['kvcount'] = @result['kvcount'].to_a.sort.to_h + @results = {} + compact_hash('', @result) + yaml = @results.to_yaml.delete_prefix!("---\n") + puts yaml.gsub('"---', '').gsub('---"', '') end end -- 2.23.0

1 0

[PATCH compass-ci 3/3] container - libvirt-helper: upload function
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/libvirt-helper/libvirt_app.rb | 37 +++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/libvirt-helper/libvirt_app.rb diff --git a/container/libvirt-helper/libvirt_app.rb b/container/libvirt-helper/libvirt_app.rb new file mode 100755 index 0000000..551dfc4 --- /dev/null +++ b/container/libvirt-helper/libvirt_app.rb @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'digest/md5' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['LIBVIRT_HELPER_PORT'] +end + +LIBVIRT_PATH = ENV['LIBVIRT_PATH'] +VERSION = '0.1' + +get '/' do + "version: #{VERSION}" +end + +post '/upload' do + if params[:file] && (tempfile = params[:file][:tempfile]) && (filename = params[:file][:filename]) + dirname = File.basename(filename, '.xml') + content = tempfile.read + md5_filename = Digest::MD5.hexdigest(content) + '.xml' + + dir_path = "#{LIBVIRT_PATH}/#{dirname}" + Dir.exist?(dir_path) || Dir.mkdir(dir_path) + + target = "#{dir_path}/#{md5_filename}" + File.open(target, 'wb') { |f| f.write content } + + File.join(dirname, md5_filename) + else + '' + end +end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/3] container/assist-cci: build & start script
by Ren Wen 15 Jan '21

15 Jan '21

Mount '/srv/cci' and upload config to dir: 'libvirt-xml'. Use port 'ASSIST_CCI_PORT' or '11392' by default. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/build | 5 +++++ container/assist-cci/start | 43 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) create mode 100755 container/assist-cci/build create mode 100755 container/assist-cci/start diff --git a/container/assist-cci/build b/container/assist-cci/build new file mode 100755 index 0000000..f82de7f --- /dev/null +++ b/container/assist-cci/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-cci . diff --git a/container/assist-cci/start b/container/assist-cci/start new file mode 100755 index 0000000..9ac31c1 --- /dev/null +++ b/container/assist-cci/start @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ASSIST_CCI_HOST + ASSIST_CCI_PORT +] + +defaults = relevant_defaults(names) + +ASSIST_CCI_PORT = defaults["ASSIST_CCI_PORT"] || 11392 +SRV_CCI_PATH = '/srv/cci' +LIBVIRT_XML_PATH = "#{SRV_CCI_PATH}/libvirt-xml" +DEFAULT_CCI = '/c/compass-ci' + +defaults['ASSIST_CCI_PORT'] = ASSIST_CCI_PORT +defaults['LIBVIRT_XML_PATH'] = LIBVIRT_XML_PATH +env = docker_env(defaults) + +docker_rm 'assist-cci' + +cmd = %w[ + docker run + --name assist-cci + --restart=always + -d +] + env + %W[ + -p #{ASSIST_CCI_PORT}:#{ASSIST_CCI_PORT} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v /etc/localtime:/etc/localtime:ro + -v #{SRV_CCI_PATH}:#{SRV_CCI_PATH} + -w #{DEFAULT_CCI}/container/assist-cci/ + assist-cci +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 0