January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] lib/monitor: replace IP with domain name
by Wu Fengguang 16 Jan '21

16 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- lib/monitor.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index eca70f196..4785477ec 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -107,7 +107,7 @@ class Monitor def lftp_mirror @result_root.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || '124.160.11.58' + srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' srv_http_port = job['SRV_HTTP_PORT'] || '11300' url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" system "lftp -c mirror #{url} >/dev/null 2>&1" -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] container/assist-cci: upload function
by Ren Wen 15 Jan '21

15 Jan '21

Upload an XML template of libvirt. Response is the modified filename. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/routes.rb | 37 ++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/assist-cci/routes.rb diff --git a/container/assist-cci/routes.rb b/container/assist-cci/routes.rb new file mode 100755 index 0000000..0e71a75 --- /dev/null +++ b/container/assist-cci/routes.rb @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'digest/md5' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_CCI_PORT'] +end + +LIBVIRT_XML_PATH = ENV['LIBVIRT_XML_PATH'] +VERSION = '0.1' + +get '/' do + "version: #{VERSION}" +end + +post '/upload' do + if params[:file] && (tempfile = params[:file][:tempfile]) && (filename = params[:file][:filename]) + dirname = File.basename(filename, '.xml') + content = tempfile.read + md5_filename = Digest::MD5.hexdigest(content) + '.xml' + + dir_path = "#{LIBVIRT_XML_PATH}/#{dirname}" + Dir.exist?(dir_path) || Dir.mkdir(dir_path) + + target = "#{dir_path}/#{md5_filename}" + File.open(target, 'wb') { |f| f.write content } + + File.join(dirname, md5_filename) + else + '' + end +end -- 2.23.0

3 4

[PATCH compass-ci] sbin/auto_submit: fix invalid lab value
by Liu Yinsi 15 Jan '21

15 Jan '21

lab should set by dev env $lab Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index c121ae0..ecc4d70 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -86,7 +86,7 @@ end option = { my_name: 'Auto Submit Robot', my_email: 'autosubmit@localhost', - lab: 'autosubmit' + lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp } # fix verification problems when submit jobs by auto-submit service -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/0-package: add packages
by Liu Yinsi 15 Jan '21

15 Jan '21

[why] 1. providers/qemu/kvm.sh needs wget, curl, qemu command, so add it. 2. use qemu-system-ARCH packages to replace qemue in debian, refer: https://packages.debian.org/sid/qemu. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 2 ++ sparrow/0-package/os/debian | 6 +++++- sparrow/0-package/os/openEuler | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 8df36c0..b25dade 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -22,6 +22,8 @@ pkgs=( busybox iproute hostname + wget + curl ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index c45e8c2..4d13413 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -13,13 +13,17 @@ pkgs=( docker.io ruby-full ruby-dev - qemu dmidecode lockfile-progs bridge-utils busybox iproute2 hostname + wget + curl + qemu-system-aarch64 + qemu-system-x86 + qemu-system-riscv64 ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 37c74d0..04996f5 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -22,6 +22,7 @@ pkgs=( iproute hostname libvirt + curl ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v5 compass-ci] lib/es_jobs.rb: compact es-jobs output results
by Lu Kaiyi 15 Jan '21

15 Jan '21

[why] Reduce the nesting of results and output yaml [example] es-jobs os=archLinux [output] kvcount.arch=aarch64: 43 kvcount.arch=x86_64: 2 kvcount.category=benchmark: 12 kvcount.category=functional: 33 kvcount.job_state=failed: 20 kvcount.job_state=finished: 25 kvcount.os=archLinux: 45 kvcount.suite=cci-depends: 28 kvcount.suite=cci-makepkg: 5 kvcount.suite=iperf: 3 kvcount.suite=iperf-walk-os-test: 9 kvcount.tbox_group=vm-2p16g--wangyong: 1 kvcount.tbox_group=vm-2p16g--wcl: 1 kvcount.tbox_group=vm-2p16g.wangyong: 34 kvcount.tbox_group=vm-2p8g: 9 sum.stats.stderr./lkp/lkp/src/monitors/perf-stat:#:main: 12 sum.stats.stderr.Can_not_find_perf_command: 12 sum.stats.stderr./lkp/lkp/src/tests/wrapper:line#:which:command_not_found: 10 sum.stats.last_state.test.iperf.exit_code.127: 12 ... raw.stats.sched_debug.cfs_rq:/.load.stddev: [524288.0, nil, nil, nil, nil, nil, nil, nil, nil, 524288.0, 524288.0, nil, nil, nil, nil, nil, 516608.0, nil, 1048576.0, nil, nil, nil, nil, nil, nil, nil, 524288.0, nil, nil, nil, nil, 2104832.0, 1572864.0, nil, nil, 2097152.0, nil, nil, nil, nil, nil, nil, nil, nil, nil] ... avg.stats.sched_debug.cfs_rq:/.load.stddev: 1048576.0 avg.stats.softirqs.CPU1.NET_RX: 2.5833333333333335 avg.stats.slabinfo.kmalloc-512.active_objs: 1372.75 ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_jobs.rb | 106 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 99 insertions(+), 7 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index a90b1ba..07533ce 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -2,7 +2,9 @@ # frozen_string_literal: true LKP_SRC = ENV['LKP_SRC'] || '/c/lkp-tests' +KEYWORD = %w[suite os arch category job_state tbox_group upstream_repo submit_id group_id] +require 'yaml' require "#{LKP_SRC}/lib/stats" require_relative './es_query' @@ -123,17 +125,107 @@ class ESJobs return 0 end - def output + def query_jobs_from_es(items) + es = ESQuery.new(ES_HOST, ES_PORT) + result = es.multi_field_query items + jobs = result['hits']['hits'] + jobs.map! { |job| job['_source'] } + return jobs + end + + def get_all_metrics(jobs) + metrics = [] + jobs.each do |job| + stats = job['stats'] + next unless stats + + metrics.concat(stats.keys) + end + metrics.uniq! + end + + def initialize_result_hash(jobs, metrics) result = { - 'stats.count' => @stats['stats.count'] + 'kvcount' => {}, + 'sum.stats' => {}, + 'raw.stats' => {}, + 'avg.stats' => {}, + 'max.stats' => {}, + 'min.stats' => {} } + metrics.each { |metric| result['raw.stats'][metric] = [] } + result + end + + def set_default_value(result, stats, metrics) + left_metrics = metrics - stats.keys + left_metrics.each { |metric| result['raw.stats'][metric] << nil } + + stats.each do |key, value| + result['raw.stats'][key] << value + end + end + + def kvcount(result, job) + KEYWORD.each do |keyword| + next unless job[keyword] + + result['kvcount']["#{keyword}=#{job[keyword]}"] ||= 0 + result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 + end + end + + def stats_count(result) + result['raw.stats'].each do |key, value| + if function_stat?(key) + result['sum.stats'][key] = value.compact.size + else + result['avg.stats'][key] = value.compact.sum / value.compact.size.to_f + result['max.stats'][key] = value.compact.max + result['min.stats'][key] = value.compact.min + end + result['raw.stats'][key] = '---' + value.inspect + '---' + end + end + + def query_jobs_state(jobs) + metrics = get_all_metrics(jobs) + result = initialize_result_hash(jobs, metrics) + jobs.each do |job| + stats = job['stats'] + next unless stats - @stats.each do |key, value| - result[key] = value if @fields.include?(key) + set_default_value(result, stats, metrics) + kvcount(result, job) end - @result['stats_filter_result'] = @stats_filter_result unless @stats_filter.empty? - @result.merge!(result) - puts JSON.pretty_generate(@result) + stats_count(result) + result + end + + def compact_hash(prefix, result) + result.each do |key, value| + if prefix.empty? + prefix_key = "#{key}" + else + prefix_key = "#{prefix}.#{key}" + end + + if value.is_a? Hash + compact_hash(prefix_key, value) + else + @results[prefix_key] = value + end + end + end + + def output + jobs = query_jobs_from_es(@es_query) + @result = query_jobs_state(jobs) + @result['kvcount'] = @result['kvcount'].to_a.sort.to_h + @results = {} + compact_hash('', @result) + yaml = @results.to_yaml.delete_prefix!("---\n") + puts yaml.gsub('"---', '').gsub('---"', '') end end -- 2.23.0

1 0

[PATCH compass-ci 3/3] container - libvirt-helper: upload function
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/libvirt-helper/libvirt_app.rb | 37 +++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/libvirt-helper/libvirt_app.rb diff --git a/container/libvirt-helper/libvirt_app.rb b/container/libvirt-helper/libvirt_app.rb new file mode 100755 index 0000000..551dfc4 --- /dev/null +++ b/container/libvirt-helper/libvirt_app.rb @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'digest/md5' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['LIBVIRT_HELPER_PORT'] +end + +LIBVIRT_PATH = ENV['LIBVIRT_PATH'] +VERSION = '0.1' + +get '/' do + "version: #{VERSION}" +end + +post '/upload' do + if params[:file] && (tempfile = params[:file][:tempfile]) && (filename = params[:file][:filename]) + dirname = File.basename(filename, '.xml') + content = tempfile.read + md5_filename = Digest::MD5.hexdigest(content) + '.xml' + + dir_path = "#{LIBVIRT_PATH}/#{dirname}" + Dir.exist?(dir_path) || Dir.mkdir(dir_path) + + target = "#{dir_path}/#{md5_filename}" + File.open(target, 'wb') { |f| f.write content } + + File.join(dirname, md5_filename) + else + '' + end +end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/3] container/assist-cci: build & start script
by Ren Wen 15 Jan '21

15 Jan '21

Mount '/srv/cci' and upload config to dir: 'libvirt-xml'. Use port 'ASSIST_CCI_PORT' or '11392' by default. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/build | 5 +++++ container/assist-cci/start | 43 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 48 insertions(+) create mode 100755 container/assist-cci/build create mode 100755 container/assist-cci/start diff --git a/container/assist-cci/build b/container/assist-cci/build new file mode 100755 index 0000000..f82de7f --- /dev/null +++ b/container/assist-cci/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-cci . diff --git a/container/assist-cci/start b/container/assist-cci/start new file mode 100755 index 0000000..9ac31c1 --- /dev/null +++ b/container/assist-cci/start @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ASSIST_CCI_HOST + ASSIST_CCI_PORT +] + +defaults = relevant_defaults(names) + +ASSIST_CCI_PORT = defaults["ASSIST_CCI_PORT"] || 11392 +SRV_CCI_PATH = '/srv/cci' +LIBVIRT_XML_PATH = "#{SRV_CCI_PATH}/libvirt-xml" +DEFAULT_CCI = '/c/compass-ci' + +defaults['ASSIST_CCI_PORT'] = ASSIST_CCI_PORT +defaults['LIBVIRT_XML_PATH'] = LIBVIRT_XML_PATH +env = docker_env(defaults) + +docker_rm 'assist-cci' + +cmd = %w[ + docker run + --name assist-cci + --restart=always + -d +] + env + %W[ + -p #{ASSIST_CCI_PORT}:#{ASSIST_CCI_PORT} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v /etc/localtime:/etc/localtime:ro + -v #{SRV_CCI_PATH}:#{SRV_CCI_PATH} + -w #{DEFAULT_CCI}/container/assist-cci/ + assist-cci +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] container/assist-cci: Dockerfile
by Ren Wen 15 Jan '21

15 Jan '21

Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/assist-cci/Dockerfile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 container/assist-cci/Dockerfile diff --git a/container/assist-cci/Dockerfile b/container/assist-cci/Dockerfile new file mode 100644 index 0000000..49007f7 --- /dev/null +++ b/container/assist-cci/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER Ren Wen <15991987063(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install activesupport json yaml sinatra puma -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: add package
by Liu Yinsi 15 Jan '21

15 Jan '21

[why] 1. providers/qemu/kvm.sh needs wget, curl, qemu command, so add it. 2. current debian 'qemu' package makes no sense anymore, and it is very unlikely one will need all of them together, so delete it, install qemu-system-ARCH packages, can see:https://packages.debian.org/sid/qemu Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 2 ++ sparrow/0-package/os/debian | 6 +++++- sparrow/0-package/os/openEuler | 1 + 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 8df36c0..b25dade 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -22,6 +22,8 @@ pkgs=( busybox iproute hostname + wget + curl ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index c45e8c2..4d13413 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -13,13 +13,17 @@ pkgs=( docker.io ruby-full ruby-dev - qemu dmidecode lockfile-progs bridge-utils busybox iproute2 hostname + wget + curl + qemu-system-aarch64 + qemu-system-x86 + qemu-system-riscv64 ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 37c74d0..04996f5 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -22,6 +22,7 @@ pkgs=( iproute hostname libvirt + curl ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

2 2

[PATCH compass-ci 1/2] lib/error_messages.rb: obtain_error_messages_by_error_id function add keyword argument
by Lin Jiaxin 15 Jan '21

15 Jan '21

Determine whether to return the error line or error messages by the value of eturn_error_line Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 3e1564b..81e0034 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -40,12 +40,16 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_error_id(error_id) + def obtain_error_messages_by_error_id(error_id, return_error_line = false) error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| if ('build-pkg.' + build_pkg_error_id(k)) == error_id - error_messages_by_error_id += v.to_a + if return_error_line + error_messages_by_error_id << k + else + error_messages_by_error_id += v.to_a + end end end error_messages_by_error_id -- 2.23.0

2 1