- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/4] sbin/compare: cancel fill 0 to matrix
by Lu Weitao 29 Apr '21

29 Apr '21

[Why] matrix => { iperf.xxx => [100, 80, 0], ... } like above, if a stat_metric fill 0, the avg and stddev will be incorrect Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 13 +++++++++---- lib/matrix2.rb | 13 +------------ 2 files changed, 10 insertions(+), 16 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index fef5714a..da2c2b92 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -129,10 +129,15 @@ def get_values_by_field(matrixes_list, field, matrixes_size, success, options) # values = {} matrixes_list.length.times do |index| - value_list = fill_missing_with_zeros( - matrixes_list[index][field], - matrixes_size[index] - ) + value_list = matrixes_list[index][field] + unless value_list + if success + value_list = [0] unless value_list + else + value_list = [1] unless value_list + end + end + values[index] = get_values(value_list, success) next if index.zero? diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2fd63243..82ed9a7a 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -62,16 +62,6 @@ def create_stats(result_root) # stats end -def samples_fill_missing_zeros(value, size) - value.concat([0] * (size - value.size)) -end - -def matrix_fill_miss_zeros(matrix, col_size) - matrix.each_value do |value| - samples_fill_missing_zeros(value, col_size) - end -end - # input: job_list # return: matrix of Hash(String, Array(Number)) # Eg: matrix: { @@ -94,8 +84,7 @@ def create_matrix(job_list) matrix[key] << value end end - col_size = job_list.size - matrix_fill_miss_zeros(matrix, col_size) + return matrix, suites end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: add deps libxml2-devel libxslt-devel
by Xu Xijian 29 Apr '21

29 Apr '21

These 2 deps are for nokogiri, and nokogiri is dep of libvirt. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- sparrow/0-package/os/openEuler | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 5829a12..5153239 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -29,6 +29,8 @@ pkgs=( curl sudo iputils + libxml2-devel + libxslt-devel ) public_network_ok && { -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] sbin/dashboard: add a new script to operate kibana dashboard
by Wu Zhende 29 Apr '21

29 Apr '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/dashboard | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100755 sbin/dashboard diff --git a/sbin/dashboard b/sbin/dashboard new file mode 100755 index 0000000..c15bd75 --- /dev/null +++ b/sbin/dashboard @@ -0,0 +1,67 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'optparse' +require_relative '../lib/constants.rb' +require_relative '../lib/kibana_dashboard.rb' + +opt_type = nil +opt_host = 'kibana' +opt_hash = {} + +options = OptionParser.new do |opts| + opts.banner = 'Usage: dashboard [options] file/id file/id ...' + opts.separator ' importing or exporting kibana dashboards' + opts.separator ' use "dashboard -e" to export kibana dashboards' + opts.separator ' use "dashboard -e -l" to export logging-kibana dashboards' + opts.separator ' use "dashboard -i" to import kibana dashboards' + opts.separator ' use "dashboard -e -h xx.xx.xx.xx -p xx" to assign kibana' + opts.separator '' + opts.separator 'options:' + + opts.on('-e', '--export', 'export kibana dashboards') do + opt_type = 'export' + end + + opts.on('-i', '--import', 'import kibana dashboards') do + opt_type = 'import' + end + + opts.on('-l', '--logging', 'sends requests to logging-kibana, the default value is kibana') do + opt_host = 'logging-kibana' + end + + opts.on('-h host', '--host host', 'set host') do |host| + opt_hash['host'] = host + end + + opts.on('-p port', '--port port', 'set port') do |port| + opt_hash['port'] = port + end +end + +options.parse!(ARGV) + +unless opt_type + puts(options) + exit +end + +parameters = ARGV + +hash = {} +hash['host'] = opt_host == 'kibana' ? KIBANA_HOST : LOGGING_KIBANA_HOST +hash['port'] = opt_host == 'kibana' ? KIBANA_PORT : LOGGING_KIBANA_PORT +hash['user'] = opt_host == 'kibana' ? ES_USER : LOGGING_ES_USER +hash['password'] = opt_host == 'kibana' ? ES_PASSWORD : LOGGING_ES_PASSWORD +hash.merge!(opt_hash) + +kibana_dashboard = KibanaDashboard.new(hash['host'], hash['port'], hash['user'], hash['password']) + +if opt_type == 'export' + kibana_dashboard.export(parameters) +elsif opt_type == 'import' + kibana_dashboard.import(parameters) +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] lib/kibana_dashboard.rb: add new class
by Wu Zhende 29 Apr '21

29 Apr '21

Add a public class to operate the kibana dashboard Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/constants.rb | 7 ++++ lib/kibana_dashboard.rb | 72 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 lib/kibana_dashboard.rb diff --git a/lib/constants.rb b/lib/constants.rb index c71c3de..ca66718 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'set' require_relative '../container/defconfig.rb' config = cci_defaults @@ -30,6 +31,12 @@ ES_HOSTS = [{ password: ES_PASSWORD }] +KIBANA_HOST ||= config['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT ||= config['KIBANA_PORT'] || '20017' + +LOGGING_KIBANA_HOST ||= config['LOGGING_KIBANA_HOST'] || '172.17.0.1' +LOGGING_KIBANA_PORT ||= config['LOGGING_KIBANA_PORT'] || '20000' + SEND_MAIL_HOST ||= config['SEND_MAIL_HOST'] || '172.17.0.1' SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 10001 diff --git a/lib/kibana_dashboard.rb b/lib/kibana_dashboard.rb new file mode 100644 index 0000000..d8cffea --- /dev/null +++ b/lib/kibana_dashboard.rb @@ -0,0 +1,72 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +CCI_SRC = ENV['CCI_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require_relative 'constants.rb' + +# operat the kibana dashboards +class KibanaDashboard + def initialize(host = KIBANA_HOST, port = KIBANA_PORT, user = ES_USER, password = ES_PASSWORD) + @port = port + @profix = "curl -u #{user}:#{password}" + @profix += " http://#{host}:#{port}/api/kibana/dashboards/" + end + + def get_default_ids(dashboard_ids) + return dashboard_ids unless dashboard_ids.empty? + + return %w[data resource all] if @port == KIBANA_PORT + return %w[data all] if @port == LOGGING_KIBANA_PORT + + dashboard_ids + end + + def get_default_files(files) + return files unless files.empty? + + files = [] + tmp = [] + profix = nil + + if @port == KIBANA_PORT + profix = CCI_SRC + '/container/kibana/' + tmp = %w[data.json resource.json all.json] + elsif @port == LOGGING_KIBANA_PORT + profix = CCI_SRC + '/container/logging-kibana/' + tmp = %w[data.json all.json] + end + + tmp.each do |f| + files << profix + f + end + + files + end + + def export(dashboard_ids) + dashboard_ids = get_default_ids(dashboard_ids) + raise 'empty dashboard ids' if dashboard_ids.empty? + + profix = @profix + 'export?dashboard=' + dashboard_ids.each do |id| + cmd = profix + "#{id} >> #{id}.json" + puts cmd + system cmd + end + end + + def import(files) + files = get_default_files(files) + raise 'empty json files' if files.empty? + + profix = @profix + 'import' + profix += " -H 'Content-Type: application/json' -H 'kbn-xsrf: reporting'" + files.each do |f| + cmd = profix + " -d @#{f}" + puts cmd + system cmd + end + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] scheduler-https/start: fix start error
by Wu Zhende 29 Apr '21

29 Apr '21

Missing ENV key: "NODE_PORT" Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 5e79d2b..59e7278 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -69,6 +69,7 @@ cmd = %W[ ] + env + %W[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} + -e NODE_PORT=#{SCHED_HTTPS_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -- 2.23.0

1 0

[PATCH compass-ci 2/2] dracut-initrd: allow different dir layout for local boot type
by Yu Chuan 29 Apr '21

29 Apr '21

[Why] The current way to get timestamp is strongly dependent on the directory layout, but our os_version is a symbolic link, which corresponds to a source folder structure that is not fixed. But its suffix format is the same, so we get timestamp by intercepting the suffix. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index c6aa8d114d68..0a1125eaace7 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -14,7 +14,11 @@ analyse_kernel_cmdline_params() { # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') - timestamp="$(basename $rootfs_src)" + + # adapt $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-2021-03-10-00-59-59 + timestamp="$(echo ${rootfs_src//-/} | grep -oE '[0-9]{14}$')" + [ -n "$timestamp" ] || reboot_with_msg "cannot find right timestamp" + os="$(echo $rootfs_src | awk -F '/|-' '{print $2}')" os_arch="$(echo $rootfs_src | awk -F '/|-' '{print $3}')" os_version="$(echo $rootfs_src | awk -F '/|-' '{print $4}')" -- 2.23.0

2 1

[PATCH compass-ci 1/2] dracut-initrd: optimize the log
by Yu Chuan 29 Apr '21

29 Apr '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 25 ++++++++----------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 1a2edafaf967..c6aa8d114d68 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -1,5 +1,11 @@ #!/bin/sh +reboot_with_msg() +{ + echo "[compass-ci reboot] $1" + reboot +} + analyse_kernel_cmdline_params() { rootfs="$(getarg root=)" @@ -25,28 +31,22 @@ sync_src_lv() { # need create volume group, usually in first use of this machine. $pv_device e.g. /dev/sda pv_device="$(getarg pv_device=)" [ -n "$pv_device" ] && { - [ -b "$pv_device" ] || { - echo "warn dracut: FATAL: device not found: $pv_device, reboot" - reboot - } + [ -b "$pv_device" ] || reboot_with_msg "warn dracut: FATAL: device not found: $pv_device" # ensure the physical disk has been initialized as physical volume real_pv_device="$(lvm pvs | grep -w $pv_device | awk '{print $1}')" [ "$real_pv_device" = "$pv_device" ] || { - lvm pvcreate -y "$pv_device" || reboot + lvm pvcreate -y "$pv_device" || reboot_with_msg "create pv failed: $pv_device" } # ensure the volume group $vg_name exists real_vg_name="$(lvm pvs | grep -w $vg_name | awk '{print $2}')" [ "$real_vg_name" = "$vg_name" ] || { - lvm vgcreate -y "$vg_name" "$pv_device" || reboot + lvm vgcreate -y "$vg_name" "$pv_device" || reboot_with_msg "create vg failed: $vg_name" } } - lvm vgs "$vg_name" || { - echo "warn dracut: FATAL: vg os not found, reboot" - reboot - } + lvm vgs "$vg_name" || reboot_with_msg "warn dracut: FATAL: vg os not found" # create logical volume src_lv_devname="$(basename $src_lv)" @@ -94,10 +94,7 @@ if [ -z "$use_root_partition" ]; then sync_src_lv "$src_lv" else src_lv="$use_root_partition" - [ -e "$src_lv" ] || { - echo "warn dracut: FATAL: no src_lv with local mount, reboot" - reboot - } + [ -e "$src_lv" ] || reboot_with_msg "warn dracut: FATAL: no src_lv with local mount" fi save_root_partition="$(getarg save_root_partition=)" -- 2.23.0

2 1

[PATCH v2 lkp-tests] add email feedback for rpmbuild-pkg
by Li Ping 29 Apr '21

29 Apr '21

tests/rpmbuild-pkg: add a new input parameter email [usage]: rpmbuild-pkg: whether_email: 1 when job failed, send an rpmbuild failed email. when job run success, send an rpmbuild success email. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/email.sh | 39 +++++++++++++++++++++++++++++++++++++++ tests/rpmbuild-pkg | 15 +++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/lib/email.sh b/lib/email.sh index b5e56df79..e43f390ae 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -96,6 +96,45 @@ Compass-CI " } +rpmbuild_success_content() +{ + local author_email=$(git_pretty_format "%ae") + local author_name=$(git_pretty_format "%an") + email_content="To: $author_email +Subject: [NOTIFY Compass-CI] rpmbuild report + +Dear $author_name: + + Your RPM Package is successfully built. + + You can click the follow link to obtain your RPM Package: + http://api.compass-ci.openeuler.org:20012/rpm/pub/${os}-${os_version}/${com… + + And you can click the following link to view RPM Package build details: + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… + +Regards +Compass-CI +" +} + +rpmbuild_failed_content() +{ + local author_email=$(git_pretty_format "%ae") + local author_name=$(git_pretty_format "%an") + email_content="To: $author_email +Subject: [NOTIFY Compass-CI] rpmbuild report + +Dear $author_name: + + Sorry to inform you that RPM Package built failed, you can click the following link to view details. + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… + +Regards +Compass-CI +" +} + send_email() { local subject=$1 diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 3dd75b896..970418848 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -5,6 +5,7 @@ # - repo_addr # - custom_repo_name # - custom_repo_addr +# - whether_email # Git repo jobs only use the first two parameters, SRPMs jobs use all # user can add custom_repo_name and custom_repo_addr to setup # local repo, value can be an array @@ -12,6 +13,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh . $LKP_SRC/lib/rpm.sh +. $LKP_SRC/lib/email.sh : "${compat_os:=budding-openeuler}" @@ -19,6 +21,13 @@ dest_dir="/rpm/upload/${os}-${os_version}/${compat_os}" +die() +{ + [ -n "$whether_email" ] && send_email "rpmbuild_failed" + echo "$@" 1>&2 + exit 99 +} + from_git() { package_name=${upstream_repo##*/} @@ -29,6 +38,11 @@ from_git() download_upstream_repo } +git_pretty_format() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' -d '{"git_repo": "'${upstream_dir}/${upstream_repo}.git'", "git_command": ["git-log","-n1","--pretty=format:'$1'", "'${upstream_commit}'"]}' +} + from_srpm() { [ -n "$repo_name" ] || die "repo_name is empty" @@ -143,3 +157,4 @@ add_repo build_rpm upload_rpm_pkg update_repo_mq +[ -n "$whether_email" ] && send_email "rpmbuild_success" -- 2.23.0

2 1

[PATCH compass-ci] extract-stats: delete end with ".message" error id
by Cao Xueliang 29 Apr '21

29 Apr '21

we add the .message in error id, the error id can't be the bisect error id, so we delete it when select bisect error id. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/stats_worker.cr | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index f385fee..a244d65 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -137,6 +137,11 @@ class StatsWorker end error_ids.concat(content.as_h.keys) end - error_ids + + error_ids.each do |error_id| + error_ids.delete(error_id) if error_id.ends_with?(".message") + end + + return error_ids end end -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: fix that no need to_json for log
by Li Yuanchao 28 Apr '21

28 Apr '21

When in ruby, log for kibana no need to_json with hash. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 71d6956..a25207a 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -563,7 +563,7 @@ class MirrorMain @log.info({ msg: 'new repo', repo: git_repo - }.to_json) + }) end def git_fail_log(git_repo, possible_new_refs) @@ -571,7 +571,7 @@ class MirrorMain @log.info({ msg: msg, repo: git_repo - }.to_json) + }) end def new_refs_log(git_repo, nr_new_branch) @@ -579,7 +579,7 @@ class MirrorMain msg: 'new refs', repo: git_repo, nr_new_branch: nr_new_branch - }.to_json) + }) end def last_commit_new?(git_repo) -- 2.23.0

1 0