April 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 3/4] lib/es_query.rb: support unmatch fields
by Lu Weitao 29 Apr '21

29 Apr '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 8c5a28fe..9d596bb2 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -15,17 +15,20 @@ class ESQuery end # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} + # Example @unmatch_items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) - def multi_field_query(items, size: 10_000, desc_keyword: nil) + def multi_field_query(items, unmatch_items: {}, size: 10_000, desc_keyword: nil) unless items warn 'empty filter!' exit end query_fields = build_multi_field_subquery_body items + unmatch_fields = build_multi_field_subquery_body unmatch_items query = { query: { bool: { - must: query_fields + must: query_fields, + must_not: unmatch_fields } }, size: size } -- 2.23.0

1 0

[PATCH compass-ci 2/4] lib/compare_matrixes.rb cancel remove 0 changed metric
by Lu Weitao 29 Apr '21

29 Apr '21

[Why] some user are still need check 0 changed metric Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index da2c2b92..cff73cbe 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -191,7 +191,7 @@ def remove_unchanged_field(matrixes_values, suite_list) # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| - if suite_list.any? { |suite| field.start_with?(suite) } + if success matrixes_values[success][field][:changed] = true end -- 2.23.0

1 0

[PATCH compass-ci 1/4] sbin/compare: cancel fill 0 to matrix
by Lu Weitao 29 Apr '21

29 Apr '21

[Why] matrix => { iperf.xxx => [100, 80, 0], ... } like above, if a stat_metric fill 0, the avg and stddev will be incorrect Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_matrixes.rb | 13 +++++++++---- lib/matrix2.rb | 13 +------------ 2 files changed, 10 insertions(+), 16 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index fef5714a..da2c2b92 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -129,10 +129,15 @@ def get_values_by_field(matrixes_list, field, matrixes_size, success, options) # values = {} matrixes_list.length.times do |index| - value_list = fill_missing_with_zeros( - matrixes_list[index][field], - matrixes_size[index] - ) + value_list = matrixes_list[index][field] + unless value_list + if success + value_list = [0] unless value_list + else + value_list = [1] unless value_list + end + end + values[index] = get_values(value_list, success) next if index.zero? diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2fd63243..82ed9a7a 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -62,16 +62,6 @@ def create_stats(result_root) # stats end -def samples_fill_missing_zeros(value, size) - value.concat([0] * (size - value.size)) -end - -def matrix_fill_miss_zeros(matrix, col_size) - matrix.each_value do |value| - samples_fill_missing_zeros(value, col_size) - end -end - # input: job_list # return: matrix of Hash(String, Array(Number)) # Eg: matrix: { @@ -94,8 +84,7 @@ def create_matrix(job_list) matrix[key] << value end end - col_size = job_list.size - matrix_fill_miss_zeros(matrix, col_size) + return matrix, suites end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package: add deps libxml2-devel libxslt-devel
by Xu Xijian 29 Apr '21

29 Apr '21

These 2 deps are for nokogiri, and nokogiri is dep of libvirt. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- sparrow/0-package/os/openEuler | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 5829a12..5153239 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -29,6 +29,8 @@ pkgs=( curl sudo iputils + libxml2-devel + libxslt-devel ) public_network_ok && { -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] sbin/dashboard: add a new script to operate kibana dashboard
by Wu Zhende 29 Apr '21

29 Apr '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/dashboard | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100755 sbin/dashboard diff --git a/sbin/dashboard b/sbin/dashboard new file mode 100755 index 0000000..c15bd75 --- /dev/null +++ b/sbin/dashboard @@ -0,0 +1,67 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'optparse' +require_relative '../lib/constants.rb' +require_relative '../lib/kibana_dashboard.rb' + +opt_type = nil +opt_host = 'kibana' +opt_hash = {} + +options = OptionParser.new do |opts| + opts.banner = 'Usage: dashboard [options] file/id file/id ...' + opts.separator ' importing or exporting kibana dashboards' + opts.separator ' use "dashboard -e" to export kibana dashboards' + opts.separator ' use "dashboard -e -l" to export logging-kibana dashboards' + opts.separator ' use "dashboard -i" to import kibana dashboards' + opts.separator ' use "dashboard -e -h xx.xx.xx.xx -p xx" to assign kibana' + opts.separator '' + opts.separator 'options:' + + opts.on('-e', '--export', 'export kibana dashboards') do + opt_type = 'export' + end + + opts.on('-i', '--import', 'import kibana dashboards') do + opt_type = 'import' + end + + opts.on('-l', '--logging', 'sends requests to logging-kibana, the default value is kibana') do + opt_host = 'logging-kibana' + end + + opts.on('-h host', '--host host', 'set host') do |host| + opt_hash['host'] = host + end + + opts.on('-p port', '--port port', 'set port') do |port| + opt_hash['port'] = port + end +end + +options.parse!(ARGV) + +unless opt_type + puts(options) + exit +end + +parameters = ARGV + +hash = {} +hash['host'] = opt_host == 'kibana' ? KIBANA_HOST : LOGGING_KIBANA_HOST +hash['port'] = opt_host == 'kibana' ? KIBANA_PORT : LOGGING_KIBANA_PORT +hash['user'] = opt_host == 'kibana' ? ES_USER : LOGGING_ES_USER +hash['password'] = opt_host == 'kibana' ? ES_PASSWORD : LOGGING_ES_PASSWORD +hash.merge!(opt_hash) + +kibana_dashboard = KibanaDashboard.new(hash['host'], hash['port'], hash['user'], hash['password']) + +if opt_type == 'export' + kibana_dashboard.export(parameters) +elsif opt_type == 'import' + kibana_dashboard.import(parameters) +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] lib/kibana_dashboard.rb: add new class
by Wu Zhende 29 Apr '21

29 Apr '21

Add a public class to operate the kibana dashboard Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/constants.rb | 7 ++++ lib/kibana_dashboard.rb | 72 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 lib/kibana_dashboard.rb diff --git a/lib/constants.rb b/lib/constants.rb index c71c3de..ca66718 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'set' require_relative '../container/defconfig.rb' config = cci_defaults @@ -30,6 +31,12 @@ ES_HOSTS = [{ password: ES_PASSWORD }] +KIBANA_HOST ||= config['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT ||= config['KIBANA_PORT'] || '20017' + +LOGGING_KIBANA_HOST ||= config['LOGGING_KIBANA_HOST'] || '172.17.0.1' +LOGGING_KIBANA_PORT ||= config['LOGGING_KIBANA_PORT'] || '20000' + SEND_MAIL_HOST ||= config['SEND_MAIL_HOST'] || '172.17.0.1' SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 10001 diff --git a/lib/kibana_dashboard.rb b/lib/kibana_dashboard.rb new file mode 100644 index 0000000..d8cffea --- /dev/null +++ b/lib/kibana_dashboard.rb @@ -0,0 +1,72 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +CCI_SRC = ENV['CCI_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require_relative 'constants.rb' + +# operat the kibana dashboards +class KibanaDashboard + def initialize(host = KIBANA_HOST, port = KIBANA_PORT, user = ES_USER, password = ES_PASSWORD) + @port = port + @profix = "curl -u #{user}:#{password}" + @profix += " http://#{host}:#{port}/api/kibana/dashboards/" + end + + def get_default_ids(dashboard_ids) + return dashboard_ids unless dashboard_ids.empty? + + return %w[data resource all] if @port == KIBANA_PORT + return %w[data all] if @port == LOGGING_KIBANA_PORT + + dashboard_ids + end + + def get_default_files(files) + return files unless files.empty? + + files = [] + tmp = [] + profix = nil + + if @port == KIBANA_PORT + profix = CCI_SRC + '/container/kibana/' + tmp = %w[data.json resource.json all.json] + elsif @port == LOGGING_KIBANA_PORT + profix = CCI_SRC + '/container/logging-kibana/' + tmp = %w[data.json all.json] + end + + tmp.each do |f| + files << profix + f + end + + files + end + + def export(dashboard_ids) + dashboard_ids = get_default_ids(dashboard_ids) + raise 'empty dashboard ids' if dashboard_ids.empty? + + profix = @profix + 'export?dashboard=' + dashboard_ids.each do |id| + cmd = profix + "#{id} >> #{id}.json" + puts cmd + system cmd + end + end + + def import(files) + files = get_default_files(files) + raise 'empty json files' if files.empty? + + profix = @profix + 'import' + profix += " -H 'Content-Type: application/json' -H 'kbn-xsrf: reporting'" + files.each do |f| + cmd = profix + " -d @#{f}" + puts cmd + system cmd + end + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] scheduler-https/start: fix start error
by Wu Zhende 29 Apr '21

29 Apr '21

Missing ENV key: "NODE_PORT" Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 5e79d2b..59e7278 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -69,6 +69,7 @@ cmd = %W[ ] + env + %W[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} + -e NODE_PORT=#{SCHED_HTTPS_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -- 2.23.0

1 0

[PATCH compass-ci 2/2] dracut-initrd: allow different dir layout for local boot type
by Yu Chuan 29 Apr '21

29 Apr '21

[Why] The current way to get timestamp is strongly dependent on the directory layout, but our os_version is a symbolic link, which corresponds to a source folder structure that is not fixed. But its suffix format is the same, so we get timestamp by intercepting the suffix. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index c6aa8d114d68..0a1125eaace7 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -14,7 +14,11 @@ analyse_kernel_cmdline_params() { # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') - timestamp="$(basename $rootfs_src)" + + # adapt $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-2021-03-10-00-59-59 + timestamp="$(echo ${rootfs_src//-/} | grep -oE '[0-9]{14}$')" + [ -n "$timestamp" ] || reboot_with_msg "cannot find right timestamp" + os="$(echo $rootfs_src | awk -F '/|-' '{print $2}')" os_arch="$(echo $rootfs_src | awk -F '/|-' '{print $3}')" os_version="$(echo $rootfs_src | awk -F '/|-' '{print $4}')" -- 2.23.0

2 1

[PATCH compass-ci 1/2] dracut-initrd: optimize the log
by Yu Chuan 29 Apr '21

29 Apr '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 25 ++++++++----------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 1a2edafaf967..c6aa8d114d68 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -1,5 +1,11 @@ #!/bin/sh +reboot_with_msg() +{ + echo "[compass-ci reboot] $1" + reboot +} + analyse_kernel_cmdline_params() { rootfs="$(getarg root=)" @@ -25,28 +31,22 @@ sync_src_lv() { # need create volume group, usually in first use of this machine. $pv_device e.g. /dev/sda pv_device="$(getarg pv_device=)" [ -n "$pv_device" ] && { - [ -b "$pv_device" ] || { - echo "warn dracut: FATAL: device not found: $pv_device, reboot" - reboot - } + [ -b "$pv_device" ] || reboot_with_msg "warn dracut: FATAL: device not found: $pv_device" # ensure the physical disk has been initialized as physical volume real_pv_device="$(lvm pvs | grep -w $pv_device | awk '{print $1}')" [ "$real_pv_device" = "$pv_device" ] || { - lvm pvcreate -y "$pv_device" || reboot + lvm pvcreate -y "$pv_device" || reboot_with_msg "create pv failed: $pv_device" } # ensure the volume group $vg_name exists real_vg_name="$(lvm pvs | grep -w $vg_name | awk '{print $2}')" [ "$real_vg_name" = "$vg_name" ] || { - lvm vgcreate -y "$vg_name" "$pv_device" || reboot + lvm vgcreate -y "$vg_name" "$pv_device" || reboot_with_msg "create vg failed: $vg_name" } } - lvm vgs "$vg_name" || { - echo "warn dracut: FATAL: vg os not found, reboot" - reboot - } + lvm vgs "$vg_name" || reboot_with_msg "warn dracut: FATAL: vg os not found" # create logical volume src_lv_devname="$(basename $src_lv)" @@ -94,10 +94,7 @@ if [ -z "$use_root_partition" ]; then sync_src_lv "$src_lv" else src_lv="$use_root_partition" - [ -e "$src_lv" ] || { - echo "warn dracut: FATAL: no src_lv with local mount, reboot" - reboot - } + [ -e "$src_lv" ] || reboot_with_msg "warn dracut: FATAL: no src_lv with local mount" fi save_root_partition="$(getarg save_root_partition=)" -- 2.23.0

2 1

[PATCH v2 lkp-tests] add email feedback for rpmbuild-pkg
by Li Ping 29 Apr '21

29 Apr '21

tests/rpmbuild-pkg: add a new input parameter email [usage]: rpmbuild-pkg: whether_email: 1 when job failed, send an rpmbuild failed email. when job run success, send an rpmbuild success email. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/email.sh | 39 +++++++++++++++++++++++++++++++++++++++ tests/rpmbuild-pkg | 15 +++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/lib/email.sh b/lib/email.sh index b5e56df79..e43f390ae 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -96,6 +96,45 @@ Compass-CI " } +rpmbuild_success_content() +{ + local author_email=$(git_pretty_format "%ae") + local author_name=$(git_pretty_format "%an") + email_content="To: $author_email +Subject: [NOTIFY Compass-CI] rpmbuild report + +Dear $author_name: + + Your RPM Package is successfully built. + + You can click the follow link to obtain your RPM Package: + http://api.compass-ci.openeuler.org:20012/rpm/pub/${os}-${os_version}/${com… + + And you can click the following link to view RPM Package build details: + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… + +Regards +Compass-CI +" +} + +rpmbuild_failed_content() +{ + local author_email=$(git_pretty_format "%ae") + local author_name=$(git_pretty_format "%an") + email_content="To: $author_email +Subject: [NOTIFY Compass-CI] rpmbuild report + +Dear $author_name: + + Sorry to inform you that RPM Package built failed, you can click the following link to view details. + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… + +Regards +Compass-CI +" +} + send_email() { local subject=$1 diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 3dd75b896..970418848 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -5,6 +5,7 @@ # - repo_addr # - custom_repo_name # - custom_repo_addr +# - whether_email # Git repo jobs only use the first two parameters, SRPMs jobs use all # user can add custom_repo_name and custom_repo_addr to setup # local repo, value can be an array @@ -12,6 +13,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh . $LKP_SRC/lib/rpm.sh +. $LKP_SRC/lib/email.sh : "${compat_os:=budding-openeuler}" @@ -19,6 +21,13 @@ dest_dir="/rpm/upload/${os}-${os_version}/${compat_os}" +die() +{ + [ -n "$whether_email" ] && send_email "rpmbuild_failed" + echo "$@" 1>&2 + exit 99 +} + from_git() { package_name=${upstream_repo##*/} @@ -29,6 +38,11 @@ from_git() download_upstream_repo } +git_pretty_format() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' -d '{"git_repo": "'${upstream_dir}/${upstream_repo}.git'", "git_command": ["git-log","-n1","--pretty=format:'$1'", "'${upstream_commit}'"]}' +} + from_srpm() { [ -n "$repo_name" ] || die "repo_name is empty" @@ -143,3 +157,4 @@ add_repo build_rpm upload_rpm_pkg update_repo_mq +[ -n "$whether_email" ] && send_email "rpmbuild_success" -- 2.23.0

2 1