- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 2/3] lib/kibana_dashboard.rb: add new class
by Wu Zhende 29 Apr '21

29 Apr '21

Add a public class to operate the kibana dashboard Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/constants.rb | 7 ++++ lib/kibana_dashboard.rb | 72 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+) create mode 100644 lib/kibana_dashboard.rb diff --git a/lib/constants.rb b/lib/constants.rb index c71c3de..ca66718 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'set' require_relative '../container/defconfig.rb' config = cci_defaults @@ -30,6 +31,12 @@ ES_HOSTS = [{ password: ES_PASSWORD }] +KIBANA_HOST ||= config['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT ||= config['KIBANA_PORT'] || '20017' + +LOGGING_KIBANA_HOST ||= config['LOGGING_KIBANA_HOST'] || '172.17.0.1' +LOGGING_KIBANA_PORT ||= config['LOGGING_KIBANA_PORT'] || '20000' + SEND_MAIL_HOST ||= config['SEND_MAIL_HOST'] || '172.17.0.1' SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 10001 diff --git a/lib/kibana_dashboard.rb b/lib/kibana_dashboard.rb new file mode 100644 index 0000000..d8cffea --- /dev/null +++ b/lib/kibana_dashboard.rb @@ -0,0 +1,72 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +CCI_SRC = ENV['CCI_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require_relative 'constants.rb' + +# operat the kibana dashboards +class KibanaDashboard + def initialize(host = KIBANA_HOST, port = KIBANA_PORT, user = ES_USER, password = ES_PASSWORD) + @port = port + @profix = "curl -u #{user}:#{password}" + @profix += " http://#{host}:#{port}/api/kibana/dashboards/" + end + + def get_default_ids(dashboard_ids) + return dashboard_ids unless dashboard_ids.empty? + + return %w[data resource all] if @port == KIBANA_PORT + return %w[data all] if @port == LOGGING_KIBANA_PORT + + dashboard_ids + end + + def get_default_files(files) + return files unless files.empty? + + files = [] + tmp = [] + profix = nil + + if @port == KIBANA_PORT + profix = CCI_SRC + '/container/kibana/' + tmp = %w[data.json resource.json all.json] + elsif @port == LOGGING_KIBANA_PORT + profix = CCI_SRC + '/container/logging-kibana/' + tmp = %w[data.json all.json] + end + + tmp.each do |f| + files << profix + f + end + + files + end + + def export(dashboard_ids) + dashboard_ids = get_default_ids(dashboard_ids) + raise 'empty dashboard ids' if dashboard_ids.empty? + + profix = @profix + 'export?dashboard=' + dashboard_ids.each do |id| + cmd = profix + "#{id} >> #{id}.json" + puts cmd + system cmd + end + end + + def import(files) + files = get_default_files(files) + raise 'empty json files' if files.empty? + + profix = @profix + 'import' + profix += " -H 'Content-Type: application/json' -H 'kbn-xsrf: reporting'" + files.each do |f| + cmd = profix + " -d @#{f}" + puts cmd + system cmd + end + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] scheduler-https/start: fix start error
by Wu Zhende 29 Apr '21

29 Apr '21

Missing ENV key: "NODE_PORT" Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 5e79d2b..59e7278 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -69,6 +69,7 @@ cmd = %W[ ] + env + %W[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} + -e NODE_PORT=#{SCHED_HTTPS_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -- 2.23.0

1 0

[PATCH compass-ci 2/2] dracut-initrd: allow different dir layout for local boot type
by Yu Chuan 29 Apr '21

29 Apr '21

[Why] The current way to get timestamp is strongly dependent on the directory layout, but our os_version is a symbolic link, which corresponds to a source folder structure that is not fixed. But its suffix format is the same, so we get timestamp by intercepting the suffix. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index c6aa8d114d68..0a1125eaace7 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -14,7 +14,11 @@ analyse_kernel_cmdline_params() { # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') - timestamp="$(basename $rootfs_src)" + + # adapt $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-2021-03-10-00-59-59 + timestamp="$(echo ${rootfs_src//-/} | grep -oE '[0-9]{14}$')" + [ -n "$timestamp" ] || reboot_with_msg "cannot find right timestamp" + os="$(echo $rootfs_src | awk -F '/|-' '{print $2}')" os_arch="$(echo $rootfs_src | awk -F '/|-' '{print $3}')" os_version="$(echo $rootfs_src | awk -F '/|-' '{print $4}')" -- 2.23.0

2 1

[PATCH compass-ci 1/2] dracut-initrd: optimize the log
by Yu Chuan 29 Apr '21

29 Apr '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 25 ++++++++----------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 1a2edafaf967..c6aa8d114d68 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -1,5 +1,11 @@ #!/bin/sh +reboot_with_msg() +{ + echo "[compass-ci reboot] $1" + reboot +} + analyse_kernel_cmdline_params() { rootfs="$(getarg root=)" @@ -25,28 +31,22 @@ sync_src_lv() { # need create volume group, usually in first use of this machine. $pv_device e.g. /dev/sda pv_device="$(getarg pv_device=)" [ -n "$pv_device" ] && { - [ -b "$pv_device" ] || { - echo "warn dracut: FATAL: device not found: $pv_device, reboot" - reboot - } + [ -b "$pv_device" ] || reboot_with_msg "warn dracut: FATAL: device not found: $pv_device" # ensure the physical disk has been initialized as physical volume real_pv_device="$(lvm pvs | grep -w $pv_device | awk '{print $1}')" [ "$real_pv_device" = "$pv_device" ] || { - lvm pvcreate -y "$pv_device" || reboot + lvm pvcreate -y "$pv_device" || reboot_with_msg "create pv failed: $pv_device" } # ensure the volume group $vg_name exists real_vg_name="$(lvm pvs | grep -w $vg_name | awk '{print $2}')" [ "$real_vg_name" = "$vg_name" ] || { - lvm vgcreate -y "$vg_name" "$pv_device" || reboot + lvm vgcreate -y "$vg_name" "$pv_device" || reboot_with_msg "create vg failed: $vg_name" } } - lvm vgs "$vg_name" || { - echo "warn dracut: FATAL: vg os not found, reboot" - reboot - } + lvm vgs "$vg_name" || reboot_with_msg "warn dracut: FATAL: vg os not found" # create logical volume src_lv_devname="$(basename $src_lv)" @@ -94,10 +94,7 @@ if [ -z "$use_root_partition" ]; then sync_src_lv "$src_lv" else src_lv="$use_root_partition" - [ -e "$src_lv" ] || { - echo "warn dracut: FATAL: no src_lv with local mount, reboot" - reboot - } + [ -e "$src_lv" ] || reboot_with_msg "warn dracut: FATAL: no src_lv with local mount" fi save_root_partition="$(getarg save_root_partition=)" -- 2.23.0

2 1

[PATCH v2 lkp-tests] add email feedback for rpmbuild-pkg
by Li Ping 29 Apr '21

29 Apr '21

tests/rpmbuild-pkg: add a new input parameter email [usage]: rpmbuild-pkg: whether_email: 1 when job failed, send an rpmbuild failed email. when job run success, send an rpmbuild success email. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/email.sh | 39 +++++++++++++++++++++++++++++++++++++++ tests/rpmbuild-pkg | 15 +++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/lib/email.sh b/lib/email.sh index b5e56df79..e43f390ae 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -96,6 +96,45 @@ Compass-CI " } +rpmbuild_success_content() +{ + local author_email=$(git_pretty_format "%ae") + local author_name=$(git_pretty_format "%an") + email_content="To: $author_email +Subject: [NOTIFY Compass-CI] rpmbuild report + +Dear $author_name: + + Your RPM Package is successfully built. + + You can click the follow link to obtain your RPM Package: + http://api.compass-ci.openeuler.org:20012/rpm/pub/${os}-${os_version}/${com… + + And you can click the following link to view RPM Package build details: + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… + +Regards +Compass-CI +" +} + +rpmbuild_failed_content() +{ + local author_email=$(git_pretty_format "%ae") + local author_name=$(git_pretty_format "%an") + email_content="To: $author_email +Subject: [NOTIFY Compass-CI] rpmbuild report + +Dear $author_name: + + Sorry to inform you that RPM Package built failed, you can click the following link to view details. + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… + +Regards +Compass-CI +" +} + send_email() { local subject=$1 diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 3dd75b896..970418848 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -5,6 +5,7 @@ # - repo_addr # - custom_repo_name # - custom_repo_addr +# - whether_email # Git repo jobs only use the first two parameters, SRPMs jobs use all # user can add custom_repo_name and custom_repo_addr to setup # local repo, value can be an array @@ -12,6 +13,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh . $LKP_SRC/lib/rpm.sh +. $LKP_SRC/lib/email.sh : "${compat_os:=budding-openeuler}" @@ -19,6 +21,13 @@ dest_dir="/rpm/upload/${os}-${os_version}/${compat_os}" +die() +{ + [ -n "$whether_email" ] && send_email "rpmbuild_failed" + echo "$@" 1>&2 + exit 99 +} + from_git() { package_name=${upstream_repo##*/} @@ -29,6 +38,11 @@ from_git() download_upstream_repo } +git_pretty_format() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' -d '{"git_repo": "'${upstream_dir}/${upstream_repo}.git'", "git_command": ["git-log","-n1","--pretty=format:'$1'", "'${upstream_commit}'"]}' +} + from_srpm() { [ -n "$repo_name" ] || die "repo_name is empty" @@ -143,3 +157,4 @@ add_repo build_rpm upload_rpm_pkg update_repo_mq +[ -n "$whether_email" ] && send_email "rpmbuild_success" -- 2.23.0

2 1

[PATCH compass-ci] extract-stats: delete end with ".message" error id
by Cao Xueliang 29 Apr '21

29 Apr '21

we add the .message in error id, the error id can't be the bisect error id, so we delete it when select bisect error id. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/stats_worker.cr | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index f385fee..a244d65 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -137,6 +137,11 @@ class StatsWorker end error_ids.concat(content.as_h.keys) end - error_ids + + error_ids.each do |error_id| + error_ids.delete(error_id) if error_id.ends_with?(".message") + end + + return error_ids end end -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: fix that no need to_json for log
by Li Yuanchao 28 Apr '21

28 Apr '21

When in ruby, log for kibana no need to_json with hash. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 71d6956..a25207a 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -563,7 +563,7 @@ class MirrorMain @log.info({ msg: 'new repo', repo: git_repo - }.to_json) + }) end def git_fail_log(git_repo, possible_new_refs) @@ -571,7 +571,7 @@ class MirrorMain @log.info({ msg: msg, repo: git_repo - }.to_json) + }) end def new_refs_log(git_repo, nr_new_branch) @@ -579,7 +579,7 @@ class MirrorMain msg: 'new refs', repo: git_repo, nr_new_branch: nr_new_branch - }.to_json) + }) end def last_commit_new?(git_repo) -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow/3-code: delete special symbols in vars values
by Liu Yinsi 28 Apr '21

28 Apr '21

[why] special symbols will be parsed in shell command "docker build -t --build-arg $PASSWORD ...", caused failed to build images, so delete. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index fe86b6cf..014221ee 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -190,7 +190,7 @@ auto-submit: EOF [[ -f /etc/compass-ci/passwd.yaml ]] || { -passwd=$(< /dev/urandom tr -dc "a-zA-Z0-9-=+@#_:/[](){}$%&.^*\!" | head -c 10;echo) +passwd=$(< /dev/urandom tr -dc "a-zA-Z0-9" | head -c 10; echo) cat > /etc/compass-ci/passwd.yaml <<EOF ES_USER: root -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sparrow/1-storage: set permission /etc/compass-ci/password.yaml 660
by Liu Yinsi 28 Apr '21

28 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 1 + sparrow/1-storage/tiny | 1 + 2 files changed, 2 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 3c229c5a..3a8bfdaa 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -44,5 +44,6 @@ chgrp team /srv/cci/serial/logs chgrp team /srv/dc chmod g+ws /srv/dc +chmod 660 /etc/compass-ci/password.yaml find /tftpboot -type d | xargs chmod 775 diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index c7636b02..1dbe8a32 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -22,6 +22,7 @@ dirs=( /c /c/git-repos /etc/qemu + /etc/compass-ci/password.yaml /srv/cci/Maildir/.compass-ci/new /srv/cci/Maildir/.compass-ci/cur /srv/cci/Maildir/.compass-ci/sent -- 2.23.0

1 0

[PATCH compass-ci] container: add build depends
by Liu Yinsi 28 Apr '21

28 Apr '21

delimiter's and updaterepo's base image is scheduler-dev. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/delimiter/build-depends | 1 + container/updaterepo/build-depends | 1 + 2 files changed, 2 insertions(+) create mode 100755 container/updaterepo/build-depends diff --git a/container/delimiter/build-depends b/container/delimiter/build-depends index 14bc599b..2ec2b37a 100755 --- a/container/delimiter/build-depends +++ b/container/delimiter/build-depends @@ -1 +1,2 @@ registry +scheduler-dev diff --git a/container/updaterepo/build-depends b/container/updaterepo/build-depends new file mode 100755 index 00000000..5dd403e1 --- /dev/null +++ b/container/updaterepo/build-depends @@ -0,0 +1 @@ +scheduler-dev -- 2.23.0

1 0