April 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] extract-stats: delete end with ".message" error id
by Cao Xueliang 29 Apr '21

29 Apr '21

we add the .message in error id, the error id can't be the bisect error id, so we delete it when select bisect error id. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/stats_worker.cr | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index f385fee..a244d65 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -137,6 +137,11 @@ class StatsWorker end error_ids.concat(content.as_h.keys) end - error_ids + + error_ids.each do |error_id| + error_ids.delete(error_id) if error_id.ends_with?(".message") + end + + return error_ids end end -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: fix that no need to_json for log
by Li Yuanchao 28 Apr '21

28 Apr '21

When in ruby, log for kibana no need to_json with hash. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 71d6956..a25207a 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -563,7 +563,7 @@ class MirrorMain @log.info({ msg: 'new repo', repo: git_repo - }.to_json) + }) end def git_fail_log(git_repo, possible_new_refs) @@ -571,7 +571,7 @@ class MirrorMain @log.info({ msg: msg, repo: git_repo - }.to_json) + }) end def new_refs_log(git_repo, nr_new_branch) @@ -579,7 +579,7 @@ class MirrorMain msg: 'new refs', repo: git_repo, nr_new_branch: nr_new_branch - }.to_json) + }) end def last_commit_new?(git_repo) -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow/3-code: delete special symbols in vars values
by Liu Yinsi 28 Apr '21

28 Apr '21

[why] special symbols will be parsed in shell command "docker build -t --build-arg $PASSWORD ...", caused failed to build images, so delete. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index fe86b6cf..014221ee 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -190,7 +190,7 @@ auto-submit: EOF [[ -f /etc/compass-ci/passwd.yaml ]] || { -passwd=$(< /dev/urandom tr -dc "a-zA-Z0-9-=+@#_:/[](){}$%&.^*\!" | head -c 10;echo) +passwd=$(< /dev/urandom tr -dc "a-zA-Z0-9" | head -c 10; echo) cat > /etc/compass-ci/passwd.yaml <<EOF ES_USER: root -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sparrow/1-storage: set permission /etc/compass-ci/password.yaml 660
by Liu Yinsi 28 Apr '21

28 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 1 + sparrow/1-storage/tiny | 1 + 2 files changed, 2 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 3c229c5a..3a8bfdaa 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -44,5 +44,6 @@ chgrp team /srv/cci/serial/logs chgrp team /srv/dc chmod g+ws /srv/dc +chmod 660 /etc/compass-ci/password.yaml find /tftpboot -type d | xargs chmod 775 diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index c7636b02..1dbe8a32 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -22,6 +22,7 @@ dirs=( /c /c/git-repos /etc/qemu + /etc/compass-ci/password.yaml /srv/cci/Maildir/.compass-ci/new /srv/cci/Maildir/.compass-ci/cur /srv/cci/Maildir/.compass-ci/sent -- 2.23.0

1 0

[PATCH compass-ci] container: add build depends
by Liu Yinsi 28 Apr '21

28 Apr '21

delimiter's and updaterepo's base image is scheduler-dev. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/delimiter/build-depends | 1 + container/updaterepo/build-depends | 1 + 2 files changed, 2 insertions(+) create mode 100755 container/updaterepo/build-depends diff --git a/container/delimiter/build-depends b/container/delimiter/build-depends index 14bc599b..2ec2b37a 100755 --- a/container/delimiter/build-depends +++ b/container/delimiter/build-depends @@ -1 +1,2 @@ registry +scheduler-dev diff --git a/container/updaterepo/build-depends b/container/updaterepo/build-depends new file mode 100755 index 00000000..5dd403e1 --- /dev/null +++ b/container/updaterepo/build-depends @@ -0,0 +1 @@ +scheduler-dev -- 2.23.0

1 0

[PATCH lkp-tests] lib/bootstrap.sh: remove packages after installation
by Wei Jihui 28 Apr '21

28 Apr '21

[why] /opt/rpms is for save rpm package which the task depends. but if task is cci-depends, it mv $downloaded_rpms to /opt/rpms, then pack /opt/rpms/*.rpm, deps cgz will obtain useless rpm from the task depends. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/bootstrap.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 58f0531a7..417c8662a 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -411,6 +411,7 @@ install_rpms() [ -d /opt/rpms ] || return echo "install $(ls /opt/rpms/*.rpm)" rpm -ivh --force --ignoresize /opt/rpms/*.rpm + rm -f /opt/rpms/*.rpm } try_get_and_set_distro() -- 2.23.0

1 0

[PATCH lkp-tests] add email feedback for rpmbuild-pkg
by Li Ping 28 Apr '21

28 Apr '21

tests/rpmbuild-pkg: add a new input parameter email [usage]: rpmbuild-pkg: email: 1 when job failed, send an rpmbuild failed email. when job run success, send an rpmbuild success email. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/email.sh | 41 +++++++++++++++++++++++++++++++++++++++++ tests/rpmbuild-pkg | 10 ++++++++++ 2 files changed, 51 insertions(+) diff --git a/lib/email.sh b/lib/email.sh index b5e56df79..613d9fd70 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -96,6 +96,47 @@ Compass-CI " } +git_pretty_format() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' -d '{"git_repo": "'${upstream_dir}/${upstream_repo}.git'", "git_command": ["git-log","-n1","--pretty=format:'$1'", "'${upstream_commit}'"]}' +} + +rpmbuild_success_content() +{ + local author_email=$(git_pretty_format "%ae") + local author_name=$(git_pretty_format "%an") + email_content="To: $author_email +Subject: [NOTIFY Compass-CI] rpmbuild report + +Dear $author_name: + + Your RPM Package is successfully built. + + You can click the following link to view details: + http://api.compass-ci.openeuler.org:20012/rpm/pub/${os}-${os_version}/${com… + +Regards +Compass-CI +" +} + +rpmbuild_failed_content() +{ + local author_email=$(git_pretty_format "%ae") + local author_name=$(git_pretty_format "%an") + email_content="To: $author_email +Subject: [NOTIFY Compass-CI] rpmbuild report + +Dear $author_name: + + Sorry to inform you that RPM Package built failed, you can click the following link to view details. + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… + +Regards +Compass-CI +" +} + send_email() { local subject=$1 diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 3dd75b896..8edcb9d1a 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -5,6 +5,7 @@ # - repo_addr # - custom_repo_name # - custom_repo_addr +# - email # Git repo jobs only use the first two parameters, SRPMs jobs use all # user can add custom_repo_name and custom_repo_addr to setup # local repo, value can be an array @@ -12,6 +13,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh . $LKP_SRC/lib/rpm.sh +. $LKP_SRC/lib/email.sh : "${compat_os:=budding-openeuler}" @@ -19,6 +21,13 @@ dest_dir="/rpm/upload/${os}-${os_version}/${compat_os}" +die() +{ + [ -n "$email" ] && send_email "rpmbuild_failed" + echo "$@" 1>&2 + exit 99 +} + from_git() { package_name=${upstream_repo##*/} @@ -143,3 +152,4 @@ add_repo build_rpm upload_rpm_pkg update_repo_mq +[ -n "$email" ] && send_email "rpmbuild_success" -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] kibana/dashboard_import: import kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can import kibana dashboards by json files. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_import | 40 +++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 container/kibana/dashboard_import diff --git a/container/kibana/dashboard_import b/container/kibana/dashboard_import new file mode 100755 index 0000000..e20e8ff --- /dev/null +++ b/container/kibana/dashboard_import @@ -0,0 +1,40 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'pathname' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +files = ARGV +if files.empty? + profile = Pathname.new(File.dirname(__FILE__)).realpath + files << profile + 'data.json' + files << profile + 'resource.json' + files << profile + 'all.json' +end + +cmd = "curl -XPOST -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/import" +cmd += " -H 'Content-Type: application/json' -H 'kbn-xsrf: reporting'" + +files.each do |f| + new_cmd = cmd + new_cmd += " -d @#{f}" + puts new_cmd + system new_cmd +end -- 2.23.0

2 1

[PATCH compass-ci] scheduler-https/start: fix start error
by Wu Zhende 28 Apr '21

28 Apr '21

Missing ENV key: "NODE_PORT" Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 5e79d2b..59e7278 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -69,6 +69,7 @@ cmd = %W[ ] + env + %W[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} + -e NODE_PORT=#{SCHED_HTTPS_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] kibana/dashboard_export: exporting kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can export kibana dashboards by ids. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_export | 33 +++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100755 container/kibana/dashboard_export diff --git a/container/kibana/dashboard_export b/container/kibana/dashboard_export new file mode 100755 index 0000000..1532aae --- /dev/null +++ b/container/kibana/dashboard_export @@ -0,0 +1,33 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +dashboard_ids = ARGV +dashboard_ids = %w[data resource all] if dashboard_ids.empty? + +cmd = "curl -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/export?dashboard=" + +dashboard_ids.each do |id| + new_cmd = cmd + new_cmd += "#{id} >> #{id}.json" + puts new_cmd + system new_cmd +end -- 2.23.0

1 0