- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] lib/bootstrap.sh: remove packages after installation
by Wei Jihui 28 Apr '21

28 Apr '21

[why] /opt/rpms is for save rpm package which the task depends. but if task is cci-depends, it mv $downloaded_rpms to /opt/rpms, then pack /opt/rpms/*.rpm, deps cgz will obtain useless rpm from the task depends. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/bootstrap.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 58f0531a7..417c8662a 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -411,6 +411,7 @@ install_rpms() [ -d /opt/rpms ] || return echo "install $(ls /opt/rpms/*.rpm)" rpm -ivh --force --ignoresize /opt/rpms/*.rpm + rm -f /opt/rpms/*.rpm } try_get_and_set_distro() -- 2.23.0

1 0

[PATCH lkp-tests] add email feedback for rpmbuild-pkg
by Li Ping 28 Apr '21

28 Apr '21

tests/rpmbuild-pkg: add a new input parameter email [usage]: rpmbuild-pkg: email: 1 when job failed, send an rpmbuild failed email. when job run success, send an rpmbuild success email. Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/email.sh | 41 +++++++++++++++++++++++++++++++++++++++++ tests/rpmbuild-pkg | 10 ++++++++++ 2 files changed, 51 insertions(+) diff --git a/lib/email.sh b/lib/email.sh index b5e56df79..613d9fd70 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -96,6 +96,47 @@ Compass-CI " } +git_pretty_format() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' -d '{"git_repo": "'${upstream_dir}/${upstream_repo}.git'", "git_command": ["git-log","-n1","--pretty=format:'$1'", "'${upstream_commit}'"]}' +} + +rpmbuild_success_content() +{ + local author_email=$(git_pretty_format "%ae") + local author_name=$(git_pretty_format "%an") + email_content="To: $author_email +Subject: [NOTIFY Compass-CI] rpmbuild report + +Dear $author_name: + + Your RPM Package is successfully built. + + You can click the following link to view details: + http://api.compass-ci.openeuler.org:20012/rpm/pub/${os}-${os_version}/${com… + +Regards +Compass-CI +" +} + +rpmbuild_failed_content() +{ + local author_email=$(git_pretty_format "%ae") + local author_name=$(git_pretty_format "%an") + email_content="To: $author_email +Subject: [NOTIFY Compass-CI] rpmbuild report + +Dear $author_name: + + Sorry to inform you that RPM Package built failed, you can click the following link to view details. + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… + +Regards +Compass-CI +" +} + send_email() { local subject=$1 diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 3dd75b896..8edcb9d1a 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -5,6 +5,7 @@ # - repo_addr # - custom_repo_name # - custom_repo_addr +# - email # Git repo jobs only use the first two parameters, SRPMs jobs use all # user can add custom_repo_name and custom_repo_addr to setup # local repo, value can be an array @@ -12,6 +13,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh . $LKP_SRC/lib/rpm.sh +. $LKP_SRC/lib/email.sh : "${compat_os:=budding-openeuler}" @@ -19,6 +21,13 @@ dest_dir="/rpm/upload/${os}-${os_version}/${compat_os}" +die() +{ + [ -n "$email" ] && send_email "rpmbuild_failed" + echo "$@" 1>&2 + exit 99 +} + from_git() { package_name=${upstream_repo##*/} @@ -143,3 +152,4 @@ add_repo build_rpm upload_rpm_pkg update_repo_mq +[ -n "$email" ] && send_email "rpmbuild_success" -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] kibana/dashboard_import: import kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can import kibana dashboards by json files. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_import | 40 +++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 container/kibana/dashboard_import diff --git a/container/kibana/dashboard_import b/container/kibana/dashboard_import new file mode 100755 index 0000000..e20e8ff --- /dev/null +++ b/container/kibana/dashboard_import @@ -0,0 +1,40 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'pathname' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +files = ARGV +if files.empty? + profile = Pathname.new(File.dirname(__FILE__)).realpath + files << profile + 'data.json' + files << profile + 'resource.json' + files << profile + 'all.json' +end + +cmd = "curl -XPOST -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/import" +cmd += " -H 'Content-Type: application/json' -H 'kbn-xsrf: reporting'" + +files.each do |f| + new_cmd = cmd + new_cmd += " -d @#{f}" + puts new_cmd + system new_cmd +end -- 2.23.0

2 1

[PATCH compass-ci] scheduler-https/start: fix start error
by Wu Zhende 28 Apr '21

28 Apr '21

Missing ENV key: "NODE_PORT" Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 5e79d2b..59e7278 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -69,6 +69,7 @@ cmd = %W[ ] + env + %W[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} + -e NODE_PORT=#{SCHED_HTTPS_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v #{CCI_REPOS}/lab-#{LAB}:#{CCI_REPOS}/lab-#{LAB} -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] kibana/dashboard_export: exporting kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can export kibana dashboards by ids. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_export | 33 +++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100755 container/kibana/dashboard_export diff --git a/container/kibana/dashboard_export b/container/kibana/dashboard_export new file mode 100755 index 0000000..1532aae --- /dev/null +++ b/container/kibana/dashboard_export @@ -0,0 +1,33 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +dashboard_ids = ARGV +dashboard_ids = %w[data resource all] if dashboard_ids.empty? + +cmd = "curl -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/export?dashboard=" + +dashboard_ids.each do |id| + new_cmd = cmd + new_cmd += "#{id} >> #{id}.json" + puts new_cmd + system new_cmd +end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] kibana/dashboard_import: import kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can import kibana dashboards by json files. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_import | 44 +++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100755 container/kibana/dashboard_import diff --git a/container/kibana/dashboard_import b/container/kibana/dashboard_import new file mode 100755 index 0000000..a6129a5 --- /dev/null +++ b/container/kibana/dashboard_import @@ -0,0 +1,44 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'pathname' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +files = [] +ARGV.each do |f| + files >> f +end + +if files.empty? + profile = Pathname.new(File.dirname(__FILE__)).realpath + files << profile + 'data.json' + files << profile + 'resource.json' + files << profile + 'all.json' +end + +cmd = "curl -XPOST -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/import" +cmd += " -H 'Content-Type: application/json' -H 'kbn-xsrf: reporting'" + +files.each do |f| + new_cmd = cmd + new_cmd += " -d @#{f}" + puts new_cmd + system new_cmd +end -- 2.23.0

2 2

[PATCH v2 compass-ci 1/2] kibana/dashboard_export: exporting kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can export kibana dashboards by ids. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_export | 37 +++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/kibana/dashboard_export diff --git a/container/kibana/dashboard_export b/container/kibana/dashboard_export new file mode 100755 index 0000000..d6e0054 --- /dev/null +++ b/container/kibana/dashboard_export @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +dashboard_ids = [] +ARGV.each do |i| + dashboard_ids << i +end + +dashboard_ids = %w[data resource all] if dashboard_ids.empty? + +cmd = "curl -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/export?dashboard=" + +dashboard_ids.each do |id| + new_cmd = cmd + new_cmd += "#{id} >> #{id}.json" + puts new_cmd + system new_cmd +end -- 2.23.0

2 1

[PATCH lkp-tests] tests/rpmbuild-pkg: change the way obtaining spec file
by Li Ping 28 Apr '21

28 Apr '21

[why] tests/rpmbuild-pkg: package_name=${upstream_repo##*/} when the package name is inconsistent with the repo name, it will cause the error like: + grep http -rl /root/rpmbuild/SPECS/zstd-test.spec grep: /root/rpmbuild/SPECS/zstd-test.spec: No such file or directory + spectool -g -R /root/rpmbuild/SPECS/zstd-test.spec Can't open '/root/rpmbuild/SPECS/zstd-test.spec': No such file or directory at /usr/bin/spectool line 314. Signed-off-by: Li Ping <1477412247(a)qq.com> --- tests/rpmbuild-pkg | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 5c694b9d4..3dd75b896 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -49,7 +49,7 @@ init_workspace() download_upstream_repo() { - local url="git://${GIT_SERVER}/openeuler/${upstream_repo}" + local url="git://${GIT_SERVER}/${upstream_dir}/${upstream_repo}" git clone "$url" || die "clone git repo ${package_name} failed: $url" cd "$package_name" || exit @@ -75,14 +75,14 @@ build_rpm() [ -n "$package_name" ] && { # HTTP is proxy cache friendly - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/${package_name}.spec")" + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/"*.spec)" } # Install build depends yum-builddep -y "$spec_dir"/*.spec || die "failed to solve dependencies" # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE [ -n "$package_name" ] && { - spectool -g -R "$spec_dir/${package_name}.spec" || die "failed to download source file" + spectool -g -R "$spec_dir/"*.spec || die "failed to download source file" } # Building rpm or srpm packages rpmbuild -ba "$spec_dir"/*.spec || die "failed to build rpms" -- 2.23.0

1 0

[PATCH v6 compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 28 Apr '21

28 Apr '21

To make sure git_repos displayed on the web are right. examples of git_repo: a/AvxToNeon/AvxToNeon 0/caterqiu/001 aur-a/a2png/a2png m/my-sql_with_boostkit_patch/my-sql_with_boostkit_patch Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 9e899d9..e305ac6 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -400,6 +400,14 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + + # What does this regular expression want: + # 1. a three-segment structure "xxx/xxx/xxx" + # 2. there can be '-' or '_' in every segment, but can't be at first or last of the segment. + # 3. the first segment can have lowercase of letters or numbers in it. + # 4. the other two segments can have letters(lowercase or uppercase) or numbers in it. + next unless r['git_repo'] =~ %r{^([a-z0-9]([a-z0-9\-_]*[a-z0-9])*(/[a-zA-Z0-9][\w\-]*[a-zA-Z0-9]){2})$} + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH lkp-tests] jobs/send-email.yaml: change the default value.
by Li Ping 28 Apr '21

28 Apr '21

the default value of email subject should not be job_failed, and there is already a send-email-on-fail.yaml. Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/send-email.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/send-email.yaml b/jobs/send-email.yaml index 8f8b48b96..2689634c2 100644 --- a/jobs/send-email.yaml +++ b/jobs/send-email.yaml @@ -3,4 +3,4 @@ # values of $subject are limited to job_failed/job_debug/borrow_success/borrow_failed. send_email: - subject: job_failed + subject: -- 2.23.0

1 0