December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container: mount "/srv/git/archlinux" to srv-http
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index 8c8ca0b..d9f461b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -15,6 +15,7 @@ cmd=( -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/os:/srv/os:ro + -v /srv/git/archlinux:/srv/git/archlinux:ro -d srv-http ) -- 2.23.0

1 1

[PATCH v3 compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..8edd28e 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,11 @@ dirs=( ) mkdir -p "${dirs[@]}" + +if [ "$os_mount" = "nfs" ]; then + mount -t "$os_mount" $LKP_SERVER:os /srv/os + mount -t "$os_mount" $LKP_SERVER:initrd /srv/initrd +elif [ "$os_mount" = "cifs" ]; then + mount -t "$os_mount" //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr + mount -t "$os_mount" //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr +fi -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] mail-robot: optimize parse email_file
by Luan Shengde 29 Dec '20

29 Dec '20

[why] avoid build too much messages in the code files [how] just raise error code in the code file, and build the error message in the file building email message Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 4 +-- .../lib/parse-apply-account-email.rb | 31 ++++++------------- 2 files changed, 11 insertions(+), 24 deletions(-) diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb index 7cac0d5..af31e17 100755 --- a/container/mail-robot/lib/gitee-commit-url-check.rb +++ b/container/mail-robot/lib/gitee-commit-url-check.rb @@ -38,9 +38,7 @@ class GiteeCommitUrlCheck def gitee_commit_exist(email_index) return if email_index - error_message = "We can not confirm whether the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with your email.' + error_message = 'COMMIT_URL_NOT_AVAILABLE' raise error_message end diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 7024731..a41a89a 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -5,6 +5,7 @@ require 'json' require 'mail' +require 'rest-client' require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey @@ -27,7 +28,7 @@ require_relative 'gitee-commit-url-check' # gitee.com: # call GiteeCommitUrlCheck to check the commit # non-gitee.com: -# execute curl to check the commit +# get request date for the commit url to check the commit # parse_pub_key # check my_ssh_pubkey exists and return it class ParseApplyAccountEmail @@ -63,10 +64,8 @@ class ParseApplyAccountEmail # the commit url should be in a standart format, example: # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) - error_message = "No matched commit url found.\n" - error_message += 'Ensure that you have add a right commit url, ' - error_message += "and with prefix 'my oss commit:'." - raise error_message + raise 'URL_PREFIX_ERR' unless mail_content_line.match?(%r{my oss commit:\s*https?://}) + raise 'NO_COMMIT_URL' unless mail_content_line.match?(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) end mail_content_line.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] @@ -88,16 +87,13 @@ class ParseApplyAccountEmail return unless match_out.empty? - error_message = "The repo url for your commit is not in our upstream-repo list.\n" - error_message += 'Use a new one, or consulting the manager for available repo list.' - - raise error_message + raise 'NOT_REGISTERED' end def commit_url_availability(url, base_url) hub_name = url.split('/')[2] - # it requires authentication when execute curl to get the commit information + # it requires authentication when sending request to get the commit information # clone the repo and then validate the commit for the email address if hub_name.eql? 'gitee.com' gitee_commit(url, base_url) @@ -112,24 +108,17 @@ class ParseApplyAccountEmail end def non_gitee_commit(url) - url_fdback = %x(curl #{url}) + url_fdback = RestClient.get(url).body email_index = url_fdback.index @my_info['my_email'] return unless email_index.nil? - error_message = "We can not confirm the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with you email.' - - raise error_message + raise 'COMMIT_URL_NOT_AVAILABLE' end def parse_pub_key - error_message = "No pub_key found.\n" - error_message += 'Please add a pub_key as an attachment to your email.' - - raise error_message if @mail_content.attachments.empty? - raise error_message unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ + raise 'NO_PUBKEY' if @mail_content.attachments.empty? + raise 'PUBKEY_NAME_ERR' unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ pub_key = @mail_content.attachments[0].body.decoded -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] mail-robot: add error code/message
by Luan Shengde 29 Dec '20

29 Dec '20

add error message for each error code before building the email_message. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 87 +++++++++++++++---- 1 file changed, 69 insertions(+), 18 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 593f345..242dfe1 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -3,32 +3,83 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_fail_email(my_info, message) - email_msg = <<~EMAIL_MESSAGE - To: #{my_info['my_email']} - Subject: [compass-ci] apply account failed +def email_err_message(message) + case message + when 'NO_COMMIT_URL' + err_message = <<~EMAIL_MESSAGE + No commit url found. + You should add a commit url for the 'apply account' email. - Dear user: + For example: + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'URL_PREFIX_ERR' + err_message = <<~EMAIL_MESSAGE + Please add a correct prefix for the commit url. + + my oss commit + + For example: + + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'NOT_REGISTERED' + err_message = <<~EMAIL_MESSAGE + Your repo has not been registered to our upstream-repos yet. + You should register your repo to the upstream first. + Try again after you have done it. + + Reference the following url to learn how to register the repo: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… + EMAIL_MESSAGE + when 'COMMIT_URL_NOT_AVAILABLE' + err_message = <<~EMAIL_MESSAGE + We cannot confirm the commit url matches your email. + + Make sure the commit exists. + Make sure it is truely submitted with your email. - Your application for account failed with following error: + Go to the commit history and find your commit. + Enter the commit and you will get your commit url. - #{message} + Try again after you have correctted the url + EMAIL_MESSAGE + when 'NO_PUBKEY' + err_message = <<~EMAIL_MESSAGE + No pub_key found. + Please add a pubkey to the email and then try again. + EMAIL_MESSAGE + when 'PUBKEY_NAME_ERR' + err_message = <<~EMAIL_MESSAGE + Pubkey file name error, keep its name as when it was generated. + The pubkey file name should like: - In order to successfully apply an account, please pay attention to the following points: + id_{{ xxx }}.pub + EMAIL_MESSAGE + end - 1. mail subject - The subject should exactly: apply account + err_message += <<~EMAIL_MESSAGE - 2. commit url - When you writing the url, add prefix: my oss commit - example: - my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + Manual for how to apply account: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… + EMAIL_MESSAGE + + err_message +end + +def build_apply_account_fail_email(my_info, message) + err_message = email_err_message(message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: - attention: - Ensure you commit url exist and available to access. + Your application for account failed. - 3. ssh pubkey - You need to add a pubkey as an attachment to the email. + #{err_message} regards compass-ci -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] assign-account: change get to post
by Luan Shengde 29 Dec '20

29 Dec '20

use post is more easier to use rest-client to send request with data for applying account. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index 4ed95c0..c5d77db 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -12,9 +12,9 @@ require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 -get '/assign_account' do +post '/assign_account' do begin - data = YAML.safe_load request.body.read + data = JSON.parse request.body.read rescue StandardError => e puts e.message puts e.backtrace -- 2.23.0

1 0

[PATCH v5 compass-ci] lib: add build_my_info_client
by Luan Shengde 29 Dec '20

29 Dec '20

purpose: build my_info for users when they installing compass-ci as private environment, if not, will face account verification problems when they submitting jobs - store my_info to ES - write my_info/lab to default/lab yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 76 +++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..162bf9f --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,76 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'nolab' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp, + 'my_login_name' => nil, + 'my_commit_url' => nil, + 'my_ssh_pubkey' => [] + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + default_yaml_info['lab'] = @lab + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + store_account_info + end +end -- 2.23.0

1 0

[PATCH compass-ci] container/auto-submit: add user lkp
by Li Yuanchao 29 Dec '20

29 Dec '20

[error message] auto-submit: whoami: unknown uid 1090 Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/auto-submit/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/container/auto-submit/Dockerfile b/container/auto-submit/Dockerfile index 5b44551..52cd996 100644 --- a/container/auto-submit/Dockerfile +++ b/container/auto-submit/Dockerfile @@ -3,6 +3,7 @@ FROM alpine:3.11 +RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk update && \ -- 2.23.0

1 0

[PATCH compass-ci] lib/mail-post: add cc/bcc for send mail service
by Luan Shengde 29 Dec '20

29 Dec '20

enable user send Cc and Bcc email. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index ac455fa..9d3e601 100644 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -20,6 +20,8 @@ post '/send_mail_yaml' do mail_info = { 'subject' => data['subject'], 'to' => data['to'], + 'cc' => data['cc'], + 'bcc' => data['bcc'], 'body' => data['body'] } @@ -32,6 +34,8 @@ post '/send_mail_text' do mail_info = { 'subject' => data.subject, 'to' => data.to, + 'cc' => data.cc, + 'bcc' => data.bcc, 'body' => data.body.decoded } @@ -45,6 +49,8 @@ post '/send_mail_encode' do mail_info = { 'subject' => data.subject, 'to' => data.to, + 'cc' => data.cc, + 'bcc' => data.bcc, 'body' => data.body.decoded } @@ -63,10 +69,16 @@ def send_mail(mail_info) from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] to mail_info['to'] + cc mail_info['cc'] + bcc mail_info['bcc'] body mail_info['body'] end mail.deliver! + check_to_store_email(mail) +end + +def check_to_store_email(mail) return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/bisect: add PKGBUILD info in bisect report
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 7 +++++++ lib/mail_bisect_result.rb | 10 +++++++--- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index f789a71..2043730 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -22,6 +22,7 @@ class GitBisect set_bad_job set_build_pkg_dir set_upstream + set_pkgbuild_repo set_work_dir set_bad_commit set_good_commit @@ -61,6 +62,11 @@ class GitBisect raise "checkout repo: #{@upstream_repo} to commit: #{@upstream_commit} failed!" unless @work_dir end + def set_pkgbuild_repo + @pkgbuild_repo = @bad_job['pkgbuild_repo'] + raise 'pkgbuild_repo is null' unless @pkgbuild_repo + end + def set_bad_commit @bad_commit = @upstream_commit end @@ -100,6 +106,7 @@ class GitBisect 'commit' => first_bad_commit, 'bisect_error' => bisect_error, 'all_errors' => all_errors, + 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 1d4ed69..5f88c36 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -9,10 +9,11 @@ require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @bisect_error = bisect_info['bisect_error'] - @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @all_errors = bisect_info['all_errors'] + @bisect_error = bisect_info['bisect_error'] + @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) # now send mail to review @@ -28,6 +29,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY @@ -40,9 +42,11 @@ class MailBisectResult commit: #{@commit_id} ("#{(a)git_commit.subject}") compiler: gcc (GCC) 7.3.0 - all errors/warnings: + all errors/warnings (new ones prefixed by >>): #{@all_errors} + reference information: + #{pkgbuild_repo_url} #{bisect_job_url} #{first_bad_commit_job_url} Regards, -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/READEME.md: fix format error
by Liu Yinsi 29 Dec '20

29 Dec '20

correct READEME format and content. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index a9a4bb0..c7a80e7 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -52,10 +52,10 @@ submit iperf.yaml testbox=vm-2p8g ``` - 执行上述命令正常情况下会提示信息如下： - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci搭建完毕，下面就可以开始进行测试了。 + 执行上述命令正常情况下会提示信息如下: + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci搭建完毕，执行步骤8下载所需要的rootfs文件就可以开始进行测试了。 8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） ```bash @@ -81,7 +81,9 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… id查看output文件 +14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… + 本地搭建compass-ci用户可在srv/result目录下根据yaml名称/日期/$testbox/$os-$os-version-$os-arch/$job_id查看output文件(可用tab键自动补全多级目录方便查找) + 以查看iperf任务结果为例操如下： ```bash - cd /srv/result/iperf/$job_id/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/nolab.1/output ``` -- 2.23.0

1 0