- Compass-ci - mailweb.openeuler.org

[PATCH v1 compass-ci 3/4] container: the build script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assist-result/build diff --git a/container/assist-result/build b/container/assist-result/build new file mode 100755 index 0000000..130127b --- /dev/null +++ b/container/assist-result/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-result . -- 2.23.0

1 0

[PATCH v1 compass-ci 2/4] container: the Dockerfile for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 container/assist-result/Dockerfile diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile new file mode 100644 index 0000000..5672652 --- /dev/null +++ b/container/assist-result/Dockerfile @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 0

[PATCH v1 compass-ci 1/4] container: add get_job_yaml in assist-result container
by Cao Xueliang 28 Dec '20

28 Dec '20

The assist-result container mount the "/srv/result" dir, we should implement some API about "/srv/result". An API is provided to support to get job yaml for now. Example: input: curl localhost:8102/get_job_yaml/crystal.304189 output: the yaml content about crystal.304189 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 24 +++++++++++++++++++ container/assist-result/views/get_job_yaml.rb | 13 ++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/assist-result/routes.rb create mode 100644 container/assist-result/views/get_job_yaml.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb new file mode 100755 index 0000000..1013643 --- /dev/null +++ b/container/assist-result/routes.rb @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'sinatra' + +require_relative './views/get_job_yaml' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_RESULT_PORT'] +end + +get '/get_job_yaml/:job_id' do + begin + result = get_job_yaml(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_yaml.rb b/container/assist-result/views/get_job_yaml.rb new file mode 100644 index 0000000..4544835 --- /dev/null +++ b/container/assist-result/views/get_job_yaml.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_yaml(job_id) + content = ESQuery.new.query_by_id(job_id) + result_root = content['result_root'] + return YAML.load(File.open(File.join('/srv', "#{result_root}", 'job.yaml'))) +end -- 2.23.0

1 0

[PATCH compass-ci 2/4] container: the Dockerfile for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 container/assist-result/Dockerfile diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile new file mode 100644 index 0000000..5672652 --- /dev/null +++ b/container/assist-result/Dockerfile @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 1

[PATCH compass-ci] lib/params_group.rb optimize filter_groups
by Lu Weitao 28 Dec '20

28 Dec '20

[Why] We had for-each in hash, needn't query value by key once more Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 7bb7f5b..bf229e6 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -72,10 +72,10 @@ end def filter_groups(groups) groups.each do |group_key, value| - value.each_key do |dim_key| - value.delete(dim_key) if value[dim_key].empty? + value.each do |dim_key, job_list| + value.delete(dim_key) if job_list.empty? end - groups.delete(group_key) if groups[group_key].empty? + groups.delete(group_key) if value.empty? end end -- 2.23.0

2 5

[PATCH v2 lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 28 Dec '20

28 Dec '20

[error] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..29b423b69 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,7 +48,10 @@ get_config() upload_config() { - upload_to_target_dir "${cci_dir}/${pkgname}" + local upload_dir="${cci_dir}/${pkgname}" + [ -d "${upload_dir}" ] || mkdir -p "${upload_dir}" + + upload_to_target_dir "${upload_dir}" } mount_dest() -- 2.23.0

1 0

[PATCH compass-ci] lib/json_logger: delete caller
by Wu Zhende 28 Dec '20

28 Dec '20

caller does not provide error information. It's not helpful for fault locating. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/json_logger.rb | 1 - src/lib/json_logger.cr | 1 - 2 files changed, 2 deletions(-) diff --git a/lib/json_logger.rb b/lib/json_logger.rb index 6d1173e..78f95ca 100644 --- a/lib/json_logger.rb +++ b/lib/json_logger.rb @@ -25,7 +25,6 @@ class JSONLogger < Logger } logger_hash['progname'] = progname if progname - logger_hash['caller'] = caller if level_num >= 2 msg = { 'message' => msg } unless msg.is_a?(Hash) logger_hash.merge!(msg) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index 6d61197..fcd65fb 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -25,7 +25,6 @@ class JSONLogger < Logger })).as_h logger_hash.any_merge!({"progname" => progname}) unless progname.empty? - logger_hash.merge!(JSON.parse(%({"caller": #{caller}})).as_h) if level_num >= 2 begin message = JSON.parse(msg).as_h -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 28 Dec '20

28 Dec '20

[why] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..f6c27a754 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,6 +48,8 @@ get_config() upload_config() { + [ -d "${cci_dir}/${pkgname}" ] || mkdir -p "${cci_dir}/${pkgname}" + upload_to_target_dir "${cci_dir}/${pkgname}" } -- 2.23.0

2 2

[PATCH compass-ci] 3-code/git: iadd repository version pointing
by Hu Xuejiao 28 Dec '20

28 Dec '20

[why] the code downloaded from upstream_repo occupies too much memory, the repository version needs to specify a version Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 2159ddc..e2b3cbf 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -29,4 +29,7 @@ cd /c || exit [ -d "upstream-repos/.git" ] || { git clone https://gitee.com/wu_fengguang/upstream-repos.git + + cd /c/upstream-repos || exit + [ -n "$upstream_commitid" ] && git reset --hard "$upstream_commitid" } -- 2.23.0

2 1

[PATCH compass-ci 5/6] mail-robot: add error code/message
by Luan Shengde 28 Dec '20

28 Dec '20

add error message for each error code before building the email_message. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 87 +++++++++++++++---- 1 file changed, 69 insertions(+), 18 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 593f345..5871a6a 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -3,32 +3,83 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_fail_email(my_info, message) - email_msg = <<~EMAIL_MESSAGE - To: #{my_info['my_email']} - Subject: [compass-ci] apply account failed +def email_err_message(message) + case message + when 'NO_COMMIT_URL' + err_message = <<~EMAIL_MESSAGE + No commit url found. + You should add a commit url for the 'apply account' email. - Dear user: + For example: + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'URL_PREFIX_ERR' + err_message = <<~EMAIL_MESSAGE + Please add a correct prefix for the commit url. + + my oss commit + + For example: + + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'NOT_REGISTERED' + err_message = <<~EMAIL_MESSAGE + Your repo has not been registered to our upstream-repos yet. + You should register your repo to the upstream first. + Try again after you have done it. + + Reference the following url to learn how to register the repo: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… + EMAIL_MESSAGE + when 'COMMIT_URL_NOT_AVAILABLE' + err_message = <<~EMAIL_MESSAGE + We cannot confirm the commit url matches your email. + + Make sure the commit exists. + Make sure it is truely submitted with you email. - Your application for account failed with following error: + Go to the commit history and find your commit. + Enter the commit and you will get your commit url. - #{message} + Try again after you have correctted the url + EMAIL_MESSAGE + when 'NO_PUBKEY' + err_message = <<~EMAIL_MESSAGE + No pub_key found. + Please add a pubkey to the email and then try again. + EMAIL_MESSAGE + when 'PUBKEY_NAME_ERR' + err_message = <<~EMAIL_MESSAGE + Pubkey file name error, keep it's name as when it was generated. + The pubkey file name should like: - In order to successfully apply an account, please pay attention to the following points: + id_{{ xxx }}.pub + EMAIL_MESSAGE + end - 1. mail subject - The subject should exactly: apply account + err_message += <<~EMAIL_MESSAGE - 2. commit url - When you writing the url, add prefix: my oss commit - example: - my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + Manual for how to apply account: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… + EMAIL_MESSAGE + + err_message +end + +def build_apply_account_fail_email(my_info, message) + err_message = email_err_message(message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: - attention: - Ensure you commit url exist and available to access. + Your application for account failed. - 3. ssh pubkey - You need to add a pubkey as an attachment to the email. + #{err_message} regards compass-ci -- 2.23.0

1 0

[PATCH compass-ci 4/6] mail-robot: optimize message message
by Luan Shengde 28 Dec '20

28 Dec '20

optimize mail message for successfully applied account [why] its used linux command 'curl' to send request data when sending email. so the ${HOME} and ${SHELL##*/} is treated as a system variable and replaced with the variable value. as a result, we will send a unexpected email. to avoid this problem, need add '\\' before '$' in the old. [how] use rest-client instead of command 'curl' to send the request. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 466586d..ee3b574 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -37,7 +37,7 @@ def build_apply_account_email(my_info) git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source \\${HOME}/.\\${SHELL##*/}rc + source ${HOME}/.${SHELL##*/}rc 3) submit job reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md -- 2.23.0

1 0

[PATCH compass-ci 3/6] mail-robot: optimize apply-jumper-account
by Luan Shengde 28 Dec '20

28 Dec '20

use rest-client instead of linux command 'curl' to send http request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-jumper-account.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/apply-jumper-account.rb b/container/mail-robot/lib/apply-jumper-account.rb index 44f412b..73d2946 100755 --- a/container/mail-robot/lib/apply-jumper-account.rb +++ b/container/mail-robot/lib/apply-jumper-account.rb @@ -22,8 +22,8 @@ class ApplyJumperAccount end def apply_jumper_account - account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ - -d '#{(a)my_info.to_json}') + assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" + account_info_str = RestClient.post assign_account_url, @my_info.to_json account_info = JSON.parse account_info_str account_info_exist(account_info) -- 2.23.0

1 0

[PATCH compass-ci 2/6] mail-robot: optimize apply-account
by Luan Shengde 28 Dec '20

28 Dec '20

use rest-client instead of linux command 'curl' to send request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index aea1aa5..e8288d7 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -5,6 +5,7 @@ require 'json' require 'mail' +require 'rest-client' require_relative '../../../lib/es_client' require_relative 'assign-account-email' require_relative 'assign-account-fail-eamil' @@ -131,6 +132,7 @@ class ApplyAccount build_apply_account_fail_email(@my_info, error_message) end - %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + send_mail_url = "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" + RestClient.post send_mail_url, email_message end end -- 2.23.0

1 0

[PATCH compass-ci 1/6] mail-robot: update Dockerfile
by Luan Shengde 28 Dec '20

28 Dec '20

[why] the old way is use linux command 'curl' to send http request. it's not human friendly to read for user [how] install rest-client instead of linux command 'curl' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile index 76dc94d..f0aebbd 100644 --- a/container/mail-robot/Dockerfile +++ b/container/mail-robot/Dockerfile @@ -8,7 +8,8 @@ MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + apt-get install -y git uuid-runtime ruby-listen ruby-json ruby-mail ruby-rest-client + RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: fix spell error
by Lu Kaiyi 28 Dec '20

28 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- src/lib/web_backend.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 0ca71e1..ea67064 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -328,7 +328,7 @@ def get_banner(git_repo, branches) } end -def get_optimize_jobs_braches(jobs) +def get_optimize_jobs_branches(jobs) branch_set = Set.new jobs.size.times do |i| branch = jobs[i]['upstream_branch'].to_s @@ -347,7 +347,7 @@ def get_jobs_body(params) page_size = get_positive_number(params.delete(:page_size), 20) page_num = get_positive_number(params.delete(:page_num), 1) - 1 jobs, total = search_job(params, page_size, page_num) - jobs, branches = get_optimize_jobs_braches(jobs) + jobs, branches = get_optimize_jobs_branches(jobs) { total: total, filter: params, -- 2.23.0

1 0

[PATCH compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 26 Dec '20

26 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..eb7270d 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,6 @@ dirs=( ) mkdir -p "${dirs[@]}" + +mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr +mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr -- 2.23.0

1 0

[PATCH compass-ci 3/4] assign-account: optimize answerback-email
by Luan Shengde 26 Dec '20

26 Dec '20

[why] the old way to send request is use linux command 'curl', it's not good for reading and has low efficiency use ruby gem resource rest-client instead of linux command Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 21285e7..a45fef2 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -11,6 +11,7 @@ require 'json' require 'mail' require 'set' require 'optparse' +require 'rest-client' require_relative '../defconfig' require_relative '../../lib/es_client' require_relative 'build-send-account-email' @@ -180,7 +181,8 @@ def apply_account(my_info, conf_info) apply_info.update conf_info apply_info['lab'] = LAB - account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" + account_info_str = RestClient.post assign_account_url, apply_info.to_json JSON.parse account_info_str end @@ -285,7 +287,8 @@ def send_mail(my_info, account_info, conf_info) build_message(my_info['my_email'], account_info) end - %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") + send_mail_url = "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" + RestClient.post send_mail_url, message end def store_account_info(my_info) -- 2.23.0

1 0

[PATCH compass-ci 1/4] assign-account: update Dockerfile
by Luan Shengde 26 Dec '20

26 Dec '20

[why] no need to install command 'curl' in the container Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/assign-account/Dockerfile b/container/assign-account/Dockerfile index 12faafa..1d9c36c 100644 --- a/container/assign-account/Dockerfile +++ b/container/assign-account/Dockerfile @@ -8,7 +8,7 @@ ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ssh ruby-sinatra ruby-json curl && \ + apt-get install -y ssh ruby-sinatra ruby-json && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone COPY assign-account.rb /usr/local/bin/ -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/run_job: Automatic task execution
by Hu Xuejiao 26 Dec '20

26 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 00000000..c6cf9e85 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="${tbox_group}.${HOSTNAME}" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

1 0

[PATCH v3 compass-ci] lib: add build_my_info_client
by Luan Shengde 26 Dec '20

26 Dec '20

purpose: build my_info for users when they installing compass-ci as private environment if not, will face account verification problems when they submitting jobs - store my_info to ES - write my_info to default/lab yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 79 +++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..2a664b4 --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,79 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'z9' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def complete_my_info + @my_info['my_login_name'] = nil + @my_info['my_commit_url'] = nil + @my_info['my_ssh_pubkey'] = [] + end + + def store_account_info + complete_my_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + store_account_info + end +end -- 2.23.0

2 5

[PATCH compass-ci] providers: qemu/kvm.sh check kernel file exists
by Xiao Shenwei 26 Dec '20

26 Dec '20

[why] if os_http not working, the kernel file will can't be download or the size is 0, then the qemu-kvm will boot by pxe, the vm will request the next job again. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/qemu/kvm.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b9f5716..fe2345f 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -51,6 +51,11 @@ do #done < /tftpboot/boot.ipxe-centos done < $ipxe_script +[ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit +} + [ -n "$initrds" ] || { exit } -- 2.23.0

1 0

[PATCH compass-ci] [RFC] providers/qemu: adapt kvm.sh for x86_64/riscv64
by Liu Yinsi 26 Dec '20

26 Dec '20

run qemu in different arch have different options, put same options in $kvm, different options in options_$arch. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 41 +++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index b9f5716..56cf8c0 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -16,7 +16,7 @@ if [ ! -f "$log_file" ]; then fi qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +command -v $qemu >/dev/null || qemu=qemu-kvm || qemu=qemu-system-riscv64 echo $SCHED_PORT ipxe_script=ipxe_script @@ -67,15 +67,10 @@ sleep 5 kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -device virtio-gpu-pci - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic @@ -83,4 +78,38 @@ kvm=( -monitor null ) +options_aarch64=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -device virtio-gpu-pci + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +options_x86_64=( + -device virtio-gpu-pci + -bios /usr/share/ovmf/OVMF.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +options_riscv64=( + -machine virt + -device virtio-net-device,netdev=net1,mac=${mac} + -netdev bridge,br=br0,id=net1,helper=/usr/lib/qemu/qemu-bridge-helper + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} +) + +case $qemu in + qemu-system-aarch64) + kvm=(${kvm[@]} ${args_aarch64[@]}) + ;; + qemu-kvm) + [ $(arch) == "x86_64" ] && kvm=(${kvm[@]} ${args_x86_64[@]}) + [ $(arch) == "aarch64" ] && kvm=(${kvm[@]} ${args_aarch64[@]}) + ;; + qemu-system-riscv) + kvm=(${kvm[@]} ${args_riscv64[@]}) + ;; +esac + "${kvm[@]}" --append "${append}" -- 2.23.0

1 1

[PATCH v2 compass-ci] result-webdav/nginx.conf: solve the problem that the symbolic link cannot be established
by Cui Yili 25 Dec '20

25 Dec '20

[why] Solve the problem that the symbolic link cannot be established when the directory is not created [how] Create a directory first. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 1feddd4..cc21926 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,7 +54,7 @@ http { end link_name = link_name..".cgz" root_path = "/srv/"..path - io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) + io.popen("mkdir -p "..root_path.." && cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) end redis_host = os.getenv("REDIS_HOST") -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/nginx.conf: solve the problem that the symbolic link cannot be established
by Cui Yili 25 Dec '20

25 Dec '20

[why] Solve the problem that the symbolic link cannot be established when the directory is not created [how] Create a directory first. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/nginx.conf | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/nginx.conf b/container/result-webdav/nginx.conf index 1feddd4..00abc30 100644 --- a/container/result-webdav/nginx.conf +++ b/container/result-webdav/nginx.conf @@ -54,6 +54,7 @@ http { end link_name = link_name..".cgz" root_path = "/srv/"..path + io.popen("mkdir -p "..root_path) io.popen("cd "..root_path.."&& ln -sf "..upload_file.." "..link_name) end -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] lib/matrix2.rb: fix incorrect average and stddev
by Lu Weitao 25 Dec '20

25 Dec '20

[Why] We just skip useless job when make job_list --> matrix, had not did: job_list.size - 1, the average: sum($job['stats']['metric'])/job_list.size will be incorrect. standard deviation is the same reason [How] Remove useless job before job_list --> matrix, and do minor refactoring Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 11 ++--------- lib/params_group.rb | 16 ++++++++++++++-- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 14e9558..3c4035e 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -83,7 +83,7 @@ def create_matrix(job_list) suites = [] job_list.each do |job| stats = job['stats'] - next unless stats && assign_suites(suites, job, stats) + suites << job['suite'] if job['suite'] stats.each do |key, value| next if key.include?('timestamp') @@ -97,17 +97,10 @@ def create_matrix(job_list) return matrix, suites end -def assign_suites(suites, job, stats) - return unless job['suite'] && stats.keys.any? { |stat| stat.start_with?(job['suite']) } - - suites << job['suite'] -end - # input: query results from es_query # return: matrix def combine_query_data(query_data) - job_list = query_data['hits']['hits'] - job_list.map! { |job| job['_source'] } + job_list = extract_jobs_list(query_data['hits']['hits']) create_matrix(job_list) end diff --git a/lib/params_group.rb b/lib/params_group.rb index 57cd5c1..7bb7f5b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -38,9 +38,21 @@ def auto_group(jobs_list, dimensions) end def extract_jobs_list(jobs_list) - jobs_list.map do |job| - job['_source'] + jobs_list.map! do |job| + job['_source'] if job_is_useful?(job) end + + jobs_list.compact +end + +def job_is_useful?(job) + stats = job['_source']['stats'] + return unless stats + + suite = job['_source']['suite'] + return unless suite && stats.keys.any? { |stat| stat.start_with?(suite) } + + true end def group(jobs_list, dimensions) -- 2.23.0

1 0

2025

2024

2023

2022

2021

2020

Compass-ci