- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] contsiner/https-scheduler: raise warning if missing ca
by Wu Zhende 12 Nov '20

12 Nov '20

Not all environments require https scheduler. So start it only if ca certificate exists. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 38c4892..6d6dda0 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -6,6 +6,9 @@ require 'set' require_relative '../defconfig.rb' +warning = 'CA certificate is missing.Ignore if the HTTPS scheduler is not required' +raise warning unless File.exist?('/etc/compass-ci/ca/server.crt') && File.exist?('/etc/compass-ci/ca/server.key') + names = Set.new %w[ SCHED_HOST SCHED_PORT -- 2.23.0

3 2

[PATCH v2 compass-ci 2/4] assign-account: add new option
by Luan Shengde 12 Nov '20

12 Nov '20

add -r|--rsa-key option for answerback-email [why] enable one click login testbox from jumper server [how] add -r|--rsa-key option to enable generate rsa private/public key on jumper server register the public key to testbox Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 686a327..6d8e3de 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -32,7 +32,8 @@ my_info = { 'my_email' => nil, 'my_name' => nil, 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => nil + 'my_ssh_pubkey' => nil, + 'my_jumper_pubkey' => false } def init_info(email_file, my_info) @@ -68,6 +69,10 @@ options = OptionParser.new do |opts| init_info(email_file, my_info) end + opts.on('-r', '--rsa-key', 'generate jumper ras public/private key and return pubkey') do + my_info['my_jumper_pubkey'] = true + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -114,10 +119,12 @@ def send_account(my_info) raise message if my_info['my_email'].nil? account_info = apply_account(my_info) + # for manually assign account, there will be no my_commit_url # but the key my_commit_url is required for es my_info['my_commit_url'] = '' my_info['my_login_name'] = account_info['my_login_name'] + my_info['my_jumper_pubkey'] = account_info['my_jumper_pubkey'] my_info.delete 'my_ssh_pubkey' store_account_info(my_info) message = build_message(my_info['my_email'], account_info) -- 2.23.0

2 4

[PATCH v2 compass-ci 4/4] assign-account: add /etc/hostname mapping
by Luan Shengde 12 Nov '20

12 Nov '20

add /etc/hostname mapping for assign-account [why] the hostname is needed when execute ssh-keygen to generate ras public/private key with -C option in the container %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@#{hostname}") [how] add file mapping for file /etc/hostname Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/start | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/assign-account/start b/container/assign-account/start index 2290982..379dbd3 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -14,6 +14,8 @@ cmd=( -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v /srv/account-info:/opt/account_data/:rw + -v /etc/hostname:/etc/hostname:ro + -v /etc/localtime:/etc/localtime:ro -v /home:/home:rw -p 29999:29999 assign-account -- 2.23.0

2 2

[PATCH v5 lkp-tests] job2sh.rb: define on_fail
by Hu Xuejiao 12 Nov '20

12 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh run_test $LKP_SRC/tests/wrapper sleep 300 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7e6a5d7c..e15888ea 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -358,6 +360,25 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

2 1

[PATCH v5 lkp-tests] submit on_fail to the scheduler
by Hu Xuejiao 12 Nov '20

12 Nov '20

[why] when the task is failed, it need in the waiting state [how] When job2sh.rb defined sh_on_fail, it will be invoked by the scheduler. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..0bb41f2d 100755 --- a/sbin/submit +++ b/sbin/submit @@ -115,12 +115,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_fail sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

1 0

[PATCH v4 compass-ci 03/12] mail-robot: check to assign account
by Luan Shengde 12 Nov '20

12 Nov '20

AssignAccount assign account for user required data: mail_content - send_account parse email_content parse commit url parse pubkey apply_my_account generate uuid apply account store account info to es send email send success email send fail email usage: reference mail-robot.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account.rb | 78 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 lib/assign-account.rb diff --git a/lib/assign-account.rb b/lib/assign-account.rb new file mode 100755 index 0000000..5521ae6 --- /dev/null +++ b/lib/assign-account.rb @@ -0,0 +1,78 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'es_client' + +# assign uuid/account for user +# send_account +# parse my_commit_url and my_ssh_pubkey +# apply account +# store account info +# send mail +class AssignAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def send_account + my_commit_url, my_ssh_pubkey = parse_commit_url_pub_key + acct_info = apply_my_account(my_ssh_pubkey) + + @my_info['my_commit_url'] = my_commit_url + @my_info['my_login_name'] = acct_info['my_login_name'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + my_commit_url = parse_apply_account_email.parse_commit_url + my_ssh_pubkey = parse_apply_account_email.parse_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def apply_my_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info, my_ssh_pubkey) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(message) + email_message = if message.empty? + build_apply_account_email(@my_info) + else + build_apply_account_fail_email(@my_info, message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

2 3

[PATCH v2 compass-ci 1/4] assign-account: Dockerfile add install ssh
by Luan Shengde 12 Nov '20

12 Nov '20

add install ssh for assign-account [why] need ssh-keygen to generate rsa public/private key for user [how] add install ssh when build the image Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/Dockerfile | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/container/assign-account/Dockerfile b/container/assign-account/Dockerfile index f84b5e6..c1d360e 100644 --- a/container/assign-account/Dockerfile +++ b/container/assign-account/Dockerfile @@ -2,15 +2,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER luanshd0525(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ruby-sinatra ruby-json curl && \ - ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ - echo 'Asia/Shanghai' > /etc/timezone + apt-get install -y ssh ruby-sinatra ruby-json curl + COPY assign-account.rb /usr/local/bin/ COPY get_account_info.rb /usr/local/bin/ -- 2.23.0

2 2

[PATCH v2 compass-ci] obtain the queues parameter from CLI
by Xiao Shenwei 12 Nov '20

12 Nov '20

Usage: multi-docker -n -c -q -n, --name HOSTNAME_PREFIX format: $tbox_group.$HOSTNAME -c, --count count how many container do you need -q, --queues queues separated by "," -h, --help show this message Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-docker | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/providers/multi-docker b/providers/multi-docker index 64cafaa..984359f 100755 --- a/providers/multi-docker +++ b/providers/multi-docker @@ -9,17 +9,21 @@ require_relative './docker/docker' opt = {} options = OptionParser.new do |opts| - opts.banner = 'Usage: multi-qemu [-t] [-n]' + opts.banner = 'Usage: multi-docker -n -c -q' opts.separator '' - opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'specify used hostname_prefix') do |name| + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'format: $tbox_group.$HOSTNAME') do |name| opt['hostname_prefix'] = name end - opts.on('-c count', '--count count', 'how much container do you need') do |num| + opts.on('-c count', '--count count', 'how many container do you need') do |num| opt['nr_container'] = num end + opts.on('-q queues', '--queues queues', 'separated by ","') do |queues| + opt['queues'] = queues + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -28,13 +32,14 @@ end if ARGV.size.zero? puts options - exit + exit 1 end options.parse!(ARGV) -hostname = opt['hostname_prefix'] || 'dc-1g' +hostname = opt['hostname_prefix'] || "dc-1g.#{ENV['HOSTNAME']}" nr_container = opt['nr_container'] || 1 +queues = opt['queues'] || "dc-1g.#{ENV['HOSTNAME']}" -pids = multi_docker(hostname, nr_container) +pids = multi_docker(hostname, nr_container, queues) save_pid pids -- 2.23.0

2 2

[PATCH lkp-tests] fix /cci-makepkg A softlink will be established if the package fails
by Sun Yukui 12 Nov '20

12 Nov '20

[why] After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the soft connection should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..c6fce675 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,13 @@ update_softlink() rm $sync_dest/$cgz_name } -update_softlink && echo "Makepkg finished successfully" +build_source_pkg() +{ + echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ + $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + + [ "$?" == 0 ] || update_softlink +} + +build_source_pkg -- 2.23.0

2 1

[PATCH compass-ci] container/srv-http: fix download *.log file in browser
by Lu Weitao 12 Nov '20

12 Nov '20

[why] haven't define some file(*.log) behavior of browser in mime.types, so download this file on browser. [how] 1. add correspondence of file in mime.types 2. add default type in default.conf in case of other type file that haven't define in mime.types Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 1 + container/srv-http/root/etc/nginx/mime.types | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..91371f1 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -13,6 +13,7 @@ server { } location / { + default_type text/html; autoindex on; autoindex_localtime on; allow all; diff --git a/container/srv-http/root/etc/nginx/mime.types b/container/srv-http/root/etc/nginx/mime.types index e7f2f1c..a4ee74b 100644 --- a/container/srv-http/root/etc/nginx/mime.types +++ b/container/srv-http/root/etc/nginx/mime.types @@ -4,8 +4,8 @@ types { text/css css; text/xml xml; text/yaml yaml; - text/plain txt time data sh; + text/plain txt time data sh log; application/json json; - application/x-gzip gz tgz; + application/x-gzip gz tgz cgz; } -- 2.23.0

2 3