- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] sparrow: fix 3-code/dev-env no such file error
by Liu Yinsi 12 Nov '20

12 Nov '20

when execute 3-code/dev-env, error message as follows: ./dev-env: line 45: ./os/openEuler: No such file or directory Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 77027d9..e17ddcc 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -42,4 +42,5 @@ EOF source /etc/os-release -. $(dirname ${BASH_SOURCE[0]})/os/${ID} +[ -x $(dirname ${BASH_SOURCE[0]})/os/${ID} ] && + . $(dirname ${BASH_SOURCE[0]})/os/${ID} -- 2.23.0

2 2

[PATCH v4 compass-ci] container: fix failed to build kibana images
by Liu Yinsi 12 Nov '20

12 Nov '20

[why] when build kibana images in x86_machine, error message: standard_init_linux.go:220: exec user process caused "exec format error" libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error" The command '/bin/sh -c sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/kibana/Dockerfile | 4 +++- container/kibana/build | 8 +++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 35802fe..6e0dba0 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -1,7 +1,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM gagara/kibana-oss-arm64:7.6.2 +ARG BASE_IMAGE + +FROM BASE_IMAGE # docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 diff --git a/container/kibana/build b/container/kibana/build index a7e4717..52d5a2a 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -3,4 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -system 'docker build -t kibana:7.6.2 .' +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', + 'x86_64' => 'kibana:7.6.2' }.freeze + +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] + +system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." -- 2.23.0

3 4

[PATCH v2 lkp-tests] sbin/makepkg: download submodules code from LKP server
by Liu Shaofei 12 Nov '20

12 Nov '20

[why] Some repository have submodules. So when build packages, we need download submodules from LKP server, avoid the waste of time in downloading files from the remote end. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..050f4fac 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -623,6 +623,15 @@ extract_git() { cd_safe "${dir##*/}" + # modify submodules address to local server address. + # upstream_repo is externlly imported variable, like this: + # upstream_repo="c/czmq/czmq.git" + if [[ -f ".gitmodules" ]]; then + local project=${upstream_repo%/*} + local url="git://${LKP_SERVER}/submodules/${project}/" + sed -i "s|url = .*\/|${url}|g" ".gitmodules" + fi + local ref=origin/HEAD if [[ -n $fragment ]]; then case ${fragment%%=*} in -- 2.23.0

2 2

[PATCH compass-ci] sparrow: fix 3-code/dev-env no such file error
by Liu Yinsi 12 Nov '20

12 Nov '20

[why] when execute 3-code/dev-env, error: ./dev-env: line 45: ./os/openEuler: No such file or directory Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 77027d9..e17ddcc 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -42,4 +42,5 @@ EOF source /etc/os-release -. $(dirname ${BASH_SOURCE[0]})/os/${ID} +[ -x $(dirname ${BASH_SOURCE[0]})/os/${ID} ] && + . $(dirname ${BASH_SOURCE[0]})/os/${ID} -- 2.23.0

4 5

[PATCH v2 lkp-tests] tests/build-pkg: fix to create soft link failure
by Liu Shaofei 12 Nov '20

12 Nov '20

[why] the cgz file is not generated, so soft link cannot be created. error messages: ==> ERROR: Failure while downloading avro-c-1.9.2.tar.gz Aborting... ==> /tmp/stderr <== realpath: /initrd/build-pkg/nfs/openeuler/aarch64/20.03/avro-c/HEADCHECK.cgz: No such file or directory ln: failed to create symbolic link '/initrd/build-pkg/nfs/openeuler/aarch64/20.03/avro-c/latest.cgz' -> '': No such file or directory Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index ad8d4843..1eed784f 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -90,6 +90,8 @@ build_source_pkg() create_softlink() { + [ -e "$cgz_name" ] || return 0 + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" local bm_name=$(basename $(realpath ${cgz_name})) -- 2.23.0

1 0

[PATCH compass-ci] contsiner/https-scheduler: raise warning if missing ca
by Wu Zhende 12 Nov '20

12 Nov '20

Not all environments require https scheduler. So start it only if ca certificate exists. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 3 +++ 1 file changed, 3 insertions(+) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 38c4892..6d6dda0 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -6,6 +6,9 @@ require 'set' require_relative '../defconfig.rb' +warning = 'CA certificate is missing.Ignore if the HTTPS scheduler is not required' +raise warning unless File.exist?('/etc/compass-ci/ca/server.crt') && File.exist?('/etc/compass-ci/ca/server.key') + names = Set.new %w[ SCHED_HOST SCHED_PORT -- 2.23.0

3 2

[PATCH v2 compass-ci 2/4] assign-account: add new option
by Luan Shengde 12 Nov '20

12 Nov '20

add -r|--rsa-key option for answerback-email [why] enable one click login testbox from jumper server [how] add -r|--rsa-key option to enable generate rsa private/public key on jumper server register the public key to testbox Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 686a327..6d8e3de 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -32,7 +32,8 @@ my_info = { 'my_email' => nil, 'my_name' => nil, 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => nil + 'my_ssh_pubkey' => nil, + 'my_jumper_pubkey' => false } def init_info(email_file, my_info) @@ -68,6 +69,10 @@ options = OptionParser.new do |opts| init_info(email_file, my_info) end + opts.on('-r', '--rsa-key', 'generate jumper ras public/private key and return pubkey') do + my_info['my_jumper_pubkey'] = true + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -114,10 +119,12 @@ def send_account(my_info) raise message if my_info['my_email'].nil? account_info = apply_account(my_info) + # for manually assign account, there will be no my_commit_url # but the key my_commit_url is required for es my_info['my_commit_url'] = '' my_info['my_login_name'] = account_info['my_login_name'] + my_info['my_jumper_pubkey'] = account_info['my_jumper_pubkey'] my_info.delete 'my_ssh_pubkey' store_account_info(my_info) message = build_message(my_info['my_email'], account_info) -- 2.23.0

2 4

[PATCH v2 compass-ci 4/4] assign-account: add /etc/hostname mapping
by Luan Shengde 12 Nov '20

12 Nov '20

add /etc/hostname mapping for assign-account [why] the hostname is needed when execute ssh-keygen to generate ras public/private key with -C option in the container %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@#{hostname}") [how] add file mapping for file /etc/hostname Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/start | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/assign-account/start b/container/assign-account/start index 2290982..379dbd3 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -14,6 +14,8 @@ cmd=( -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v /srv/account-info:/opt/account_data/:rw + -v /etc/hostname:/etc/hostname:ro + -v /etc/localtime:/etc/localtime:ro -v /home:/home:rw -p 29999:29999 assign-account -- 2.23.0

2 2

[PATCH v5 lkp-tests] job2sh.rb: define on_fail
by Hu Xuejiao 12 Nov '20

12 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/job.sh . $LKP_SRC/lib/env.sh run_test $LKP_SRC/tests/wrapper sleep 300 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7e6a5d7c..e15888ea 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -358,6 +360,25 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

2 1

[PATCH v5 lkp-tests] submit on_fail to the scheduler
by Hu Xuejiao 12 Nov '20

12 Nov '20

[why] when the task is failed, it need in the waiting state [how] When job2sh.rb defined sh_on_fail, it will be invoked by the scheduler. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..0bb41f2d 100755 --- a/sbin/submit +++ b/sbin/submit @@ -115,12 +115,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_fail sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

1 0

[PATCH v4 compass-ci 03/12] mail-robot: check to assign account
by Luan Shengde 12 Nov '20

12 Nov '20

AssignAccount assign account for user required data: mail_content - send_account parse email_content parse commit url parse pubkey apply_my_account generate uuid apply account store account info to es send email send success email send fail email usage: reference mail-robot.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account.rb | 78 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 lib/assign-account.rb diff --git a/lib/assign-account.rb b/lib/assign-account.rb new file mode 100755 index 0000000..5521ae6 --- /dev/null +++ b/lib/assign-account.rb @@ -0,0 +1,78 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'es_client' + +# assign uuid/account for user +# send_account +# parse my_commit_url and my_ssh_pubkey +# apply account +# store account info +# send mail +class AssignAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def send_account + my_commit_url, my_ssh_pubkey = parse_commit_url_pub_key + acct_info = apply_my_account(my_ssh_pubkey) + + @my_info['my_commit_url'] = my_commit_url + @my_info['my_login_name'] = acct_info['my_login_name'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + my_commit_url = parse_apply_account_email.parse_commit_url + my_ssh_pubkey = parse_apply_account_email.parse_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def apply_my_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info, my_ssh_pubkey) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(message) + email_message = if message.empty? + build_apply_account_email(@my_info) + else + build_apply_account_fail_email(@my_info, message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

2 3

[PATCH v2 compass-ci 1/4] assign-account: Dockerfile add install ssh
by Luan Shengde 12 Nov '20

12 Nov '20

add install ssh for assign-account [why] need ssh-keygen to generate rsa public/private key for user [how] add install ssh when build the image Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/Dockerfile | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/container/assign-account/Dockerfile b/container/assign-account/Dockerfile index f84b5e6..c1d360e 100644 --- a/container/assign-account/Dockerfile +++ b/container/assign-account/Dockerfile @@ -2,15 +2,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER luanshd0525(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ruby-sinatra ruby-json curl && \ - ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ - echo 'Asia/Shanghai' > /etc/timezone + apt-get install -y ssh ruby-sinatra ruby-json curl + COPY assign-account.rb /usr/local/bin/ COPY get_account_info.rb /usr/local/bin/ -- 2.23.0

2 2

[PATCH v2 compass-ci] obtain the queues parameter from CLI
by Xiao Shenwei 12 Nov '20

12 Nov '20

Usage: multi-docker -n -c -q -n, --name HOSTNAME_PREFIX format: $tbox_group.$HOSTNAME -c, --count count how many container do you need -q, --queues queues separated by "," -h, --help show this message Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-docker | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/providers/multi-docker b/providers/multi-docker index 64cafaa..984359f 100755 --- a/providers/multi-docker +++ b/providers/multi-docker @@ -9,17 +9,21 @@ require_relative './docker/docker' opt = {} options = OptionParser.new do |opts| - opts.banner = 'Usage: multi-qemu [-t] [-n]' + opts.banner = 'Usage: multi-docker -n -c -q' opts.separator '' - opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'specify used hostname_prefix') do |name| + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'format: $tbox_group.$HOSTNAME') do |name| opt['hostname_prefix'] = name end - opts.on('-c count', '--count count', 'how much container do you need') do |num| + opts.on('-c count', '--count count', 'how many container do you need') do |num| opt['nr_container'] = num end + opts.on('-q queues', '--queues queues', 'separated by ","') do |queues| + opt['queues'] = queues + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -28,13 +32,14 @@ end if ARGV.size.zero? puts options - exit + exit 1 end options.parse!(ARGV) -hostname = opt['hostname_prefix'] || 'dc-1g' +hostname = opt['hostname_prefix'] || "dc-1g.#{ENV['HOSTNAME']}" nr_container = opt['nr_container'] || 1 +queues = opt['queues'] || "dc-1g.#{ENV['HOSTNAME']}" -pids = multi_docker(hostname, nr_container) +pids = multi_docker(hostname, nr_container, queues) save_pid pids -- 2.23.0

2 2

[PATCH lkp-tests] fix /cci-makepkg A softlink will be established if the package fails
by Sun Yukui 12 Nov '20

12 Nov '20

[why] After packaging fails, a softlink will be established, but this is wrong. If the packaging fails, the soft connection should not be established. Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- tests/cci-makepkg | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..c6fce675 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -92,10 +92,6 @@ cgz_name="${pkgver:-0}-${pkgrel:-0}.cgz" setup_proxy -echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf -PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf - update_softlink() { [ -e "$sync_dest/$cgz_name" ] || return @@ -109,4 +105,13 @@ update_softlink() rm $sync_dest/$cgz_name } -update_softlink && echo "Makepkg finished successfully" +build_source_pkg() +{ + echo $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + PACMAN=true BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" \ + $LKP_SRC/sbin/makepkg -A --skippgpcheck --config $LKP_SRC/etc/makepkg.conf + + [ "$?" == 0 ] || update_softlink +} + +build_source_pkg -- 2.23.0

2 1

[PATCH compass-ci] container/srv-http: fix download *.log file in browser
by Lu Weitao 12 Nov '20

12 Nov '20

[why] haven't define some file(*.log) behavior of browser in mime.types, so download this file on browser. [how] 1. add correspondence of file in mime.types 2. add default type in default.conf in case of other type file that haven't define in mime.types Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 1 + container/srv-http/root/etc/nginx/mime.types | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..91371f1 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -13,6 +13,7 @@ server { } location / { + default_type text/html; autoindex on; autoindex_localtime on; allow all; diff --git a/container/srv-http/root/etc/nginx/mime.types b/container/srv-http/root/etc/nginx/mime.types index e7f2f1c..a4ee74b 100644 --- a/container/srv-http/root/etc/nginx/mime.types +++ b/container/srv-http/root/etc/nginx/mime.types @@ -4,8 +4,8 @@ types { text/css css; text/xml xml; text/yaml yaml; - text/plain txt time data sh; + text/plain txt time data sh log; application/json json; - application/x-gzip gz tgz; + application/x-gzip gz tgz cgz; } -- 2.23.0

2 3

[PATCH v2 compass-ci] sbin/compare optimize help message
by Lu Weitao 12 Nov '20

12 Nov '20

[why] compare tool support output result with json | html format, but have no corresponding help message. [how] add the corresponding help message in --theme option [example] compare id=crystal.109482 id=crystal.109483 --theme json Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 8 ++++---- sbin/compare | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..b5257bc 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -16,10 +16,10 @@ server { autoindex on; autoindex_localtime on; allow all; - root /srv; - location ~* /[-_a-z]+$ { - add_header Content-Type text/plain; - } +# root /srv; +# location ~* /[-_a-z]+$ { +# add_header Content-Type text/plain; +# } } location ~* ^/result/(([^/]+)?/?|[^/]+/[^/]+/[^/]+/)$ { diff --git a/sbin/compare b/sbin/compare index b13682e..baa90d4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -12,7 +12,7 @@ # compare "commit=a12d232e" "commit=b3bacc31" # compare "os=debian" "os=centos" -c "suite=iperf" # compare "os=centos" -d "os_version os_arch" -# compare "os=centos" -d "os_version os_arch" --color "classic" +# compare "os=centos" -d "os_version os_arch" --theme "classic" require 'optparse' require_relative '../lib/compare.rb' @@ -42,9 +42,9 @@ opt_parser = OptionParser.new do |opts| is_group = true end - opts.on('--color color', 'turn on colorful display with theme: classic|focus_good|focus_bad', - '|striking|light|none') do |color| - colorful = color + opts.on('--theme theme', 'turn on colorful display with theme: classic|focus_good|focus_bad', + '|striking|light|json|html') do |theme| + colorful = theme end opts.on('-t', '--template template', 'compare with user-defined template') do |t| -- 2.23.0

2 2

[PATCH v2 compass-ci] splitting the cifs services
by Xiao Shenwei 12 Nov '20

12 Nov '20

[why] our os-cifs container will mount three dir, /srv/os /srv/initrd /srv/result it's may cause: 1. our cifs service becomes slow 2. if the container exit, none of the three folders can be accessed. in order to reduce the coupling, need split them [how] one folder mount by one container usage: mount /srv/initrd mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 22 +++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 36 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 23 ++++++++++++++++++++ 4 files changed, 86 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..d9cb28b --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN adduser -u 1090 -D lkp + +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash \ + && rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..ca9de39 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t alpine/initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..43d2b78 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,36 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp +[osimage] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..d16cc3f --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run -dt + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + alpine/initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

3 4

[PATCH v8 compass-ci 1/3] assign-account: add check my info
by Luan Shengde 12 Nov '20

12 Nov '20

check my_info when assigning account [why] my_info is required when assigning account. my_info is used to config the default yaml file [how] add check my_info before assign new account my_info: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..452a284 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,17 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + return_assign_jumper_account(data) +end + +def return_assign_jumper_account(data) + lacked_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lacked_info.join(', ')}." + raise error_message unless lacked_info.empty? + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

2 2

[PATCH compass-ci] scheduler: alter the major directory to find cluster config file
by Ren Wen 12 Nov '20

12 Nov '20

Cluster config files are placed under lab repo now. Scheduler will find cluster config files from: 1. lab repo (the major, take precedence). e.g. 'lab-z9/cluster/' 2. own lkp repo (for a quick test). e.g. 'lkp-tests/cluster/' Add a constant 'CCI_REPOS' for using. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 1 + src/scheduler/submit_job.cr | 14 +++++++++++--- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 41f2c75..db5089d 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -11,6 +11,7 @@ JOB_ES_PORT_DEBUG = 9201 JOB_INDEX_TYPE = "jobs/_doc" LAB = ENV["lab"]? || "nolab" +CCI_REPOS = ENV["CCI_REPOS"]? || "/c" SCHED_HOST = (ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1") SCHED_PORT = (ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"] : 3000).to_i32 diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 8871464..34daa3d 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -153,11 +153,19 @@ class Sched @es.set_job_content(job) end - # get cluster config using own lkp_src cluster file, - # a hash type will be returned + # parse cluster config file from lab repo and own lkp repo, + # the prior takes precedence def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) - cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) + lkp_cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) + lab_cluster_file_path = Path.new(CCI_REPOS, "lab-#{LAB}", "cluster", cluster_file) + + if File.file?(lab_cluster_file_path) + cluster_file_path = lab_cluster_file_path + else + cluster_file_path = lkp_cluster_file_path + end + return YAML.parse(File.read(cluster_file_path)) end -- 2.23.0

3 4

[PATCH lkp-tests] tests/build-pkg: fix to create soft link failure
by Liu Shaofei 12 Nov '20

12 Nov '20

[why] the cgz file is not generated, so soft link cannot be created. error messages: ==> ERROR: Failure while downloading avro-c-1.9.2.tar.gz Aborting... ==> /tmp/stderr <== realpath: /initrd/build-pkg/nfs/openeuler/aarch64/20.03/avro-c/HEADCHECK.cgz: No such file or directory ln: failed to create symbolic link '/initrd/build-pkg/nfs/openeuler/aarch64/20.03/avro-c/latest.cgz' -> '': No such file or directory Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index ad8d4843..c29b6a20 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -93,6 +93,8 @@ create_softlink() local soft_path="$PKG_MNT/${pack_to}/${pkgname}" local bm_name=$(basename $(realpath ${cgz_name})) + [ -e "$cgz_name" ] || return 0 + if [ -n "$upstream_tag" ]; then echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" -- 2.23.0

2 2

[PATCH lkp-tests 1/2] add rootfs/tools: build-deps-pkg for rootfs-build-layout
by Sun Yukui 12 Nov '20

12 Nov '20

Automatic packaging when rootfs is built ln -s $LKP_SRC/rootfs/tools/build-deps-pkg.sh \ $CCI_SRC/rootfs/$os_mount/$os/$os_arch/$os_version/build-depends ln -s $LKP_SRC/rootfs/tools/build-deps-pkg.sh \ $CCI_SRC/rootfs/$os_mount/$os/$os_arch/$os_version/build-makepkg Eg: cd $CCI_SRC/rootfs/$os_mount/$os/$os_arch/$os_version and ./build-depends OR ./build-depends $program1 $program2... for depends ./build-makepkg OR ./build-makepkg $program1 $program2... for makepkg Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- rootfs/tools/build-deps-pkg.sh | 78 ++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 rootfs/tools/build-deps-pkg.sh diff --git a/rootfs/tools/build-deps-pkg.sh b/rootfs/tools/build-deps-pkg.sh new file mode 100755 index 00000000..573414ce --- /dev/null +++ b/rootfs/tools/build-deps-pkg.sh @@ -0,0 +1,78 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ + +# Exclude 'dev' or some files that do not need to perform cci-depends. +suffix_detect() +{ + [ ${1##*.} != $1 ] || [ ${1##*-} == 'dev' ] +} + +submit_job() +{ + command submit "$CCI_SRC/rootfs/build-deps-pkg.yaml" +} + +deps_generate_yaml() +{ + export suite='cci-depends' + + for benchmark + do + suffix_detect "$benchmark" && continue + [ -f "$LKP_SRC/distro/depends/$benchmark" ] || continue + + export benchmark + submit_job + done +} + +pkg_generate_yaml() +{ + export suite='cci-makepkg' + + for benchmark + do + [ -f "$LKP_SRC/pkg/$benchmark/PKGBUILD" ] || continue + + export benchmark + submit_job + done +} + +set_vars() +{ + local work_dir=$(pwd) + local os_path=${work_dir##*/rootfs/} + local os_array=($(echo "$os_path" | tr '/' ' ')) + + [[ "${work_dir}" == "${os_path}" ]] && { + echo "error: script execution path error" + echo "cd ${CCI_SRC}/rootfs/\$os_mount/\$os/\$os_arch/\$os_version; ./${0}" + exit 1 + } + + [[ "${#os_array[@]}" == 4 ]] || { + echo "error: expect 4 parameters, found ${#os_array[@]}" + exit 2 + } + + export os_mount="${os_array[0]}" + export os="${os_array[1]}" + export os_arch="${os_array[2]}" + export os_version="${os_array[3]}" +} + +main() +{ + set_vars + + if [ "$#" -gt 0 ]; then + [[ "$0" == 'build-depends' ]] && deps_generate_yaml "$@" + [[ "$0" == 'build-makepkg' ]] && pkg_generate_yaml "$@" + else + [[ "$0" == 'build-depends' ]] && deps_generate_yaml $(ls "$LKP_SRC"/distro/depends) + [[ "$0" == 'build-makepkg' ]] && pkg_generate_yaml $(ls "$LKP_SRC"/pkg) + fi +} + +main "$@" -- 2.23.0

1 1

[PATCH compass-ci] fix: scheduler has 'lab' not 'LAB' in environment
by Ren Wen 12 Nov '20

12 Nov '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 839c241..41f2c75 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -10,7 +10,7 @@ JOB_ES_PORT = 9200 JOB_ES_PORT_DEBUG = 9201 JOB_INDEX_TYPE = "jobs/_doc" -LAB = (ENV.has_key?("LAB") ? ENV["LAB"] : "nolab") +LAB = ENV["lab"]? || "nolab" SCHED_HOST = (ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1") SCHED_PORT = (ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"] : 3000).to_i32 -- 2.23.0

2 2

[PATCH compass-ci] src/lib/web_backend.rb: add date range limit feature
by Zhang Yuhang 12 Nov '20

12 Nov '20

[why] The "start_time" field need limit by date. example: GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 57 +++++++++++++++++-------------- src/lib/web_backend.rb | 15 ++++++++ 2 files changed, 46 insertions(+), 26 deletions(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index fcc765f..7db82ac 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -1,4 +1,6 @@ #!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'sinatra' @@ -12,56 +14,59 @@ set :show_exceptions, false set :environment, :production # GET /compare_candidates -# return: {"query_conditions": {...}, "dimension": [...]} +# Response: +# - {"query_conditions": {...}, "dimension": [...]} get '/compare_candidates' do compare_candidates end # GET /compare?suite=borrow&dimension=tbox_group -# query_conditions: -# > must +# must: +# - query_conditions # - os / os_version / suite / os_arch / tbox_group -# -# dimension: -# > must +# - dimension # - os / os_version / os_arch / suite / tbox_group # -# return: compare result string / "No Data." / "No Difference." +# Response: +# - compare result string / "No Data." / "No Difference." get '/compare' do compare(params) end -# GET /get_jobs?upstream_repo=AvxToNeon/AvxToNeon&page_size=12&page_num=3 -# upstream_repo: -# > optional -# -# page_size: -# > optional +# GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 +# optional: +# - upstream_repo +# - regex search +# - os +# - os_version +# - os_arch +# - suite +# - category +# - testbox +# - job_state +# - id +# - start_date +# - end_date +# - page_size # - default: 20 -# -# page_num: -# > optional +# - page_num # - default: 1 # -# return: +# Response: # - default: 20 jobs get '/get_jobs' do get_jobs(params) end # GET /get_repos?git_repo=a&page_size=12&page_num=3 -# git_repo: -# > optional -# -# page_size: -# > optional +# optional: +# - git_repo +# - page_size # - default: 20 -# -# page_num: -# > optional +# - page_num # - default: 1 # -# return: +# Response: # - default: 20 repos get '/get_repos' do get_repos(params) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 55c5e4a..b8df6f4 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -1,3 +1,5 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'json' @@ -260,6 +262,17 @@ def get_jobs_result(result) jobs end +def get_job_query_range(condition_fields) + range = { start_time: {} } + start_date = condition_fields[:start_date] + end_date = condition_fields[:end_date] + + range[:start_time][:gte] = "#{start_date} 00:00:00" if start_date + range[:start_time][:lte] = "#{end_date} 23:59:59" if end_date + + { range: range } +end + def search_job(condition_fields, page_size, page_num) must = [] FIELDS.each do |field| @@ -272,6 +285,8 @@ def search_job(condition_fields, page_size, page_num) { term: { field => value } } end end + range = get_job_query_range(condition_fields) + must << range if range[:range][:start_time] result, total = es_search(must, page_size, page_num * page_size) return get_jobs_result(result), total end -- 2.23.0

2 2

[PATCH v5 compass-ci 01/12] mail-robot: run mail-robot service
by Luan Shengde 12 Nov '20

12 Nov '20

call mail-robot.rb parse-apply-account-email.rb gitee-commit-url-check.rb assign-account.rb apply-jumper-account.rb assign-account-email.rb assign-account-fail-eamil.rb to run the email-robot service Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/run-mail-robot.rb | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/mail-robot/run-mail-robot.rb diff --git a/container/mail-robot/run-mail-robot.rb b/container/mail-robot/run-mail-robot.rb new file mode 100755 index 0000000..ab65452 --- /dev/null +++ b/container/mail-robot/run-mail-robot.rb @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-fail-eamil" +require_relative "#{ENV['CCI_SRC']}/lib/apply-jumper-account" +require_relative "#{ENV['CCI_SRC']}/lib/gitee-commit-url-check.rb" +require_relative "#{ENV['CCI_SRC']}/lib/parse-apply-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account" +require_relative "#{ENV['CCI_SRC']}/lib/mail-robot" + +MAILDIR = ENV['MAILDIR'] +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

2 2

[PATCH v5 compass-ci 07/12] mail-robot: build success email
by Luan Shengde 12 Nov '20

12 Nov '20

build email message for successfully assigning account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-email.rb | 53 +++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 lib/assign-account-email.rb diff --git a/lib/assign-account-email.rb b/lib/assign-account-email.rb new file mode 100755 index 0000000..fd19e10 --- /dev/null +++ b/lib/assign-account-email.rb @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_email(my_info) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{my_info['my_name']}, + + Thank you for joining us. + + You can use the following info to submit jobs: + + notice: + the followings are ONE-TIME setup + + 1) setup default config + run the following command to add the below setup to default config file + + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF + my_email: #{my_info['my_email']} + my_name: #{my_info['my_name']} + my_uuid: #{my_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + + reference to 'how to write job yaml' section to write the job yaml + you can also reference to files in compass-ci/jobs as example. + + submit jobs, for example: + + submit jobs/iperf.yaml textbox=vm-2p8g + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

2 2