- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] sbin/compare optimize help message
by Lu Weitao 12 Nov '20

12 Nov '20

[why] compare tool support output result with json | html format, but have no corresponding help message. [how] add the corresponding help message in --theme option [example] compare id=crystal.109482 id=crystal.109483 --theme json Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 8 ++++---- sbin/compare | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 9c15d99..b5257bc 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -16,10 +16,10 @@ server { autoindex on; autoindex_localtime on; allow all; - root /srv; - location ~* /[-_a-z]+$ { - add_header Content-Type text/plain; - } +# root /srv; +# location ~* /[-_a-z]+$ { +# add_header Content-Type text/plain; +# } } location ~* ^/result/(([^/]+)?/?|[^/]+/[^/]+/[^/]+/)$ { diff --git a/sbin/compare b/sbin/compare index b13682e..baa90d4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -12,7 +12,7 @@ # compare "commit=a12d232e" "commit=b3bacc31" # compare "os=debian" "os=centos" -c "suite=iperf" # compare "os=centos" -d "os_version os_arch" -# compare "os=centos" -d "os_version os_arch" --color "classic" +# compare "os=centos" -d "os_version os_arch" --theme "classic" require 'optparse' require_relative '../lib/compare.rb' @@ -42,9 +42,9 @@ opt_parser = OptionParser.new do |opts| is_group = true end - opts.on('--color color', 'turn on colorful display with theme: classic|focus_good|focus_bad', - '|striking|light|none') do |color| - colorful = color + opts.on('--theme theme', 'turn on colorful display with theme: classic|focus_good|focus_bad', + '|striking|light|json|html') do |theme| + colorful = theme end opts.on('-t', '--template template', 'compare with user-defined template') do |t| -- 2.23.0

2 2

[PATCH v2 compass-ci] splitting the cifs services
by Xiao Shenwei 12 Nov '20

12 Nov '20

[why] our os-cifs container will mount three dir, /srv/os /srv/initrd /srv/result it's may cause: 1. our cifs service becomes slow 2. if the container exit, none of the three folders can be accessed. in order to reduce the coupling, need split them [how] one folder mount by one container usage: mount /srv/initrd mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 22 +++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 36 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 23 ++++++++++++++++++++ 4 files changed, 86 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..d9cb28b --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN adduser -u 1090 -D lkp + +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash \ + && rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..ca9de39 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t alpine/initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..43d2b78 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,36 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp +[osimage] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..d16cc3f --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,23 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run -dt + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + alpine/initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

3 4

[PATCH v8 compass-ci 1/3] assign-account: add check my info
by Luan Shengde 12 Nov '20

12 Nov '20

check my_info when assigning account [why] my_info is required when assigning account. my_info is used to config the default yaml file [how] add check my_info before assign new account my_info: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..452a284 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,17 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + return_assign_jumper_account(data) +end + +def return_assign_jumper_account(data) + lacked_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lacked_info.join(', ')}." + raise error_message unless lacked_info.empty? + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

2 2

[PATCH compass-ci] scheduler: alter the major directory to find cluster config file
by Ren Wen 12 Nov '20

12 Nov '20

Cluster config files are placed under lab repo now. Scheduler will find cluster config files from: 1. lab repo (the major, take precedence). e.g. 'lab-z9/cluster/' 2. own lkp repo (for a quick test). e.g. 'lkp-tests/cluster/' Add a constant 'CCI_REPOS' for using. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 1 + src/scheduler/submit_job.cr | 14 +++++++++++--- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 41f2c75..db5089d 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -11,6 +11,7 @@ JOB_ES_PORT_DEBUG = 9201 JOB_INDEX_TYPE = "jobs/_doc" LAB = ENV["lab"]? || "nolab" +CCI_REPOS = ENV["CCI_REPOS"]? || "/c" SCHED_HOST = (ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1") SCHED_PORT = (ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"] : 3000).to_i32 diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index 8871464..34daa3d 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -153,11 +153,19 @@ class Sched @es.set_job_content(job) end - # get cluster config using own lkp_src cluster file, - # a hash type will be returned + # parse cluster config file from lab repo and own lkp repo, + # the prior takes precedence def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) - cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) + lkp_cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) + lab_cluster_file_path = Path.new(CCI_REPOS, "lab-#{LAB}", "cluster", cluster_file) + + if File.file?(lab_cluster_file_path) + cluster_file_path = lab_cluster_file_path + else + cluster_file_path = lkp_cluster_file_path + end + return YAML.parse(File.read(cluster_file_path)) end -- 2.23.0

3 4

[PATCH lkp-tests] tests/build-pkg: fix to create soft link failure
by Liu Shaofei 12 Nov '20

12 Nov '20

[why] the cgz file is not generated, so soft link cannot be created. error messages: ==> ERROR: Failure while downloading avro-c-1.9.2.tar.gz Aborting... ==> /tmp/stderr <== realpath: /initrd/build-pkg/nfs/openeuler/aarch64/20.03/avro-c/HEADCHECK.cgz: No such file or directory ln: failed to create symbolic link '/initrd/build-pkg/nfs/openeuler/aarch64/20.03/avro-c/latest.cgz' -> '': No such file or directory Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index ad8d4843..c29b6a20 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -93,6 +93,8 @@ create_softlink() local soft_path="$PKG_MNT/${pack_to}/${pkgname}" local bm_name=$(basename $(realpath ${cgz_name})) + [ -e "$cgz_name" ] || return 0 + if [ -n "$upstream_tag" ]; then echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" -- 2.23.0

2 2

[PATCH lkp-tests 1/2] add rootfs/tools: build-deps-pkg for rootfs-build-layout
by Sun Yukui 12 Nov '20

12 Nov '20

Automatic packaging when rootfs is built ln -s $LKP_SRC/rootfs/tools/build-deps-pkg.sh \ $CCI_SRC/rootfs/$os_mount/$os/$os_arch/$os_version/build-depends ln -s $LKP_SRC/rootfs/tools/build-deps-pkg.sh \ $CCI_SRC/rootfs/$os_mount/$os/$os_arch/$os_version/build-makepkg Eg: cd $CCI_SRC/rootfs/$os_mount/$os/$os_arch/$os_version and ./build-depends OR ./build-depends $program1 $program2... for depends ./build-makepkg OR ./build-makepkg $program1 $program2... for makepkg Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- rootfs/tools/build-deps-pkg.sh | 78 ++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 rootfs/tools/build-deps-pkg.sh diff --git a/rootfs/tools/build-deps-pkg.sh b/rootfs/tools/build-deps-pkg.sh new file mode 100755 index 00000000..573414ce --- /dev/null +++ b/rootfs/tools/build-deps-pkg.sh @@ -0,0 +1,78 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ + +# Exclude 'dev' or some files that do not need to perform cci-depends. +suffix_detect() +{ + [ ${1##*.} != $1 ] || [ ${1##*-} == 'dev' ] +} + +submit_job() +{ + command submit "$CCI_SRC/rootfs/build-deps-pkg.yaml" +} + +deps_generate_yaml() +{ + export suite='cci-depends' + + for benchmark + do + suffix_detect "$benchmark" && continue + [ -f "$LKP_SRC/distro/depends/$benchmark" ] || continue + + export benchmark + submit_job + done +} + +pkg_generate_yaml() +{ + export suite='cci-makepkg' + + for benchmark + do + [ -f "$LKP_SRC/pkg/$benchmark/PKGBUILD" ] || continue + + export benchmark + submit_job + done +} + +set_vars() +{ + local work_dir=$(pwd) + local os_path=${work_dir##*/rootfs/} + local os_array=($(echo "$os_path" | tr '/' ' ')) + + [[ "${work_dir}" == "${os_path}" ]] && { + echo "error: script execution path error" + echo "cd ${CCI_SRC}/rootfs/\$os_mount/\$os/\$os_arch/\$os_version; ./${0}" + exit 1 + } + + [[ "${#os_array[@]}" == 4 ]] || { + echo "error: expect 4 parameters, found ${#os_array[@]}" + exit 2 + } + + export os_mount="${os_array[0]}" + export os="${os_array[1]}" + export os_arch="${os_array[2]}" + export os_version="${os_array[3]}" +} + +main() +{ + set_vars + + if [ "$#" -gt 0 ]; then + [[ "$0" == 'build-depends' ]] && deps_generate_yaml "$@" + [[ "$0" == 'build-makepkg' ]] && pkg_generate_yaml "$@" + else + [[ "$0" == 'build-depends' ]] && deps_generate_yaml $(ls "$LKP_SRC"/distro/depends) + [[ "$0" == 'build-makepkg' ]] && pkg_generate_yaml $(ls "$LKP_SRC"/pkg) + fi +} + +main "$@" -- 2.23.0

1 1

[PATCH compass-ci] fix: scheduler has 'lab' not 'LAB' in environment
by Ren Wen 12 Nov '20

12 Nov '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 839c241..41f2c75 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -10,7 +10,7 @@ JOB_ES_PORT = 9200 JOB_ES_PORT_DEBUG = 9201 JOB_INDEX_TYPE = "jobs/_doc" -LAB = (ENV.has_key?("LAB") ? ENV["LAB"] : "nolab") +LAB = ENV["lab"]? || "nolab" SCHED_HOST = (ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1") SCHED_PORT = (ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"] : 3000).to_i32 -- 2.23.0

2 2

[PATCH compass-ci] src/lib/web_backend.rb: add date range limit feature
by Zhang Yuhang 12 Nov '20

12 Nov '20

[why] The "start_time" field need limit by date. example: GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 57 +++++++++++++++++-------------- src/lib/web_backend.rb | 15 ++++++++ 2 files changed, 46 insertions(+), 26 deletions(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index fcc765f..7db82ac 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -1,4 +1,6 @@ #!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'sinatra' @@ -12,56 +14,59 @@ set :show_exceptions, false set :environment, :production # GET /compare_candidates -# return: {"query_conditions": {...}, "dimension": [...]} +# Response: +# - {"query_conditions": {...}, "dimension": [...]} get '/compare_candidates' do compare_candidates end # GET /compare?suite=borrow&dimension=tbox_group -# query_conditions: -# > must +# must: +# - query_conditions # - os / os_version / suite / os_arch / tbox_group -# -# dimension: -# > must +# - dimension # - os / os_version / os_arch / suite / tbox_group # -# return: compare result string / "No Data." / "No Difference." +# Response: +# - compare result string / "No Data." / "No Difference." get '/compare' do compare(params) end -# GET /get_jobs?upstream_repo=AvxToNeon/AvxToNeon&page_size=12&page_num=3 -# upstream_repo: -# > optional -# -# page_size: -# > optional +# GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 +# optional: +# - upstream_repo +# - regex search +# - os +# - os_version +# - os_arch +# - suite +# - category +# - testbox +# - job_state +# - id +# - start_date +# - end_date +# - page_size # - default: 20 -# -# page_num: -# > optional +# - page_num # - default: 1 # -# return: +# Response: # - default: 20 jobs get '/get_jobs' do get_jobs(params) end # GET /get_repos?git_repo=a&page_size=12&page_num=3 -# git_repo: -# > optional -# -# page_size: -# > optional +# optional: +# - git_repo +# - page_size # - default: 20 -# -# page_num: -# > optional +# - page_num # - default: 1 # -# return: +# Response: # - default: 20 repos get '/get_repos' do get_repos(params) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 55c5e4a..b8df6f4 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -1,3 +1,5 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'json' @@ -260,6 +262,17 @@ def get_jobs_result(result) jobs end +def get_job_query_range(condition_fields) + range = { start_time: {} } + start_date = condition_fields[:start_date] + end_date = condition_fields[:end_date] + + range[:start_time][:gte] = "#{start_date} 00:00:00" if start_date + range[:start_time][:lte] = "#{end_date} 23:59:59" if end_date + + { range: range } +end + def search_job(condition_fields, page_size, page_num) must = [] FIELDS.each do |field| @@ -272,6 +285,8 @@ def search_job(condition_fields, page_size, page_num) { term: { field => value } } end end + range = get_job_query_range(condition_fields) + must << range if range[:range][:start_time] result, total = es_search(must, page_size, page_num * page_size) return get_jobs_result(result), total end -- 2.23.0

2 2

[PATCH v5 compass-ci 01/12] mail-robot: run mail-robot service
by Luan Shengde 12 Nov '20

12 Nov '20

call mail-robot.rb parse-apply-account-email.rb gitee-commit-url-check.rb assign-account.rb apply-jumper-account.rb assign-account-email.rb assign-account-fail-eamil.rb to run the email-robot service Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/run-mail-robot.rb | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/mail-robot/run-mail-robot.rb diff --git a/container/mail-robot/run-mail-robot.rb b/container/mail-robot/run-mail-robot.rb new file mode 100755 index 0000000..ab65452 --- /dev/null +++ b/container/mail-robot/run-mail-robot.rb @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-fail-eamil" +require_relative "#{ENV['CCI_SRC']}/lib/apply-jumper-account" +require_relative "#{ENV['CCI_SRC']}/lib/gitee-commit-url-check.rb" +require_relative "#{ENV['CCI_SRC']}/lib/parse-apply-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account" +require_relative "#{ENV['CCI_SRC']}/lib/mail-robot" + +MAILDIR = ENV['MAILDIR'] +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

2 2

[PATCH v5 compass-ci 07/12] mail-robot: build success email
by Luan Shengde 12 Nov '20

12 Nov '20

build email message for successfully assigning account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-email.rb | 53 +++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 lib/assign-account-email.rb diff --git a/lib/assign-account-email.rb b/lib/assign-account-email.rb new file mode 100755 index 0000000..fd19e10 --- /dev/null +++ b/lib/assign-account-email.rb @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_email(my_info) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{my_info['my_name']}, + + Thank you for joining us. + + You can use the following info to submit jobs: + + notice: + the followings are ONE-TIME setup + + 1) setup default config + run the following command to add the below setup to default config file + + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF + my_email: #{my_info['my_email']} + my_name: #{my_info['my_name']} + my_uuid: #{my_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + + reference to 'how to write job yaml' section to write the job yaml + you can also reference to files in compass-ci/jobs as example. + + submit jobs, for example: + + submit jobs/iperf.yaml textbox=vm-2p8g + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

2 2