November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests/build-pkg: fix to create soft link failure
by Liu Shaofei 12 Nov '20

12 Nov '20

[why] the cgz file is not generated, so soft link cannot be created. error messages: ==> ERROR: Failure while downloading avro-c-1.9.2.tar.gz Aborting... ==> /tmp/stderr <== realpath: /initrd/build-pkg/nfs/openeuler/aarch64/20.03/avro-c/HEADCHECK.cgz: No such file or directory ln: failed to create symbolic link '/initrd/build-pkg/nfs/openeuler/aarch64/20.03/avro-c/latest.cgz' -> '': No such file or directory Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index ad8d4843..c29b6a20 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -93,6 +93,8 @@ create_softlink() local soft_path="$PKG_MNT/${pack_to}/${pkgname}" local bm_name=$(basename $(realpath ${cgz_name})) + [ -e "$cgz_name" ] || return 0 + if [ -n "$upstream_tag" ]; then echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" -- 2.23.0

2 2

[PATCH lkp-tests 1/2] add rootfs/tools: build-deps-pkg for rootfs-build-layout
by Sun Yukui 12 Nov '20

12 Nov '20

Automatic packaging when rootfs is built ln -s $LKP_SRC/rootfs/tools/build-deps-pkg.sh \ $CCI_SRC/rootfs/$os_mount/$os/$os_arch/$os_version/build-depends ln -s $LKP_SRC/rootfs/tools/build-deps-pkg.sh \ $CCI_SRC/rootfs/$os_mount/$os/$os_arch/$os_version/build-makepkg Eg: cd $CCI_SRC/rootfs/$os_mount/$os/$os_arch/$os_version and ./build-depends OR ./build-depends $program1 $program2... for depends ./build-makepkg OR ./build-makepkg $program1 $program2... for makepkg Signed-off-by: Sun Yukui <sun.yukui(a)foxmail.com> --- rootfs/tools/build-deps-pkg.sh | 78 ++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 rootfs/tools/build-deps-pkg.sh diff --git a/rootfs/tools/build-deps-pkg.sh b/rootfs/tools/build-deps-pkg.sh new file mode 100755 index 00000000..573414ce --- /dev/null +++ b/rootfs/tools/build-deps-pkg.sh @@ -0,0 +1,78 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ + +# Exclude 'dev' or some files that do not need to perform cci-depends. +suffix_detect() +{ + [ ${1##*.} != $1 ] || [ ${1##*-} == 'dev' ] +} + +submit_job() +{ + command submit "$CCI_SRC/rootfs/build-deps-pkg.yaml" +} + +deps_generate_yaml() +{ + export suite='cci-depends' + + for benchmark + do + suffix_detect "$benchmark" && continue + [ -f "$LKP_SRC/distro/depends/$benchmark" ] || continue + + export benchmark + submit_job + done +} + +pkg_generate_yaml() +{ + export suite='cci-makepkg' + + for benchmark + do + [ -f "$LKP_SRC/pkg/$benchmark/PKGBUILD" ] || continue + + export benchmark + submit_job + done +} + +set_vars() +{ + local work_dir=$(pwd) + local os_path=${work_dir##*/rootfs/} + local os_array=($(echo "$os_path" | tr '/' ' ')) + + [[ "${work_dir}" == "${os_path}" ]] && { + echo "error: script execution path error" + echo "cd ${CCI_SRC}/rootfs/\$os_mount/\$os/\$os_arch/\$os_version; ./${0}" + exit 1 + } + + [[ "${#os_array[@]}" == 4 ]] || { + echo "error: expect 4 parameters, found ${#os_array[@]}" + exit 2 + } + + export os_mount="${os_array[0]}" + export os="${os_array[1]}" + export os_arch="${os_array[2]}" + export os_version="${os_array[3]}" +} + +main() +{ + set_vars + + if [ "$#" -gt 0 ]; then + [[ "$0" == 'build-depends' ]] && deps_generate_yaml "$@" + [[ "$0" == 'build-makepkg' ]] && pkg_generate_yaml "$@" + else + [[ "$0" == 'build-depends' ]] && deps_generate_yaml $(ls "$LKP_SRC"/distro/depends) + [[ "$0" == 'build-makepkg' ]] && pkg_generate_yaml $(ls "$LKP_SRC"/pkg) + fi +} + +main "$@" -- 2.23.0

1 1

[PATCH compass-ci] fix: scheduler has 'lab' not 'LAB' in environment
by Ren Wen 12 Nov '20

12 Nov '20

Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/constants.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 839c241..41f2c75 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -10,7 +10,7 @@ JOB_ES_PORT = 9200 JOB_ES_PORT_DEBUG = 9201 JOB_INDEX_TYPE = "jobs/_doc" -LAB = (ENV.has_key?("LAB") ? ENV["LAB"] : "nolab") +LAB = ENV["lab"]? || "nolab" SCHED_HOST = (ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1") SCHED_PORT = (ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"] : 3000).to_i32 -- 2.23.0

2 2

[PATCH compass-ci] src/lib/web_backend.rb: add date range limit feature
by Zhang Yuhang 12 Nov '20

12 Nov '20

[why] The "start_time" field need limit by date. example: GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 57 +++++++++++++++++-------------- src/lib/web_backend.rb | 15 ++++++++ 2 files changed, 46 insertions(+), 26 deletions(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index fcc765f..7db82ac 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -1,4 +1,6 @@ #!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'sinatra' @@ -12,56 +14,59 @@ set :show_exceptions, false set :environment, :production # GET /compare_candidates -# return: {"query_conditions": {...}, "dimension": [...]} +# Response: +# - {"query_conditions": {...}, "dimension": [...]} get '/compare_candidates' do compare_candidates end # GET /compare?suite=borrow&dimension=tbox_group -# query_conditions: -# > must +# must: +# - query_conditions # - os / os_version / suite / os_arch / tbox_group -# -# dimension: -# > must +# - dimension # - os / os_version / os_arch / suite / tbox_group # -# return: compare result string / "No Data." / "No Difference." +# Response: +# - compare result string / "No Data." / "No Difference." get '/compare' do compare(params) end -# GET /get_jobs?upstream_repo=AvxToNeon/AvxToNeon&page_size=12&page_num=3 -# upstream_repo: -# > optional -# -# page_size: -# > optional +# GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 +# optional: +# - upstream_repo +# - regex search +# - os +# - os_version +# - os_arch +# - suite +# - category +# - testbox +# - job_state +# - id +# - start_date +# - end_date +# - page_size # - default: 20 -# -# page_num: -# > optional +# - page_num # - default: 1 # -# return: +# Response: # - default: 20 jobs get '/get_jobs' do get_jobs(params) end # GET /get_repos?git_repo=a&page_size=12&page_num=3 -# git_repo: -# > optional -# -# page_size: -# > optional +# optional: +# - git_repo +# - page_size # - default: 20 -# -# page_num: -# > optional +# - page_num # - default: 1 # -# return: +# Response: # - default: 20 repos get '/get_repos' do get_repos(params) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 55c5e4a..b8df6f4 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -1,3 +1,5 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'json' @@ -260,6 +262,17 @@ def get_jobs_result(result) jobs end +def get_job_query_range(condition_fields) + range = { start_time: {} } + start_date = condition_fields[:start_date] + end_date = condition_fields[:end_date] + + range[:start_time][:gte] = "#{start_date} 00:00:00" if start_date + range[:start_time][:lte] = "#{end_date} 23:59:59" if end_date + + { range: range } +end + def search_job(condition_fields, page_size, page_num) must = [] FIELDS.each do |field| @@ -272,6 +285,8 @@ def search_job(condition_fields, page_size, page_num) { term: { field => value } } end end + range = get_job_query_range(condition_fields) + must << range if range[:range][:start_time] result, total = es_search(must, page_size, page_num * page_size) return get_jobs_result(result), total end -- 2.23.0

2 2

[PATCH v5 compass-ci 01/12] mail-robot: run mail-robot service
by Luan Shengde 12 Nov '20

12 Nov '20

call mail-robot.rb parse-apply-account-email.rb gitee-commit-url-check.rb assign-account.rb apply-jumper-account.rb assign-account-email.rb assign-account-fail-eamil.rb to run the email-robot service Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/run-mail-robot.rb | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/mail-robot/run-mail-robot.rb diff --git a/container/mail-robot/run-mail-robot.rb b/container/mail-robot/run-mail-robot.rb new file mode 100755 index 0000000..ab65452 --- /dev/null +++ b/container/mail-robot/run-mail-robot.rb @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-fail-eamil" +require_relative "#{ENV['CCI_SRC']}/lib/apply-jumper-account" +require_relative "#{ENV['CCI_SRC']}/lib/gitee-commit-url-check.rb" +require_relative "#{ENV['CCI_SRC']}/lib/parse-apply-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account" +require_relative "#{ENV['CCI_SRC']}/lib/mail-robot" + +MAILDIR = ENV['MAILDIR'] +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

2 2

[PATCH v5 compass-ci 07/12] mail-robot: build success email
by Luan Shengde 12 Nov '20

12 Nov '20

build email message for successfully assigning account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-email.rb | 53 +++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 lib/assign-account-email.rb diff --git a/lib/assign-account-email.rb b/lib/assign-account-email.rb new file mode 100755 index 0000000..fd19e10 --- /dev/null +++ b/lib/assign-account-email.rb @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_email(my_info) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{my_info['my_name']}, + + Thank you for joining us. + + You can use the following info to submit jobs: + + notice: + the followings are ONE-TIME setup + + 1) setup default config + run the following command to add the below setup to default config file + + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF + my_email: #{my_info['my_email']} + my_name: #{my_info['my_name']} + my_uuid: #{my_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + + reference to 'how to write job yaml' section to write the job yaml + you can also reference to files in compass-ci/jobs as example. + + submit jobs, for example: + + submit jobs/iperf.yaml textbox=vm-2p8g + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

2 2

[PATCH v5 compass-ci 05/12] mail-robot: check commit url for gitee
by Luan Shengde 12 Nov '20

12 Nov '20

GiteeCommitUrlCheck check commit url availability for hub gitee - clone the repo - check commit log for the commit usage: reference parse-apply-account-email.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/gitee-commit-url-check.rb | 44 +++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100755 lib/gitee-commit-url-check.rb diff --git a/lib/gitee-commit-url-check.rb b/lib/gitee-commit-url-check.rb new file mode 100755 index 0000000..a9439ad --- /dev/null +++ b/lib/gitee-commit-url-check.rb @@ -0,0 +1,44 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check commit url availability for hub gitee.com +# gitee_commit_index +# clone the repo +# check commit available +class GiteeCommitUrlCheck + def initialize(my_info, url, base_url) + @my_info = my_info + @url = url + @base_url = base_url + end + + def gitee_commit_index + repo_dir = @url.split('/')[-3] + repo_url = [@base_url, 'git'].join('.') + commit_id = @url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] + + FileUtils.rm_rf repo_dir + + gitee_commit_exist(email_index) + end + + def gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = "We can not conform whether the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH v5 compass-ci 04/12] mail-robot: parse apply account email
by Luan Shengde 12 Nov '20

12 Nov '20

ParseApplyAccountEmail parse commit url and pubkey parse commit url check commit url exists check base url in upstream-repos check commit available hub: gitee.com call gitee-commit-url-check to check commit available hub: non-gitee.com check the feedback of curl command to check the commit url available parse pubkey attachment file name: id_rsa.pub usage: reference assign-account.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/parse-apply-account-email.rb | 115 +++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100755 lib/parse-apply-account-email.rb diff --git a/lib/parse-apply-account-email.rb b/lib/parse-apply-account-email.rb new file mode 100755 index 0000000..37063e0 --- /dev/null +++ b/lib/parse-apply-account-email.rb @@ -0,0 +1,115 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# parse mail_content +# parse_commit_url +# extract_commit_url +# check commit url exists +# base_url_in_upstream_repos +# check base url in upstream-repos +# commit_url_availability +# check commit available +# parse pub_key +# check pub_key exists +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + + # the commit url should be headed with a prefix: my oss commit + # the commit url must be in a standart format, example: + # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = "No matched commit url found.\n" + error_message += "Ensure that you have add a right commit url with prefix 'my oss commit:'." + + raise error_message + end + + def parse_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + base_url_in_upstream_repos('/c/upstream-repos', base_url) + commit_url_availability(url, base_url) + + return url + end + + def base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = "The repo url for your commit is not in our upstream-repo list.\n" + error_message += 'Use a new one, or consulting the manager for available repo list.' + + raise error_message + end + + def commit_url_availability(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + gitee_commit(url, base_url) + else + non_gitee_commit(url) + end + end + + def gitee_commit(url, base_url) + my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) + my_gitee_commit.gitee_commit_index + end + + def non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_info['my_email'] + + return unless email_index.nil? + + error_message = "We can not confirm the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end + + def parse_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + pub_key_exist(pub_key) + + return pub_key + end + + def pub_key_exist(pub_key) + return unless pub_key.nil? + + error_message = "No pub_key found.\n" + error_message += 'Please add the pub_key as an attachment with filename: id_rsa.pub.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH compass-ci 4/4] assign-account: add /etc/hostname mapping
by Luan Shengde 12 Nov '20

12 Nov '20

add /etc/hostname mapping for assign-account [why] it's need to get the jumper hostname when generate the rsa private/public key when execute ssh-keygen with -C option in container ssh-keygen -f /home/$login_name/.ssh/id_rsa -N '' -C $login_name@$hostname [how] add file mapping for file /etc/hostname Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/start | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/assign-account/start b/container/assign-account/start index 2290982..379dbd3 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -14,6 +14,8 @@ cmd=( -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v /srv/account-info:/opt/account_data/:rw + -v /etc/hostname:/etc/hostname:ro + -v /etc/localtime:/etc/localtime:ro -v /home:/home:rw -p 29999:29999 assign-account -- 2.23.0

2 2

[PATCH v2 lkp-tests] tests: adapt the special version of sysbench test
by Zhang Yu 12 Nov '20

12 Nov '20

[why] Use the special version of sysbench test mysql, use different test file are required. [how] Add parameters to test, and use different test file of sysbench Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 88 ++++++++++++++++++++++++++++++++------------ 1 file changed, 65 insertions(+), 23 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 4ab5b453..db9c57e1 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,52 +1,94 @@ #!/bin/sh -# mysql_user -# mysql_host -# mysql_port -# mysql_db -# db_driver # - oltp_test_mode # - oltp_tables_count # - oltp_table_size +# - max_requests +# - mysql_table_engine # - nr_threads +# - rand_type +# - rand_spec_pct # - runtime # - report_interval +# - oltp_read_only +# - events +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +# db_driver : "${mysql_user:=root}" -: "${mysql_host:=localhost}" -: "${mysql_port:=3306}" -: "${mysql_db:=test_1000}" +: "${mysql_host:=$direct_server_ips}" +: "${mysql_port:=$mysql_port}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=$mysql_password}" : "${db_driver:=mysql}" -: "${oltp_tables_count:=3}" -: "${oltp_table_size:=1000}" -: "${nr_threads:=64}" -: "${runtime:=120}" -: "${report_interval:=10}" +: "${oltp_test_mode:=complex}" +: "${oltp_tables_count:=1000}" +: "${oltp_table_size:=100000}" +: "${max_requests:=0}" +: "${mysql_table_engine:=innodb}" +: "${rand_type:=special}" +: "${rand_spec_pct:=100}" +: "${nr_threads:=256}" +: "${runtime:=600}" +: "${report_interval:=1}" +: "${oltp_read_only:=off}" +: "${events:=2000000000}" -args=( +args1=( --mysql-user=$mysql_user --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db + --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver - --tables=$oltp_tables_count - --table-size=$oltp_table_size + --mysql-password=$mysql_password + --max-requests=$max_requests + --mysql-table-engine=$mysql_table_engine + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --rand-type=$rand_type + --rand-spec-pct=$rand_spec_pct --threads=$nr_threads --time=$runtime - --report-interval=$report_interval ) +args2=( + --mysql-user=$mysql_user + --mysql-password=$mysql_password + --mysql-host=$mysql_host + --mysql-port=$mysql_port + --mysql-db=$mysql_db + --threads=$nr_threads + --oltp-read-only=$oltp_read_only + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --report-interval=$report_interval + --time=$runtime + --events=$events +) +stop_firewalld() +{ + systemctl stop firewalld + iptables -F +} + run_sysbench_step() { - sysbench /usr/share/sysbench/$1 "${args[@]}" $2 + lua_script=$1 + shift + sysbench /usr/local/share/sysbench/tests/include/oltp_legacy/$lua_script "$@" } run_sysbench_mysql() { - systemctl start mysqld - mysql -e "create database test_1000;" - run_sysbench_step oltp_common.lua prepare - run_sysbench_step oltp_read_write.lua run - run_sysbench_step oltp_common.lua cleanup + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + run_sysbench_step oltp.lua ${args2[@]} run + run_sysbench_step oltp.lua ${args2[@]} cleanup } +stop_firewalld run_sysbench_mysql + -- 2.23.0

2 2

[PATCH v2 compass-ci] support queues parameter
by Xiao Shenwei 12 Nov '20

12 Nov '20

register host2queues and requested specified queues Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 47 ++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 4e6bc2f..8e489c6 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -11,15 +11,26 @@ require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' +names = Set.new %w[ + SCHED_HOST + SCHED_PORT +] +defaults = relevant_defaults(names) +SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = defaults['SCHED_PORT'] || 3000 + def get_url(hostname) - names = Set.new %w[ - SCHED_HOST - SCHED_PORT - ] - defaults = relevant_defaults(names) - host = defaults['SCHED_HOST'] || '172.17.0.1' - port = defaults['SCHED_PORT'] || 3000 - "http://#{host}:#{port}/boot.container/hostname/#{hostname}" + "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" +end + +def set_host2queues(hostname, queues) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/set_host2queues?host=#{hostname}&queues=#{queues}'" + system cmd +end + +def del_host2queues(hostname) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/del_host2queues?host=#{hostname}'" + system cmd end def parse_response(url) @@ -67,7 +78,7 @@ def load_initrds(load_path, hash) wget_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") end -def run(hostname, load_path, hash) +def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( @@ -77,24 +88,26 @@ def run(hostname, load_path, hash) clean_dir(load_path) end -def main(hostname) +def main(hostname, queues) + set_host2queues(hostname, queues) url = get_url hostname puts url hash = parse_response url - return if hash.nil? + return del_host2queues(hostname) if hash.nil? load_path = build_load_path(hostname) load_initrds(load_path, hash) - run(hostname, load_path, hash) + start_container(hostname, load_path, hash) + del_host2queues(hostname) end -def loop_main(hostname) +def loop_main(hostname, queues) loop do begin - main(hostname) + main(hostname, queues) rescue StandardError => e puts e.backtrace - # if an exception happend, request the next time after 30 seconds + # if an exception occurs, request the next time after 30 seconds sleep 25 ensure sleep 5 @@ -109,11 +122,11 @@ def save_pid(pids) f.close end -def multi_docker(hostname, nr_container) +def multi_docker(hostname, nr_container, queues) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop_main("#{hostname}-#{i}") + loop_main("#{hostname}-#{i}", queues) end pids << pid end -- 2.23.0

2 2

[PATCH compass-ci 2/4] assign-account: add -r|--rsa-key option
by Luan Shengde 12 Nov '20

12 Nov '20

add -r|--rsa-key option for answerback-email [why] enable generate rsa private/public key on jumper server enable one click login whit the public key [how] add -r|--rsa-key option to enable generate rsa private/public key Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 686a327..6d8e3de 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -32,7 +32,8 @@ my_info = { 'my_email' => nil, 'my_name' => nil, 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => nil + 'my_ssh_pubkey' => nil, + 'my_jumper_pubkey' => false } def init_info(email_file, my_info) @@ -68,6 +69,10 @@ options = OptionParser.new do |opts| init_info(email_file, my_info) end + opts.on('-r', '--rsa-key', 'generate jumper ras public/private key and return pubkey') do + my_info['my_jumper_pubkey'] = true + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -114,10 +119,12 @@ def send_account(my_info) raise message if my_info['my_email'].nil? account_info = apply_account(my_info) + # for manually assign account, there will be no my_commit_url # but the key my_commit_url is required for es my_info['my_commit_url'] = '' my_info['my_login_name'] = account_info['my_login_name'] + my_info['my_jumper_pubkey'] = account_info['my_jumper_pubkey'] my_info.delete 'my_ssh_pubkey' store_account_info(my_info) message = build_message(my_info['my_email'], account_info) -- 2.23.0

2 2

[PATCH compass-ci 1/4] assign-account: Dockerfile add install ssh
by Luan Shengde 12 Nov '20

12 Nov '20

add install ssh for assign-account [why] it's need to generate rsa private/public key for user need install ssh for ssh-keygen command [how] add install ssh Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/Dockerfile | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/container/assign-account/Dockerfile b/container/assign-account/Dockerfile index f84b5e6..c1d360e 100644 --- a/container/assign-account/Dockerfile +++ b/container/assign-account/Dockerfile @@ -2,15 +2,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER luanshd0525(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ruby-sinatra ruby-json curl && \ - ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ - echo 'Asia/Shanghai' > /etc/timezone + apt-get install -y ssh ruby-sinatra ruby-json curl + COPY assign-account.rb /usr/local/bin/ COPY get_account_info.rb /usr/local/bin/ -- 2.23.0

2 2

[PATCH compass-ci] support queues parameter
by Xiao Shenwei 12 Nov '20

12 Nov '20

register host2queues and requested specified queues Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 47 ++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 4e6bc2f..8e489c6 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -11,15 +11,26 @@ require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' +names = Set.new %w[ + SCHED_HOST + SCHED_PORT +] +defaults = relevant_defaults(names) +SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = defaults['SCHED_PORT'] || 3000 + def get_url(hostname) - names = Set.new %w[ - SCHED_HOST - SCHED_PORT - ] - defaults = relevant_defaults(names) - host = defaults['SCHED_HOST'] || '172.17.0.1' - port = defaults['SCHED_PORT'] || 3000 - "http://#{host}:#{port}/boot.container/hostname/#{hostname}" + "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" +end + +def set_host2queues(hostname, queues) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/set_host2queues?host=#{hostname}&queues=#{queues}'" + system cmd +end + +def del_host2queues(hostname) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/del_host2queues?host=#{hostname}'" + system cmd end def parse_response(url) @@ -67,7 +78,7 @@ def load_initrds(load_path, hash) wget_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") end -def run(hostname, load_path, hash) +def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( @@ -77,24 +88,26 @@ def run(hostname, load_path, hash) clean_dir(load_path) end -def main(hostname) +def main(hostname, queues) + set_host2queues(hostname, queues) url = get_url hostname puts url hash = parse_response url - return if hash.nil? + return del_host2queues(hostname) if hash.nil? load_path = build_load_path(hostname) load_initrds(load_path, hash) - run(hostname, load_path, hash) + start_container(hostname, load_path, hash) + del_host2queues(hostname) end -def loop_main(hostname) +def loop_main(hostname, queues) loop do begin - main(hostname) + main(hostname, queues) rescue StandardError => e puts e.backtrace - # if an exception happend, request the next time after 30 seconds + # if an exception occurs, request the next time after 30 seconds sleep 25 ensure sleep 5 @@ -109,11 +122,11 @@ def save_pid(pids) f.close end -def multi_docker(hostname, nr_container) +def multi_docker(hostname, nr_container, queues) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop_main("#{hostname}-#{i}") + loop_main("#{hostname}-#{i}", queues) end pids << pid end -- 2.23.0

2 2

[PATCH v2 lab-z9] hosts: fix testbox included yaml error
by Wei Jihui 11 Nov '20

11 Nov '20

[Why] submit job with some physical machine's textbox will error: Included yaml file not found: 'hosts/taishan200-2280-2s48p-256g' (RuntimeError) [How] lkp-tests just included files that find $LKP_SRC and testbox's directory, so delete hosts is ok. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a103 | 2 +- hosts/taishan200-2280-2s48p-256g--a104 | 2 +- hosts/taishan200-2280-2s48p-256g--a106 | 2 +- hosts/taishan200-2280-2s48p-256g--a107 | 2 +- hosts/taishan200-2280-2s48p-512g--a105 | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/hosts/taishan200-2280-2s48p-256g--a103 b/hosts/taishan200-2280-2s48p-256g--a103 index 304c6fc..9fcc718 100644 --- a/hosts/taishan200-2280-2s48p-256g--a103 +++ b/hosts/taishan200-2280-2s48p-256g--a103 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bdc69 diff --git a/hosts/taishan200-2280-2s48p-256g--a104 b/hosts/taishan200-2280-2s48p-256g--a104 index a8a2789..41cdbb0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a104 +++ b/hosts/taishan200-2280-2s48p-256g--a104 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8892431 diff --git a/hosts/taishan200-2280-2s48p-256g--a106 b/hosts/taishan200-2280-2s48p-256g--a106 index ad587b3..95b2f05 100644 --- a/hosts/taishan200-2280-2s48p-256g--a106 +++ b/hosts/taishan200-2280-2s48p-256g--a106 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b883bf99 diff --git a/hosts/taishan200-2280-2s48p-256g--a107 b/hosts/taishan200-2280-2s48p-256g--a107 index fba1937..31b8dc6 100644 --- a/hosts/taishan200-2280-2s48p-256g--a107 +++ b/hosts/taishan200-2280-2s48p-256g--a107 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8917c39 diff --git a/hosts/taishan200-2280-2s48p-512g--a105 b/hosts/taishan200-2280-2s48p-512g--a105 index 096b5b0..66a0aca 100644 --- a/hosts/taishan200-2280-2s48p-512g--a105 +++ b/hosts/taishan200-2280-2s48p-512g--a105 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-512g +<<: taishan200-2280-2s48p-512g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b891945d -- 2.23.0

2 1

[PATCH v2 compass-ci] /lib: keep suite result in top for compare output
by Lu Kaiyi 11 Nov '20

11 Nov '20

[why] Currently, the output result of compare default order by alphabet. If there have too many lines in result, it will be difficult for user to find the lines that starts with $suite in metric column. [how] Divide the lines into two parts before print according to whether the line start with $suite or not, then combine output result into a whole to highlight the important info. [example] command: compare "os=openeuler" -d "tbox_group" in z9 one of group compare output result. suites_list:["iso2rootfs", "build-pkg", "deploy-cci"] Before: taishan200-2280-2s64p-256g--a101 vm-hi1620-2p8g--roota15 vm-hi1620-2p8g--roota14 taishan200-2280-2s64p-256g--a6 metric -------------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------ %stddev change %stddev change %stddev change %stddev \ | \ | \ | \ 493.41 ± 108% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.boot 47114.17 ± 109% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.idle 11.57 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.kernel_boot 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time.max 0.00 0 54815.06 ± 124% 0 52113.21 ± 173% 0 0.00 build-pkg.time.file_system_inputs 0.00 0 3730.76 ± 362% 0 9382.32 ± 552% 0 0.00 build-pkg.time.involuntary_context_switches 0.00 0 7.12 ± 365% 0 6.78 ± 379% 0 0.00 build-pkg.time.major_page_faults 0.00 0 35358.13 ± 194% 0 42519.61 ± 208% 0 0.00 build-pkg.time.maximum_resident_set_size 0.00 0 56001.11 ± 360% 0 160421.51 ± 553% 0 0.00 build-pkg.time.minor_page_faults 0.00 0 65536.00 0 65536.00 0 0.00 build-pkg.time.page_size 0.00 0 7.58 ± 755% 0 40.12 ± 903% 0 0.00 build-pkg.time.user_time 0.00 0 16269.71 ± 318% 0 29779.86 ± 394% 0 0.00 build-pkg.time.voluntary_context_switches 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time.max ... 5579.67 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU95.IPI1:Function_call_interrupts 3152478.83 ± 215% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.IPI0:Rescheduling_interrupts 570272.33 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.IPI1:Function_call_interrupts 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time.max 73066.67 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.file_system_inputs 124.50 ± 217% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.involuntary_context_switches 4.33 ± 172% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.major_page_faults 34816.00 ± 140% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.maximum_resident_set_size 23354.67 ± 202% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.minor_page_faults 65536.00 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.page_size 9.50 ± 68% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.percent_of_cpu_this_job_got 4.27 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.system_time 5.20 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.user_time 172904.17 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.voluntary_context_switches 2651512.00 ± 157% -61.9% 1011304.90 ± 25% -60.9% 1037595.96 ± 25% +582.1% 18086008.00 meminfo.Active 2253381.67 ± 176% -69.2% 694564.75 ± 35% -68.1% 718592.59 ± 35% +668.1% 17308404.00 meminfo.Active(anon) 466201.00 ± 223% -100.0% 0.00 -100.0% 0.00 +2509.3% 12164514.00 meminfo.AnonHugePages 1328043.33 ± 148% -50.7% 655286.32 ± 36% -49.3% 673334.38 ± 34% +966.0% 14157259.00 meminfo.AnonPages 0.00 0 524032.00 0 524032.00 0 520576.00 meminfo.CmaFree ... After: taishan200-2280-2s64p-256g--a101 vm-hi1620-2p8g--roota15 vm-hi1620-2p8g--roota14 taishan200-2280-2s64p-256g--a6 metric -------------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------ %stddev change %stddev change %stddev change %stddev \ | \ | \ | \ 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time.max 0.00 0 54815.06 ± 124% 0 52113.21 ± 173% 0 0.00 build-pkg.time.file_system_inputs 0.00 0 3730.76 ± 362% 0 9382.32 ± 552% 0 0.00 build-pkg.time.involuntary_context_switches 0.00 0 7.12 ± 365% 0 6.78 ± 379% 0 0.00 build-pkg.time.major_page_faults 0.00 0 35358.13 ± 194% 0 42519.61 ± 208% 0 0.00 build-pkg.time.maximum_resident_set_size 0.00 0 56001.11 ± 360% 0 160421.51 ± 553% 0 0.00 build-pkg.time.minor_page_faults 0.00 0 65536.00 0 65536.00 0 0.00 build-pkg.time.page_size 0.00 0 7.58 ± 755% 0 40.12 ± 903% 0 0.00 build-pkg.time.user_time 0.00 0 16269.71 ± 318% 0 29779.86 ± 394% 0 0.00 build-pkg.time.voluntary_context_switches 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time.max 0.00 0 0.00 0 0.00 0 1.00 deploy-cci.time.exit_status 0.00 0 0.00 0 0.00 0 503552.00 deploy-cci.time.file_system_inputs 0.00 0 0.00 0 0.00 0 3925.00 deploy-cci.time.involuntary_context_switches 0.00 0 0.00 0 0.00 0 28.00 deploy-cci.time.major_page_faults 0.00 0 0.00 0 0.00 0 271168.00 deploy-cci.time.maximum_resident_set_size 0.00 0 0.00 0 0.00 0 2280195.00 deploy-cci.time.minor_page_faults 0.00 0 0.00 0 0.00 0 65536.00 deploy-cci.time.page_size 0.00 0 0.00 0 0.00 0 3.00 deploy-cci.time.percent_of_cpu_this_job_got 0.00 0 0.00 0 0.00 0 17.61 deploy-cci.time.system_time 0.00 0 0.00 0 0.00 0 176.09 deploy-cci.time.user_time 0.00 0 0.00 0 0.00 0 384342.00 deploy-cci.time.voluntary_context_switches 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time.max 73066.67 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.file_system_inputs 124.50 ± 217% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.involuntary_context_switches 4.33 ± 172% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.major_page_faults 34816.00 ± 140% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.maximum_resident_set_size 23354.67 ± 202% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.minor_page_faults 65536.00 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.page_size 9.50 ± 68% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.percent_of_cpu_this_job_got 4.27 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.system_time 5.20 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.user_time 172904.17 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.voluntary_context_switches 493.41 ± 108% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.boot 47114.17 ± 109% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.idle 11.57 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.kernel_boot 1073.50 ± 217% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 1003392.50 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 612284.17 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.4:GICv3.26.Level.arch_timer 8151.83 ± 213% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU0.4:GICv3.26.Level.arch_timer 165405.33 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU0.IPI0:Rescheduling_interrupts 5886.83 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU0.IPI1:Function_call_interrupts 12477.50 ± 214% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU1.4:GICv3.26.Level.arch_timer ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 15 +++++++++------ lib/compare_matrixes.rb | 33 ++++++++++++++++++++++----------- lib/matrix2.rb | 15 +++++++++++---- 3 files changed, 42 insertions(+), 21 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..3db5169 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suites_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suites_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,15 @@ end def create_matrices_list(conditions) matrices_list = [] + suites_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suites_list.concat(suites) end - matrices_list + return matrices_list, suites_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +61,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..c1cee10 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suites_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suites_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -807,26 +808,36 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suites_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suites_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suites_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suites_list, matrixes_titles, success, theme) return '' if values.empty? + suites_list.uniq! result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suites_list.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end + + result_str += suite_str + common_str result_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..479d280 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,20 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end groups.delete(group_key) if value.size < 2 + suites_list << suite_list unless value.size < 2 end - groups + + return groups, suites_list end # input: @@ -139,7 +146,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], = create_matrix(jobs) end end -- 2.23.0

3 2

[PATCH v5 compass-ci 12/12] mail-robot: start file
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..eab1f12 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +defaults['CCI_SRC'] = '/c/compass-ci' +defaults['MAILDIR'] = '/srv/cci/Maildir/.compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'run-mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v5 compass-ci 11/12] mail-robot: build file
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v5 compass-ci 10/12] mail-robot: Dockerfile
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v5 compass-ci 09/12] mail-robot: documentation for service
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..2eb0dd1 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. assign-account + init my_info + - my_email + - my_name + send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

1 0

[PATCH v5 compass-ci 08/12] mail-robot: build fail message
by Luan Shengde 11 Nov '20

11 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-fail-eamil.rb | 38 ++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 lib/assign-account-fail-eamil.rb diff --git a/lib/assign-account-fail-eamil.rb b/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..1c2133c --- /dev/null +++ b/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following error: + + #{message} + + In order to successfully apply an account, please pay attention to the following points: + + 1. mail subject + the subject should exactly: apply account + + 2. commit url + when you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + ensure you commit url exist and available to access. + + 3. ssh pubkey + you need to add a pubkey as an attachment with name 'id_rsa.pub' to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v5 compass-ci 06/12] mail-robot: apply account for user
by Luan Shengde 11 Nov '20

11 Nov '20

ApplyJumperAccount apply jumper account for user required data: my_info: - my_email - my_name - my_uuid my_ssh_pubkey usage: reference assign-account.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/apply-jumper-account.rb | 40 +++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 lib/apply-jumper-account.rb diff --git a/lib/apply-jumper-account.rb b/lib/apply-jumper-account.rb new file mode 100755 index 0000000..9d07abe --- /dev/null +++ b/lib/apply-jumper-account.rb @@ -0,0 +1,40 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +# apply jumper account for user +# apply_jumper_account +# apply jumper account with my_info +# check account exists +class ApplyJumperAccount + def initialize(my_info, my_ssh_pubkey) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + @my_ssh_pubkey = my_ssh_pubkey + end + + def apply_jumper_account + @my_info['my_ssh_pubkey'] = @my_ssh_pubkey + + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = 'No more available jumper account.' + error_message += 'You may try again later or consulting the manager for a solution.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 03/12] mail-robot: check to assign account
by Luan Shengde 11 Nov '20

11 Nov '20

AssignAccount assign account for user required data: mail_content - send_account parse email_content parse commit url parse pubkey apply_my_account generate uuid apply account store account info to es send email send success email send fail email usage: reference mail-robot.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account.rb | 78 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 lib/assign-account.rb diff --git a/lib/assign-account.rb b/lib/assign-account.rb new file mode 100755 index 0000000..6efeee0 --- /dev/null +++ b/lib/assign-account.rb @@ -0,0 +1,78 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'es_client' + +# assign uuid/account for user +# send_account +# parse my_commit_url and my_ssh_pubkey +# apply account +# store account info +# send mail +class AssignAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def send_account + my_commit_url, my_ssh_pubkey = parse_commit_url_pub_key + acct_info = apply_my_account(my_ssh_pubkey) + + @my_info['my_commit_url'] = my_commit_url + @my_info['my_login_name'] = acct_info['my_login_name'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + my_commit_url = parse_apply_account_email.parse_commit_url + my_ssh_pubkey = parse_apply_account_email.parse_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def apply_my_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info, my_ssh_pubkey) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(error_message) + if error_message.empty? + email_message = build_apply_account_email(@my_info) + else + email_message = build_apply_account_fail_email(@my_info, error_message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 02/12] mail-robot: listen and handle new email
by Luan Shengde 11 Nov '20

11 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..0ae80f3 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.send_account +end -- 2.23.0

1 0

[PATCH v4 compass-ci 08/12] mail-robot: build fail message
by Luan Shengde 11 Nov '20

11 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-fail-eamil.rb | 38 ++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 lib/assign-account-fail-eamil.rb diff --git a/lib/assign-account-fail-eamil.rb b/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..d15cd69 --- /dev/null +++ b/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following errors + + #{message} + + In order to successfully apply a account, pay attention to the following points: + + 1. mail subject + the subject should exactly: apply account + + 2. commit url + when you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + ensure you commit url exist and available to access. + + 3. ssh pubkey + you need to add a pubkey as an attachment with name 'id_rsa.pub' to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

2 2

[PATCH v4 compass-ci 05/12] mail-robot: check commit url for gitee
by Luan Shengde 11 Nov '20

11 Nov '20

GiteeCommitUrlCheck check commit url availability for hub gitee - clone the repo - check commit log for the commit usage: reference parse-apply-account-email.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/gitee-commit-url-check.rb | 44 +++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100755 lib/gitee-commit-url-check.rb diff --git a/lib/gitee-commit-url-check.rb b/lib/gitee-commit-url-check.rb new file mode 100755 index 0000000..11fbb6d --- /dev/null +++ b/lib/gitee-commit-url-check.rb @@ -0,0 +1,44 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check commit url availability for hub gitee.com +# gitee_commit_index +# clone the repo +# check commit available +class GiteeCommitUrlCheck + def initialize(my_info, url, base_url) + @my_info = my_info + @url = url + @base_url = base_url + end + + def gitee_commit_index + repo_dir = @url.split('/')[-3] + repo_url = [@base_url, 'git'].join('.') + commit_id = @url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] + + FileUtils.rm_rf repo_dir + + gitee_commit_exist(email_index) + end + + def gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = "We can not conform the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH v3 compass-ci] container: fix failed to build kibana images
by Liu Yinsi 11 Nov '20

11 Nov '20

[why] when build kibana images in x86_machine, error message: [root@localhost kibana]# ./build Sending build context to Docker daemon 5.12kB Step 1/3 : FROM gagara/kibana-oss-arm64:7.6.2 7.6.2: Pulling from gagara/kibana-oss-arm64 38163f410fa0: Pull complete 69a4d016f221: Pull complete 95e6c6e7c9ca: Pull complete d13f429dd982: Pull complete 508bb3330fb2: Pull complete 9634e726f1b6: Pull complete 9c26c37850c8: Pull complete 0d0ad8467060: Pull complete 940f92726f8b: Pull complete Digest: sha256:541632b7e9780a007f8a8be82ac8853ddcebcb04a596c00500b73f77eacfbd16 Status: Downloaded newer image for gagara/kibana-oss-arm64:7.6.2 ---> f482a0472f78 Step 2/3 : MAINTAINER Wu Zhende <wuzhende666(a)163.com> ---> Running in cfa86d8ce976 Removing intermediate container cfa86d8ce976 ---> 3be6c5f24d4b Step 3/3 : RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml ---> Running in ff455f66df8b standard_init_linux.go:220: exec user process caused "exec format error" libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error" The command '/bin/sh -c sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/kibana/Dockerfile | 4 +++- container/kibana/build | 8 +++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 35802fe..6e0dba0 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -1,7 +1,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM gagara/kibana-oss-arm64:7.6.2 +ARG BASE_IMAGE + +FROM BASE_IMAGE # docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 diff --git a/container/kibana/build b/container/kibana/build index a7e4717..52d5a2a 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -3,4 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -system 'docker build -t kibana:7.6.2 .' +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', + 'x86_64' => 'kibana:7.6.2' }.freeze + +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] + +system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." -- 2.23.0

2 2

[PATCH v4 compass-ci 04/12] mail-robot: parse apply account email
by Luan Shengde 11 Nov '20

11 Nov '20

ParseApplyAccountEmail parse commit url and pubkey parse commit url check commit url exists check base url in upstream-repos check commit available hub: gitee.com call gitee-commit-url-check to check commit available hub: non-gitee.com check the feedback of curl command to check the commit url available parse pubkey attachment file name: id_rsa.pub usage: reference assign-account.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/parse-apply-account-email.rb | 115 +++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100755 lib/parse-apply-account-email.rb diff --git a/lib/parse-apply-account-email.rb b/lib/parse-apply-account-email.rb new file mode 100755 index 0000000..98d4b9b --- /dev/null +++ b/lib/parse-apply-account-email.rb @@ -0,0 +1,115 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# parse mail_content +# parse_commit_url +# extract_commit_url +# check commit url exists +# base_url_in_upstream_repos +# check base url in upstream-repos +# commit_url_availability +# check commit available +# parse pub_key +# check pub_key exists +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + + # the commit url should be headed with a prefix: my oss commit + # the commit url must be in a standart format, example: + # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = "No matched commit url found.\n" + error_message += "Ensure that you have add a right commit url with prefix 'my oss commit:'." + + raise error_message + end + + def parse_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + base_url_in_upstream_repos('/c/upstream-repos', base_url) + commit_url_availability(url, base_url) + + return url + end + + def base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = "The repo url for your commit is not in our upstream-repo list.\n" + error_message += 'Use a new one, or consulting the manager for available repo list.' + + raise error_message + end + + def commit_url_availability(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + gitee_commit(url, base_url) + else + non_gitee_commit(url) + end + end + + def gitee_commit(url, base_url) + my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) + my_gitee_commit.gitee_commit_index + end + + def non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_info['my_email'] + + return unless email_index.nil? + + error_message = "We can not conform the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end + + def parse_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + pub_key_exist(pub_key) + + return pub_key + end + + def pub_key_exist(pub_key) + return unless pub_key.nil? + + error_message = "No pub_key found.\n" + error_message += 'Please add the pub_key as an attachment with filename: id_rsa.pub.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH v4 compass-ci 06/12] mail-robot: apply account for user
by Luan Shengde 11 Nov '20

11 Nov '20

ApplyJumperAccount apply jumper account for user required data: my_info: - my_email - my_name - my_uuid my_ssh_pubkey usage: reference assign-account.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/apply-jumper-account.rb | 40 +++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 lib/apply-jumper-account.rb diff --git a/lib/apply-jumper-account.rb b/lib/apply-jumper-account.rb new file mode 100755 index 0000000..a34e147 --- /dev/null +++ b/lib/apply-jumper-account.rb @@ -0,0 +1,40 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +# apply jumper account for user +# apply_jumper_account +# apply jumper account with my_info +# check account exists +class ApplyJumperAccount + def initialize(my_info, my_ssh_pubkey) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + @my_ssh_pubkey = my_ssh_pubkey + end + + def apply_jumper_account + @my_info['my_ssh_pubkey'] = @my_ssh_pubkey + + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = ' No more available jumper account.' + error_message += ' You may try again later or consulting the managet for a solution.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH compass-ci] sbin/compare optimize help message
by Lu Weitao 11 Nov '20

11 Nov '20

[why] compare tool support output result with json | html format, but haven't corresponding direction in help message. [how] add the corresponding direction in --theme option [example] compare id=crystal.109482 id=crystal.109483 --theme json Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/compare | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sbin/compare b/sbin/compare index b13682e..baa90d4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -12,7 +12,7 @@ # compare "commit=a12d232e" "commit=b3bacc31" # compare "os=debian" "os=centos" -c "suite=iperf" # compare "os=centos" -d "os_version os_arch" -# compare "os=centos" -d "os_version os_arch" --color "classic" +# compare "os=centos" -d "os_version os_arch" --theme "classic" require 'optparse' require_relative '../lib/compare.rb' @@ -42,9 +42,9 @@ opt_parser = OptionParser.new do |opts| is_group = true end - opts.on('--color color', 'turn on colorful display with theme: classic|focus_good|focus_bad', - '|striking|light|none') do |color| - colorful = color + opts.on('--theme theme', 'turn on colorful display with theme: classic|focus_good|focus_bad', + '|striking|light|json|html') do |theme| + colorful = theme end opts.on('-t', '--template template', 'compare with user-defined template') do |t| -- 2.23.0

2 2

[PATCH v2 lkp-tests] jobs: add multi-device test parameters
by Zhang Yu 11 Nov '20

11 Nov '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- jobs/sysbench-mysql.yaml | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml index c060fc2a..83795a18 100644 --- a/jobs/sysbench-mysql.yaml +++ b/jobs/sysbench-mysql.yaml @@ -1,10 +1,22 @@ suite: sysbench-mysql category: benchmark -sysbench-mysql: - oltp_test_mode: complex - oltp_tables_count: 3 - oltp_table_size: 1000 - nr_threads: 64 - runtime: 120 - report_interval: 10 +cluster: cs-lkp-hsw-ep5 + +if role server: + mysql: + +if role client: + sysbench-mysql: + oltp_test_mode: complex + max_requests: 0 + mysql_table_engine: innodb + oltp_tables_count: 1000 + oltp_table_size: 100000 + nr_threads: 256 + runtime: 600 + rand_type: special + rand_spec_pct: 100 + oltp_read_only: off + report_interval: 1 + events: 2000000000 -- 2.23.0

1 0

[PATCH v7 lkp-tests 1/2] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 11 Nov '20

11 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=1.1 BUILDFILE -> PKGBUILD BUILDFILE-$PKGBUILD_TAG -> PKGBUILD-1.1 PKGBUILD-1.1 contains only the changed part. % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..fde82212 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -889,6 +889,7 @@ update_pkgver() { fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" source_safe "$BUILDFILE" + [[ -n $PKGBUILD_TAG ]] && source_safe "$BUILDFILE-$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -3692,6 +3693,7 @@ else BUILDFILE="$startdir/$BUILDFILE" fi source_buildfile "$BUILDFILE" + [[ -n $PKGBUILD_TAG ]] && source_buildfile "$BUILDFILE-$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

1 0

[PATCH v7 lkp-tests 2/2] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 11 Nov '20

11 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git from /srv/git/pkg/ usage: job yaml: PKGBUILD_TAG: 1.0-20 About $PKGBUILD_TAG, it is from PKGBUILD file's $pkgver-$pkgrel. You should create a git repo that name same as $benchmark in /srv/git/pkg/ before you add PKGBUILD_TAG to yaml. If not, cci-makepkg will report an error. And everything same as before if no PKGBUILD_TAG. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..67d055a4 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -39,12 +39,29 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +download_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +get_pkgfile() +{ + PKGBUILD_TAG_FILE="PKGBUILD-$PKGBUILD_TAG" + + download_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 pkg_dir=${2:-.} ( . $pkg_dir/PKGBUILD + [ -n "$PKGBUILD_TAG" ] && . $pkg_dir/PKGBUILD-$PKGBUILD_TAG eval echo '$'$var ) } @@ -71,6 +88,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && get_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

1 1

[PATCH v7 lkp-tests 1/2] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 11 Nov '20

11 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 PKGBUILD-1.1 contains only the changed part. % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..fde82212 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -889,6 +889,7 @@ update_pkgver() { fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" source_safe "$BUILDFILE" + [[ -n $PKGBUILD_TAG ]] && source_safe "$BUILDFILE-$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -3692,6 +3693,7 @@ else BUILDFILE="$startdir/$BUILDFILE" fi source_buildfile "$BUILDFILE" + [[ -n $PKGBUILD_TAG ]] && source_buildfile "$BUILDFILE-$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

1 1

[PATCH compass-ci 3/4] assign-account: generate rsa keys
by Luan Shengde 11 Nov '20

11 Nov '20

generate rsa private/public key and return public key on jumper server [why] register the public key on testbox enable user one click to login in the testbox [how] generate rsa private/public key for user Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 32 ++++++++++++-------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 137f5c6..b5a2388 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -94,30 +94,36 @@ class AccountStorage end def setup_jumper_account_info - account_info = read_account_info - jumper_info = read_jumper_info + login_name, password = read_account_info + jumper_host, jumper_port = read_jumper_info pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? - login_name = account_info[0] - password = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + password = 'Use pub_key to login' unless pub_key.nil? + + config_authorized_key(login_name, pub_key) unless pub_key.nil? + config_default_yaml(login_name) + my_jumper_pubkey = generate_ssh_key(login_name) if @data['my_jumper_pubkey'].eql? true jumper_account_info = { 'my_login_name' => login_name, 'my_password' => password, - 'jumper_host' => jumper_info[0].chomp, - 'jumper_port' => jumper_info[1].chomp + 'jumper_host' => jumper_host, + 'jumper_port' => jumper_port, + 'my_jumper_pubkey' => my_jumper_pubkey } - config_authorized_key(login_name, pub_key) unless pub_key.nil? - config_default_yaml(login_name) - return jumper_account_info end + def generate_ssh_key(login_name) + ssh_dir = File.join('/home/', login_name, '.ssh') + Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + hostname = File.read('/etc/hostname') + %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@#{hostname}") + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) + File.read("/home/#{login_name}/.ssh/id_rsa.pub") + end + def config_default_yaml(login_name) default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') FileUtils.mkdir_p default_yaml_dir -- 2.23.0

1 0

[PATCH lab-z9] hosts: fix testbox included yaml error [Why] submit job with some physical machine's textbox will error: Included yaml file not found: 'hosts/taishan200-2280-2s48p-256g' (RuntimeError)
by Wei Jihui 11 Nov '20

11 Nov '20

[How] lkp-tests just included files that find $LKP_SRC and testbox's directory, so delete hosts is ok. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a103 | 2 +- hosts/taishan200-2280-2s48p-256g--a104 | 2 +- hosts/taishan200-2280-2s48p-256g--a106 | 2 +- hosts/taishan200-2280-2s48p-256g--a107 | 2 +- hosts/taishan200-2280-2s48p-512g--a105 | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/hosts/taishan200-2280-2s48p-256g--a103 b/hosts/taishan200-2280-2s48p-256g--a103 index 304c6fc..9fcc718 100644 --- a/hosts/taishan200-2280-2s48p-256g--a103 +++ b/hosts/taishan200-2280-2s48p-256g--a103 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bdc69 diff --git a/hosts/taishan200-2280-2s48p-256g--a104 b/hosts/taishan200-2280-2s48p-256g--a104 index a8a2789..41cdbb0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a104 +++ b/hosts/taishan200-2280-2s48p-256g--a104 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8892431 diff --git a/hosts/taishan200-2280-2s48p-256g--a106 b/hosts/taishan200-2280-2s48p-256g--a106 index ad587b3..95b2f05 100644 --- a/hosts/taishan200-2280-2s48p-256g--a106 +++ b/hosts/taishan200-2280-2s48p-256g--a106 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b883bf99 diff --git a/hosts/taishan200-2280-2s48p-256g--a107 b/hosts/taishan200-2280-2s48p-256g--a107 index fba1937..31b8dc6 100644 --- a/hosts/taishan200-2280-2s48p-256g--a107 +++ b/hosts/taishan200-2280-2s48p-256g--a107 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8917c39 diff --git a/hosts/taishan200-2280-2s48p-512g--a105 b/hosts/taishan200-2280-2s48p-512g--a105 index 096b5b0..66a0aca 100644 --- a/hosts/taishan200-2280-2s48p-512g--a105 +++ b/hosts/taishan200-2280-2s48p-512g--a105 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-512g +<<: taishan200-2280-2s48p-512g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b891945d -- 2.23.0

2 1

[PATCH v4 lkp-tests] job2sh.rb: define on_fail
by Hu Xuejiao 11 Nov '20

11 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh export_top_envexport_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7e6a5d7c..370c7899 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -358,6 +360,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

3 2

[PATCH v4 compass-ci 09/12] mail-robot: documentation for service
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..f829199 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. assign-account + init my_info + - my_email + - my_name + send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

2 2

[PATCH v6 lkp-tests 1/2] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 11 Nov '20

11 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 PKGBUILD-1.1 contains only the changed part. % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..fbe9c432 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -888,7 +888,7 @@ update_pkgver() { exit 1 fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" - source_safe "$BUILDFILE" + source_safe "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -1604,10 +1604,14 @@ cd_safe() { source_safe() { shopt -u extglob - if ! source "$@"; then - error "$(gettext "Failed to source %s")" "$1" - exit 1 - fi + local file + for file in "$@" + do + if ! source "$file"; then + error "$(gettext "Failed to source %s")" "$file" + exit 1 + fi + done shopt -s extglob } @@ -1626,7 +1630,6 @@ merge_arch_attrs() { source_buildfile() { source_safe "$@" - if (( !SOURCEONLY )); then merge_arch_attrs fi @@ -3661,6 +3664,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +PKGBUILD_TAG=$PKGBUILD_TAG if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then @@ -3691,7 +3695,7 @@ else if [[ ${BUILDFILE:0:1} != "/" ]]; then BUILDFILE="$startdir/$BUILDFILE" fi - source_buildfile "$BUILDFILE" + source_buildfile "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

3 4

[PATCH lkp-tests] depends: add depends to sysbench-mysql
by Zhang Yu 11 Nov '20

11 Nov '20

Use source code install mysql, so no need mysql-server here. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/sysbench-mysql | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/distro/depends/sysbench-mysql b/distro/depends/sysbench-mysql index e9585216..cfbc4518 100644 --- a/distro/depends/sysbench-mysql +++ b/distro/depends/sysbench-mysql @@ -1,2 +1,2 @@ -mysql-server -libmysqlclient-dev +libnuma1 +xfsprogs -- 2.23.0

1 0

[PATCH v1 compass-ci] splitting the cifs services
by Xiao Shenwei 11 Nov '20

11 Nov '20

[why] our os-cifs container will mount three dir, /srv/os /srv/initrd /srv/result it's may cause: 1. our cifs service becomes slow 2. if the container exit, none of the three folders can be accessed. in order to reduce the coupling, need split them [how] one folder mount by one container usage: if you want mount these floder, you can use these command server: /srv/os client: mount -t cifs -o guest //ip/os /tmp/os server: /srv/initrd client: mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd server: /srv/result client: mount -t cifs -o guest,port=447 //ip/result /tmp/result --- container/os-cifs/start | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/container/os-cifs/start b/container/os-cifs/start index 6cc08c6..e20f887 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -8,18 +8,40 @@ lsmod | grep -q "^cifs\s" || { sudo modprobe cifs } -docker_rm samba +docker_rm os-cifs +docker_rm initrd-cifs +docker_rm result-cifs -cmd=( +os_cifs_cmd=( docker run -dt -p 445:445 + -v /etc/localtime:/etc/localtime:ro -v /srv/os:/srv/os + --name os-cifs + --restart=always + alpine/samba +) + +initrd_cifs_cmd=( + docker run -dt + -p 446:445 -v /etc/localtime:/etc/localtime:ro -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + alpine/samba +) + +result_cifs_cmd=( + docker run -dt + -p 447:445 + -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result - --name os-cifs + -name result-cifs --restart=always alpine/samba ) -"${cmd[@]}" +"${os_cifs_cmd[@]}" +"${initrd_cifs_cmd[@]}" +"${result_cifs_cmd[@]}" -- 2.23.0

2 2

[PATCH lkp-tests] job2sh: remove atomic job test
by Wu Fengguang 11 Nov '20

11 Nov '20

Make it easier to do simple demo. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/job2sh | 6 ------ sbin/job2sh.cr | 6 ------ 2 files changed, 12 deletions(-) diff --git a/sbin/job2sh b/sbin/job2sh index 4fb57453..9ada3cfa 100755 --- a/sbin/job2sh +++ b/sbin/job2sh @@ -38,12 +38,6 @@ begin job = Job2sh.new job.load(ARGV[0]) job.expand_params - - unless job.atomic_job? - log_error "Looks #{ARGV[0]} isn't a atomic jobfile, only atomic jobfile is supported" - log_error "Please run 'lkp split-job #{ARGV[0]}' first" - exit 1 - end rescue Job::ParamError => e log_error "Abandon job: #{e.message}" exit 1 diff --git a/sbin/job2sh.cr b/sbin/job2sh.cr index ffcd3359..3a921bd6 100644 --- a/sbin/job2sh.cr +++ b/sbin/job2sh.cr @@ -28,12 +28,6 @@ begin job = Job2sh.new job.load(ARGV[0]) job.expand_params - - unless job.atomic_job? - log_error "Looks #{ARGV[0]} isn't a atomic jobfile, only atomic jobfile is supported" - log_error "Please run 'lkp split-job #{ARGV[0]}' first" - exit(1) - end rescue e : Job::ParamError log_error "Abandon job: #{e.message}" exit(1) -- 2.23.0

1 0

[PATCH lkp-tests] tests: fix lmbench3 tool adapt to aarch64 framework
by Bai Jing 11 Nov '20

11 Nov '20

error: cat: results/x84_64-linux-gnu/taishan200-2280-2s48p-256g--a28.0 No such file or directory Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/lmbench3 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/lmbench3 b/tests/lmbench3 index 8570a35a..dc4d157d 100755 --- a/tests/lmbench3 +++ b/tests/lmbench3 @@ -138,7 +138,7 @@ run_lmbench3_development() eval "$test=yes" - sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/x86_64-linux-gnu/lmbench + sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/"${os_arch}"-linux-gnu/lmbench log_eval " ( echo $nr_threads @@ -177,7 +177,7 @@ run_lmbench3_development() show_result() { - cat results/x86_64-linux-gnu/$testbox.0 || { + cat results/"${os_arch}"-linux-gnu/$testbox.0 || { echo "no test result found" exit -1 } -- 2.23.0

1 0

[PATCH v4 compass-ci] container: fix failed to build archlinux images in x86_64 machine
by Liu Yinsi 11 Nov '20

11 Nov '20

[why] when build archlinux images in x86_64 machine error: error: failed retrieving file 'core.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'core.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update core (failed to retrieve some files) error: failed retrieving file 'extra.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'extra.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Resolving timed out after 10000 milliseconds error: failed to update extra (download library error) error: failed retrieving file 'community.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : The requested URL returned error:404 error: failed to update community (failed to retrieve some files) error: failed to synchronize all databases The command '/bin/sh -c pacman --needed --noprogressbar --noconfirm -Syu && pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk' returned a non-zero code: 1 because archlinux mirror not support x86_64 machine. [how] root/etc/pacman.d/mirrorlist => root/aarch64/etc/pacman.d/mirrorlist root/x86_64/etc/pacman.d/mirrorlist use ARCH=$(arch) to choose mirrorlist according to different system architecture. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/archlinux/Dockerfile | 6 +++++- container/archlinux/build | 2 +- .../archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist | 0 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist | 1 + 4 files changed, 7 insertions(+), 2 deletions(-) rename container/archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist (100%) create mode 100644 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/Dockerfile b/container/archlinux/Dockerfile index c0f05d3..1f80ae0 100644 --- a/container/archlinux/Dockerfile +++ b/container/archlinux/Dockerfile @@ -5,7 +5,11 @@ FROM lopsided/archlinux MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> -COPY root / +ARG ARCH + +COPY root/$ARCH / + RUN chmod 755 /etc + RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk diff --git a/container/archlinux/build b/container/archlinux/build index 81feda2..9749489 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,4 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t archlinux:testbed . +docker build --build-arg ARCH=$(arch) -t archlinux:testbed . diff --git a/container/archlinux/root/etc/pacman.d/mirrorlist b/container/archlinux/root/aarch64/etc/pacman.d/mirrorlist similarity index 100% rename from container/archlinux/root/etc/pacman.d/mirrorlist rename to container/archlinux/root/aarch64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist new file mode 100644 index 0000000..556fac8 --- /dev/null +++ b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist @@ -0,0 +1 @@ +Server = http://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch -- 2.23.0

1 0

[PATCH compass-ci] container: keep docker_rm '$containername' consistent with name
by Liu Yinsi 11 Nov '20

11 Nov '20

because --name=os-cifs, docker_rm os-cifs should be the same. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/os-cifs/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/os-cifs/start b/container/os-cifs/start index 6cc08c6..1b2cd23 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -8,7 +8,7 @@ lsmod | grep -q "^cifs\s" || { sudo modprobe cifs } -docker_rm samba +docker_rm os-cifs cmd=( docker run -dt -- 2.23.0

1 0

[PATCH v4 lkp-tests] submit on_fail to the scheduler
by Hu Xuejiao 11 Nov '20

11 Nov '20

[why] When job2sh.rb defined sh_on_fail, it needs to be invoked by the scheduler. Finally, the task is in the waiting state when the task is failed. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..0bb41f2d 100755 --- a/sbin/submit +++ b/sbin/submit @@ -115,12 +115,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_fail sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

2 1

[PATCH v4 lkp-tests] execute on_fail when exit-code isn't 0
by Hu Xuejiao 11 Nov '20

11 Nov '20

[why] when a job-task fails(exit_code != 0), the on_fail function needs to be called to keep the program in the waiting state to user adjustment Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- bin/run-lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/run-lkp b/bin/run-lkp index 26669331..c4d43db4 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -97,6 +97,7 @@ elif [ "$run_job_exit_code" != 0 ]; then else set_job_state 'failed' fi + $LKP_DEBUG_PREFIX $job_script on_fail elif [ -f '/var/log/wtmp' ] && has_cmd last && last -n1 | grep -q -E '(pts/|tty)[0-9].*(still logged in)'; then set_job_state 'disturbed' disturbed=1 -- 2.23.0

1 0

[PATCH compass-ci] git_mirror: fix that worker thread die after system(git clone)
by Li Yuanchao 11 Nov '20

11 Nov '20

[why] As Signal.trap(:SIGCHLD, 'SIG_IGN') has been set, the init process will handle the return status of child process, so ret = system(git clone xxx) can't get return value, which cause worker thread die. Use %x() instead of system(), and use stderr to check if 'git clone' succeed. So return value is not needed. [errmsg] in `system': No child process - Another thread waited the process started by system(). (Errno::ECHILD) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 6e6664a..99534f6 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -38,7 +38,8 @@ class GitMirror url = "/srv/git/#{url.delete_prefix('https://')}" end 10.times do - ret = system("git clone --mirror #{url} #{mirror_dir}") + stderr = %x(git clone --mirror #{url} #{mirror_dir} 2>&1) + ret = !stderr.include?('fatal') break if ret end FileUtils.rm_r(mirror_dir) unless ret -- 2.23.0

1 0

[PATCH v4 lkp-tests] job2sh.rb: define on_fail
by Hu Xuejiao 11 Nov '20

11 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh export_top_envexport_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7e6a5d7c..370c7899 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -358,6 +360,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

2 1

[PATCH v2 compass-ci] job.cr: obtain the cgz package of a specified version
by Zhang Yu 11 Nov '20

11 Nov '20

[why] We need to use specified version of performance tool to test. But the tool package uses latest.cgz by default. [how] Job yaml: ${program}_version: 1.0-19 About ${program}_version's value, it is from PKGBUILD file's $pkgver-$pkgrel. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- src/lib/job.cr | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 07873ab..d10a1e9 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -424,8 +424,14 @@ class Job program = $1 end + if @hash["#{program}_version"]? + program_version = @hash["#{program}_version"] + else + program_version = "latest" + end + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/latest.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" if File.exists?(deps_dest_file) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) -- 2.23.0

1 0

[PATCH lkp-tests] sbin/makepkg: download submodules code from LKP server
by Liu Shaofei 10 Nov '20

10 Nov '20

[why] Some repository have submodules. So when build packages, we need download submodules from LKP server, avoid the waste of time in downloading files from the remote end. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..afa665aa 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -623,6 +623,16 @@ extract_git() { cd_safe "${dir##*/}" + # modify submodules address to local server address + # url = https://github.com/nico/demumble.git + # => + # url = git://172.168.131.113/submodules/b/bloaty/demumble.git + if [[ -f ".gitmodules" ]]; then + local project=${upstream_repo%/*} + local url="git://${LKP_SERVER}/submodules/${project}/" + sed -i "s|url = .*\/|${url}|g" ".gitmodules" + fi + local ref=origin/HEAD if [[ -n $fragment ]]; then case ${fragment%%=*} in -- 2.23.0

2 2

[PATCH v4 compass-ci 12/12] mail-robot: start file
by Luan Shengde 10 Nov '20

10 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..035d4bf --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +defaults['CCI_SRC'] = '/c/compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v /srv/cci/Maildir/.compass-ci:/srv/cci/Maildir/.compass-ci:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'run-mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v4 compass-ci 11/12] mail-robot: build file
by Luan Shengde 10 Nov '20

10 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v4 compass-ci 10/12] mail-robot: Dockerfile
by Luan Shengde 10 Nov '20

10 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v4 compass-ci 07/12] mail-robot: build success email
by Luan Shengde 10 Nov '20

10 Nov '20

build email message for successfully assigning account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-email.rb | 53 +++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 lib/assign-account-email.rb diff --git a/lib/assign-account-email.rb b/lib/assign-account-email.rb new file mode 100755 index 0000000..2bed2c1 --- /dev/null +++ b/lib/assign-account-email.rb @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_email(my_info) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{my_info['my_name']}, + + Thank you for joining us. + + You can use the following info to submit jobs: + + notice: + the followings are ONE-TIME setup + + 1) setup default config + run the following command to add the below setup to default config file + + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF + my_email: #{my_info['my_email']} + my_name: #{my_info['my_name']} + my_uuid: #{my_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + + reference to 'how to write job yaml' section to write the job yaml + you can alse reference to files in compass-ci/jobs as example. + + submit jobs, for example: + + submit jobs/iperf.yaml textbox=vm-2p8g + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v4 compass-ci 02/12] mail-robot: listen and handle new email
by Luan Shengde 10 Nov '20

10 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..0ae80f3 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.send_account +end -- 2.23.0

1 0

[PATCH v4 compass-ci 01/12] mail-robot: run mail-robot service
by Luan Shengde 10 Nov '20

10 Nov '20

call mail-robot.rb parse-apply-account-email.rb gitee-commit-url-check.rb assign-account.rb apply-jumper-account.rb assign-account-email.rb assign-account-fail-eamil.rb to run the email-robot service Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/run-mail-robot.rb | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 container/mail-robot/run-mail-robot.rb diff --git a/container/mail-robot/run-mail-robot.rb b/container/mail-robot/run-mail-robot.rb new file mode 100755 index 0000000..d89e8f2 --- /dev/null +++ b/container/mail-robot/run-mail-robot.rb @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account-fail-eamil" +require_relative "#{ENV['CCI_SRC']}/lib/apply-jumper-account" +require_relative "#{ENV['CCI_SRC']}/lib/gitee-commit-url-check.rb" +require_relative "#{ENV['CCI_SRC']}/lib/parse-apply-account-email" +require_relative "#{ENV['CCI_SRC']}/lib/assign-account" +require_relative "#{ENV['CCI_SRC']}/lib/mail-robot" + +MAILDIR = '/srv/cci/Maildir/.compass-ci' + +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

1 0

[PATCH lkp-tests] depends: add nginx depends
by Wei Jihui 10 Nov '20

10 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/nginx | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 distro/depends/nginx diff --git a/distro/depends/nginx b/distro/depends/nginx new file mode 100644 index 00000000..612d3f4b --- /dev/null +++ b/distro/depends/nginx @@ -0,0 +1,6 @@ +glibc +gperftools-libs +gcc +pcre2-devel +openssl-devel +zlib-devel -- 2.23.0

3 2

[PATCH lkp-tests] lib/monitor: fix json parse error
by Wu Zhende 10 Nov '20

10 Nov '20

/usr/share/ruby/json/common.rb:156:in `parse': 765: unexpected token at 'from: {172.17.0.1:10708} <-- ack: {}' (JSON::ParserError) Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 67c2389a..f65cfccf 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -64,7 +64,7 @@ class Monitor return unless @action['output'] if data['log'] - data = JSON.parse(data['log']) + data = data['log'] elsif data['message'] data = data['message'] end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests: adapt the special version of sysbench test
by Zhang Yu 10 Nov '20

10 Nov '20

[why] Use the special version of sysbench test mysql [how] Add parameters to test script, and use different test file of sysbench Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 89 ++++++++++++++++++++++++++++++++------------ 1 file changed, 66 insertions(+), 23 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 4ab5b453..b8cd5c30 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,52 +1,95 @@ #!/bin/sh -# mysql_user -# mysql_host -# mysql_port -# mysql_db -# db_driver # - oltp_test_mode # - oltp_tables_count # - oltp_table_size +# - max_requests +# - mysql_table_engine # - nr_threads +# - rand_type +# - rand_spec_pct # - runtime # - report_interval +# - oltp_read_only +# - events +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +# db_driver : "${mysql_user:=root}" -: "${mysql_host:=localhost}" -: "${mysql_port:=3306}" -: "${mysql_db:=test_1000}" +: "${mysql_host:=$direct_server_ips}" +: "${mysql_port:=$mysql_port}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=$mysql_password}" : "${db_driver:=mysql}" -: "${oltp_tables_count:=3}" -: "${oltp_table_size:=1000}" -: "${nr_threads:=64}" -: "${runtime:=120}" -: "${report_interval:=10}" +: "${oltp_test_mode:=complex}" +: "${oltp_tables_count:=1000}" +: "${oltp_table_size:=100000}" +: "${max_requests:=0}" +: "${mysql_table_engine:=innodb}" +: "${rand_type:=special}" +: "${rand_spec_pct:=100}" +: "${nr_threads:=256}" +: "${runtime:=600}" +: "${report_interval:=1}" +: "${oltp_read_only:=off}" +: "${events:=2000000000}" -args=( +args1=( --mysql-user=$mysql_user --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db + --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver - --tables=$oltp_tables_count - --table-size=$oltp_table_size + --mysql-password=$mysql_password + --max-requests=$max_requests + --mysql-table-engine=$mysql_table_engine + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --rand-type=$rand_type + --rand-spec-pct=$rand_spec_pct --threads=$nr_threads --time=$runtime - --report-interval=$report_interval ) +args2=( + --mysql-user=$mysql_user + --mysql-password=$mysql_password + --mysql-host=$mysql_host + --mysql-port=$mysql_port + --mysql-db=$mysql_db + --threads=$nr_threads + --oltp-read-only=$oltp_read_only + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --report-interval=$report_interval + --time=$runtime + --events=$events +) +stop_firewalld() +{ + systemctl stop firewalld + iptables -F +} + run_sysbench_step() { - sysbench /usr/share/sysbench/$1 "${args[@]}" $2 + lua_script=$1 + shift + sysbench /usr/local/share/sysbench/tests/include/oltp_legacy/$lua_script "$@" + } run_sysbench_mysql() { - systemctl start mysqld - mysql -e "create database test_1000;" - run_sysbench_step oltp_common.lua prepare - run_sysbench_step oltp_read_write.lua run - run_sysbench_step oltp_common.lua cleanup + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + run_sysbench_step oltp.lua ${args2[@]} run + run_sysbench_step oltp.lua ${args2[@]} cleanup } +stop_firewalld run_sysbench_mysql + -- 2.23.0

2 2

[PATCH compass-ci] /lib: keep suite result in top for compare output
by Lu Kaiyi 10 Nov '20

10 Nov '20

[why] Currently, the output result of compare default order by alphabet. If there have too many lines in result, it will be difficult for user to find the lines that starts with $suite in metric column. [how] Divide the lines into two parts before print according to whether the line start with $suite or not, then combine output result into a whole to highlight the important info. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 15 +++++++++------ lib/compare_matrixes.rb | 35 ++++++++++++++++++++++++----------- lib/matrix2.rb | 12 ++++++++---- 3 files changed, 41 insertions(+), 21 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..ebf35b1 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suites_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suites_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,15 @@ end def create_matrices_list(conditions) matrices_list = [] + suites_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suite = combine_query_data(query_results) + matrices_list << matrix + suites_list.concat(suite) end - matrices_list + return matrices_list, suites_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +61,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..910f02f 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suites_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suites_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list, matrices_titles, k, options: options) end end result_str @@ -807,26 +808,38 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suites_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suites_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suites_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suites_list, matrixes_titles, success, theme) return '' if values.empty? + suites_list.uniq! result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + field_start_with_suite = suites_list.map { |suite| field.start_with?(suite) }.any? + if field_start_with_suite + suite_str += get_values_str(matrixes, success, theme) + suite_str += get_field_str(field) + suite_str += "\n" + else + common_str += get_values_str(matrixes, success, theme) + common_str += get_field_str(field) + common_str += "\n" + end end + + result_str += suite_str + common_str result_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..d6abf1b 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suite = [] job_list.each do |job| + suite << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suite end # input: query results from es_query @@ -110,15 +112,17 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suite = create_matrix(jobs) + suites_list.concat(suite) end groups.delete(group_key) if value.size < 2 end - groups + return groups, suites_list end # input: @@ -139,7 +143,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], = create_matrix(jobs) end end -- 2.23.0

3 3

[PATCH v6 lkp-tests 2/2] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 10 Nov '20

10 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe..c7b4b2df 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -37,12 +37,30 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +download_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +get_pkgfile() +{ + PKGBUILD_TAG="-$PKGBUILD_TAG" + PKGBUILD_TAG_FILE="PKGBUILD$PKGBUILD_TAG" + + download_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 pkg_dir=${2:-.} ( . $pkg_dir/PKGBUILD + . $pkg_dir/PKGBUILD$PKGBUILD_TAG eval echo '$'$var ) } @@ -69,6 +87,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && get_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

2 2

[PATCH v2 compass-ci] git_mirror: pull submodule if repositories have
by Li Yuanchao 10 Nov '20

10 Nov '20

[why] Some repositories have submodules. Without these submodules, they can not be built successfully. Submodules of repositories are repositories too. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9d6a45d..6e6664a 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -56,7 +56,8 @@ class GitMirror def mirror_sync fork_info = @queue.pop - mirror_dir = "/srv/git/#{fork_info['git_repo']}.git" + mirror_dir = "/srv/git/#{fork_info['git_repo']}" + mirror_dir = "#{mirror_dir}.git" unless fork_info['is_submodule'] possible_new_refs = false if File.directory?(mirror_dir) possible_new_refs = git_fetch(mirror_dir) @@ -159,6 +160,8 @@ class MirrorMain feedback_info = @feedback_queue.pop(true) git_repo = feedback_info[:git_repo] + return if check_submodule(git_repo) + update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] @@ -219,6 +222,8 @@ class MirrorMain end def get_cur_refs(git_repo) + return if @git_info[git_repo]['is_submodule'] + mirror_dir = "/srv/git/#{git_repo}.git" show_ref_out = %x(git -C #{mirror_dir} show-ref --heads) cur_refs = { heads: {} } @@ -328,4 +333,30 @@ class MirrorMain end end end + + def handle_submodule(submodule, parent_project) + submodule.each_line do |line| + next unless line.include?('url = ') + + url = line.split(' = ')[1].chomp + git_repo = "submodules/#{parent_project[0]}/#{parent_project}/#{File.basename(url)}" + @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true } + fork_stat_init(git_repo) + @priority_queue.push git_repo, @priority + @priority += 1 + end + end + + def check_submodule(git_repo) + if @git_info[git_repo]['is_submodule'] + @fork_stat[git_repo][:queued] = false + return true + end + + mirror_dir = "/srv/git/#{git_repo}.git" + submodule = %x(git -C #{mirror_dir} show HEAD:.gitmodules) + return if submodule.empty? + + handle_submodule(submodule, File.basename(git_repo)) + end end -- 2.23.0

1 0

[PATCH compass-ci] git_mirror: pull submodule if repositories have
by Li Yuanchao 10 Nov '20

10 Nov '20

[why] Some repositories have submodules. Without these submodules, they can not be built successfully. Submodules of repositories are repositories too. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 9d6a45d..6b7f80e 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -56,7 +56,8 @@ class GitMirror def mirror_sync fork_info = @queue.pop - mirror_dir = "/srv/git/#{fork_info['git_repo']}.git" + mirror_dir = "/srv/git/#{fork_info['git_repo']}" + mirror_dir = "#{mirror_dir}.git" unless fork_info['is_submodule'] possible_new_refs = false if File.directory?(mirror_dir) possible_new_refs = git_fetch(mirror_dir) @@ -159,6 +160,8 @@ class MirrorMain feedback_info = @feedback_queue.pop(true) git_repo = feedback_info[:git_repo] + return if check_submodule(git_repo) + update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] @@ -219,6 +222,8 @@ class MirrorMain end def get_cur_refs(git_repo) + return if @git_info[git_repo]['is_submodule'] + mirror_dir = "/srv/git/#{git_repo}.git" show_ref_out = %x(git -C #{mirror_dir} show-ref --heads) cur_refs = { heads: {} } @@ -328,4 +333,31 @@ class MirrorMain end end end + + def handle_submodule(submodule, parent_project) + submodule.each_line do |line| + next unless line.include?('url = ') + strings = line.split(' = ') + + url = strings[1].chomp + git_repo = "submodules/#{parent_project[0]}/#{parent_project}/#{File.basename(url)}" + @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true } + fork_stat_init(git_repo) + @priority_queue.push git_repo, @priority + @priority += 1 + end + end + + def check_submodule(git_repo) + if @git_info[git_repo]['is_submodule'] + @fork_stat[git_repo][:queued] = false + return true + end + + mirror_dir = "/srv/git/#{git_repo}.git" + submodule = %x(git -C #{mirror_dir} show HEAD:.gitmodules) + return if submodule.empty? + + handle_submodule(submodule, File.basename(git_repo)) + end end -- 2.23.0

2 2

[PATCH v2 lkp-tests] pkg: add nginx PKGBUILD
by Wei Jihui 10 Nov '20

10 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/nginx/PKGBUILD | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 pkg/nginx/PKGBUILD diff --git a/pkg/nginx/PKGBUILD b/pkg/nginx/PKGBUILD new file mode 100644 index 00000000..989676c4 --- /dev/null +++ b/pkg/nginx/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=nginx +pkgver=1.14 +pkgrel=2 +arch=('i386' 'x86_64') +url="https://nginx.org" +license=('BSD') +source=("https://nginx.org/download/nginx-${pkgver}.${pkgrel}.tar.gz") +md5sums=('SKIP') + +build() +{ + benchmark_path="/lkp/benchmarks/${pkgname}" + mkdir -p "${benchmark_path}" + cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + ./configure --with-http_ssl_module --prefix="${benchmark_path}" + make -j4 +} + +package() +{ + cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + make install DESTDIR="${pkgdir}" +} -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/*: fix double mount problem
by Wang Yong 10 Nov '20

10 Nov '20

check mountpoint before mount destination dir to prevent mount twice problem Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 2 ++ tests/cci-makepkg | 2 ++ 2 files changed, 4 insertions(+) diff --git a/tests/cci-depends b/tests/cci-depends index efa9a025b..4b885dc0c 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -7,6 +7,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh +. $LKP_SRC/lib/mount.sh [ -n "$benchmark" ] || die "benchmark is empty" [ -n "$os_mount" ] || die "os_mount is empty" @@ -28,6 +29,7 @@ pack_arch=$os_arch . $LKP_SRC/distro/$DISTRO +is_mount_point ${DEPS_MNT} && umount ${DEPS_MNT} mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$DEPS_MNT $DEPS_MNT || die "Failed to run mount" umask 002 diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe2..a105f1895 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -13,6 +13,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh . $LKP_SRC/lib/env.sh +. $LKP_SRC/lib/mount.sh [ -n "$benchmark" ] || die "benchmark is empty" [ -n "$os_mount" ] || die "os_mount is empty" @@ -34,6 +35,7 @@ pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" [ -n "$LKP_SERVER" ] && { + is_mount_point ${PKG_MNT} && umount ${PKG_MNT} mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } -- 2.23.0

3 3

[PATCH v5 lkp-tests 2/2] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 10 Nov '20

10 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe..1faa1e9f 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -37,6 +37,23 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +check_pkgfile() +{ + PKGBUILD_TAG="-$PKGBUILD_TAG" + PKGBUILD_TAG_FILE="PKGBUILD$PKGBUILD_TAG" + + get_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 @@ -69,6 +86,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && check_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

2 1

[PATCH v5 lkp-tests 1/2] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 10 Nov '20

10 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 PKGBUILD-1.1 contains only the changed part. % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..fbe9c432 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -888,7 +888,7 @@ update_pkgver() { exit 1 fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" - source_safe "$BUILDFILE" + source_safe "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -1604,10 +1604,14 @@ cd_safe() { source_safe() { shopt -u extglob - if ! source "$@"; then - error "$(gettext "Failed to source %s")" "$1" - exit 1 - fi + local file + for file in "$@" + do + if ! source "$file"; then + error "$(gettext "Failed to source %s")" "$file" + exit 1 + fi + done shopt -s extglob } @@ -1626,7 +1630,6 @@ merge_arch_attrs() { source_buildfile() { source_safe "$@" - if (( !SOURCEONLY )); then merge_arch_attrs fi @@ -3661,6 +3664,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +PKGBUILD_TAG=$PKGBUILD_TAG if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then @@ -3691,7 +3695,7 @@ else if [[ ${BUILDFILE:0:1} != "/" ]]; then BUILDFILE="$startdir/$BUILDFILE" fi - source_buildfile "$BUILDFILE" + source_buildfile "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

2 1

转发: compass-ci pkg-build 包构建相关交流 Time: 2020-11-10 18:30-20:30
by Meeting_Book (2) 10 Nov '20

10 Nov '20

-----原始约会----- 发件人: Meeting_Book (2) 发送时间: 2020年11月10日 9:50 收件人: Meeting_Book (2); Dukaitian (Dukaitian, Intelligent Computing R&D) 主题: compass-ci pkg-build 包构建相关交流 Time: 2020-11-10 18:30-20:30 时间: 2020年11月10日星期二 18:30-20:30(UTC+08:00) 北京，重庆，香港特别行政区，乌鲁木齐。地点: 重要性: 高 Topic [WeLink Meeting] compass-ci pkg-build 包构建相关交流 Time 2020-11-10 18:30-20:30 ((UTC+08:00)Beijing) Join Conference Join (Host) >><http://imeeting.huawei.com/meeting/startzoom?id=8c8b832d759dc0d20175afd8bd9…> Join (Internal) >><http://imeeting.huawei.com/meeting/joinzoom?id=908968438&app=welink> Join (External) >><https://welink-meeting.zoom.us/j/908968438> Meeting ID 908 968 438 Convener 杜开田 (d00301275) Internal (1) 杜开田 (d00301275) External (1) compass-ci(a)openeuler.org<mailto:compass-ci@openeuler.org> 注: 会议最大支持200人同时与会。(A conference supports a maximum of 200 participants.) 外部用户加入会议提示“会议已结束或会议号不存在”，请在官网下载最新版本入会。 If external users fail to join a meeting, advise them to download the latest meeting client and try again. 下载链接( DownloadURL ):https://zoom.us/download

1 0

[PATCH compass-ci] dev-env: not use local gitcache for now
by Cui Yili 10 Nov '20

10 Nov '20

Do not use the local gitcache temporarily, it will be unstable. remove gitcache first. Signed-off-by: cuiyili <2268260388(a)qq.com> --- sparrow/3-code/dev-env | 2 -- 1 file changed, 2 deletions(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f4e4fa9..77027d9 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -7,8 +7,6 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${sched_host:=$server_ip} : ${sched_port:=3000} : ${initrd_http_port:=8800} -: ${gitcache_host:=$server_ip} -: ${gitcache_port:=5000} : ${MASTER_FLUENTD_HOST:=$server_ip} : ${MASTER_FLUENTD_PORT:=24224} : ${DOCKER_REGISTRY_HOST:=$server_ip} -- 2.23.0

1 0

[PATCH v8 lkp-tests 2/2] tests/iozone: change the way of parsing parameter
by Lu Kaiyi 10 Nov '20

10 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] use case ... esac for flow control to make parsing more efficient. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 66 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..b6a1e206 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,52 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + # reset IFS to split $test by "," and then restore default + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write") args+=" -i 0" + ;; + "read") args+=" -i 1" + ;; + "rand_rw") args+=" -i 2" + ;; + "backwards") args+=" -i 3" + ;; + "record") args+=" -i 4" + ;; + "stride") args+=" -i 5" + ;; + "fwrite") args+=" -i 6" + ;; + "fread") args+=" -i 7" + ;; + "rand_mix") args+=" -i 8" + ;; + "pwrite") args+=" -i 9" + ;; + "pread") args+=" -i 10" + ;; + "pwritev") args+=" -i 11" + ;; + "preadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

2 1

[PATCH v2 lkp-tests] lib/monitor: query key support regular match
by Wu Zhende 10 Nov '20

10 Nov '20

Enables more flexible monitoring conditions. When I use "monitor job.*=", will get "query=>{"job":{"*":null}}". It's not what I want. I want "query=>{"job.*": null}". Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 56283e10..67c2389a 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -51,7 +51,7 @@ class Monitor def merge_overrides return if @overrides.empty? - revise_hash(@query, @overrides, true) + @query.merge!(@overrides) end def field_check -- 2.23.0

2 1

[PATCH compass-ci 3/3] [multi-docker] obtain the queues parameter from CLI
by Xiao Shenwei 10 Nov '20

10 Nov '20

Usage: multi-docker -n -c -q -n, --name HOSTNAME_PREFIX format: $tbox_group.$HOSTNAME -c, --count count how much container do you need -q, --queues queues separated by "," -h, --help show this message example: ./multi-docker -n dc-1g.taishan200-2280-2s48p-256g--a1 \ -c 1 \ -q dc-1g.taishan200-2280-2s48p-256g--a1 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-docker | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/providers/multi-docker b/providers/multi-docker index 64cafaa..54db8ab 100755 --- a/providers/multi-docker +++ b/providers/multi-docker @@ -9,10 +9,10 @@ require_relative './docker/docker' opt = {} options = OptionParser.new do |opts| - opts.banner = 'Usage: multi-qemu [-t] [-n]' + opts.banner = 'Usage: multi-docker -n -c -q' opts.separator '' - opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'specify used hostname_prefix') do |name| + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'format: $tbox_group.$HOSTNAME') do |name| opt['hostname_prefix'] = name end @@ -20,6 +20,10 @@ options = OptionParser.new do |opts| opt['nr_container'] = num end + opts.on('-q queues', '--queues queues', 'separated by ","') do |queues| + opt['queues'] = queues + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -28,13 +32,14 @@ end if ARGV.size.zero? puts options - exit + exit 1 end options.parse!(ARGV) -hostname = opt['hostname_prefix'] || 'dc-1g' +hostname = opt['hostname_prefix'] || "dc-1g.#{ENV['HOSTNAME']}" nr_container = opt['nr_container'] || 1 +queues = opt['queues'] || "dc-1g.#{ENV['HOSTNAME']}" -pids = multi_docker(hostname, nr_container) +pids = multi_docker(hostname, nr_container, queues) save_pid pids -- 2.23.0

2 1

[PATCH compass-ci] change the format of hostname
by Xiao Shenwei 10 Nov '20

10 Nov '20

hostname: before: dc-1g--taishan200-2280-2s48p-256g--a1 now : dc-1g.taishan200-2280-2s48p-256g--a1 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 7234dfa..9edad62 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -13,7 +13,7 @@ if [[ $hostname =~ ^(.*)-[0-9]+$ ]]; then else tbox_group=$hostname fi -host=${tbox_group%%--*} +host=${tbox_group%.*} create_yaml_variables "$LKP_SRC/hosts/${host}" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] src/lib/web_backend.rb: add date range limit feature
by Zhang Yuhang 10 Nov '20

10 Nov '20

[why] The "start_time" field need limit by date. example: GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 35 ++++++++++++++++++++++++++++++- src/lib/web_backend.rb | 15 +++++++++++++ 2 files changed, 49 insertions(+), 1 deletion(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index fcc765f..fb519de 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -1,4 +1,6 @@ #!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'sinatra' @@ -31,8 +33,39 @@ get '/compare' do compare(params) end -# GET /get_jobs?upstream_repo=AvxToNeon/AvxToNeon&page_size=12&page_num=3 +# GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 # upstream_repo: +# - regex search +# > optional +# +# os: +# > optional +# +# os_version: +# > optional +# +# os_arch: +# > optional +# +# suite: +# > optional +# +# category: +# > optional +# +# testbox: +# > optional +# +# job_state: +# > optional +# +# id: +# > optional +# +# start_date: +# > optional +# +# end_date: # > optional # # page_size: diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 55c5e4a..b8df6f4 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -1,3 +1,5 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'json' @@ -260,6 +262,17 @@ def get_jobs_result(result) jobs end +def get_job_query_range(condition_fields) + range = { start_time: {} } + start_date = condition_fields[:start_date] + end_date = condition_fields[:end_date] + + range[:start_time][:gte] = "#{start_date} 00:00:00" if start_date + range[:start_time][:lte] = "#{end_date} 23:59:59" if end_date + + { range: range } +end + def search_job(condition_fields, page_size, page_num) must = [] FIELDS.each do |field| @@ -272,6 +285,8 @@ def search_job(condition_fields, page_size, page_num) { term: { field => value } } end end + range = get_job_query_range(condition_fields) + must << range if range[:range][:start_time] result, total = es_search(must, page_size, page_num * page_size) return get_jobs_result(result), total end -- 2.23.0

2 3

[PATCH v9 compass-ci 3/3] assign-account: add my_info for request data
by Luan Shengde 10 Nov '20

10 Nov '20

add my_email, my_name, my_uuid in assign-account request data [why] my_email, my_name, my_uuid is required to config the default yaml file ~/.config/compass-ci/defaults/account.yaml [how] add my_email, my_name, my_uuid in assign-account request data when sending assign-account request. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 94 +++++++++++--------- 1 file changed, 54 insertions(+), 40 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..686a327 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,58 +12,63 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +my_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, my_info) mail_content = Mail.read(email_file) - - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + my_info['my_email'] = mail_content.from[0] + my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end end options = OptionParser.new do |opts| - opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key_file] [--raw-email email_file]\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" opts.banner += " -e or -f is required\n" opts.banner += ' -s is optional when use -e' opts.separator '' opts.separator 'options:' - opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + my_info['my_email'] = email_address + # when apply account with email address, will get no user name + my_info['my_name'] = '' end - opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ + 'ssh pub_key file, enable password-less login') do |pub_key_file| + my_info['my_ssh_pubkey'] = File.read(pub_key_file) end - opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + init_info(email_file, my_info) end - opts.on_tail('-h|--help', 'show this message') do + opts.on_tail('-h', '--help', 'show this message') do puts opts exit end @@ -71,21 +76,24 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: Thank you for joining us. You can use the following command to login the jumper server: - login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} - account password: - #{acct_infos['passwd']} + Suggest: + If you use the password to login, change it in time. regards compass-ci @@ -94,26 +102,32 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(my_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(my_info) message = "No email address specified\n" - message += "use -e email_address add a email address\n" + message += "use -e to add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? + raise message if my_info['my_email'].nil? - acct_info = account_info($apply_info['my_ssh_pubkey']) - - message = build_message($apply_info['my_email'], acct_info) + account_info = apply_account(my_info) + # for manually assign account, there will be no my_commit_url + # but the key my_commit_url is required for es + my_info['my_commit_url'] = '' + my_info['my_login_name'] = account_info['my_login_name'] + my_info.delete 'my_ssh_pubkey' + store_account_info(my_info) + message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(my_info) -- 2.23.0

1 0

[PATCH v9 compass-ci 2/3] assign-account: add config default yaml
by Luan Shengde 10 Nov '20

10 Nov '20

when assigning account, add config the default yaml file [why] easier for user to config the default yaml file [how] write my_email, my_name, my_uuid to the default yaml file ~/.config/compass-ci/default/account.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 80 ++++++++++++-------- 1 file changed, 49 insertions(+), 31 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..137f5c6 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -7,17 +7,17 @@ ACCOUNT_DIR dir layout: tree -�� assigned-users -�� user1 -�� user2 -�� user3 -�� ... -�� available-users -�� user11 -�� user12 -�� user13 -�� ... -�� jumper-info +|-- assigned-users +| |-- user1 +| |-- user2 +| |-- user3 +| |-- ... +|-- available-users +| |-- user11 +| |-- user12 +| |-- user13 +| |-- ... +|-- jumper-info assigned-users: store assigned user files available-users: store available user files @@ -30,9 +30,10 @@ API: call graph: setup_jumper_account_info read_account_info - build_account_name + build_account_info read_jumper_info - setup_authorized_key + config_default_yaml + config_authorized_key the returned data for setup_jumper_account_info like: { @@ -44,6 +45,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,35 +96,50 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) + config_authorized_key(login_name, pub_key) unless pub_key.nil? + config_default_yaml(login_name) + return jumper_account_info end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def config_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + # my_email, my_name, my_uuid is required to config default yaml file + # they are added along with 'my_ssh_pubkey' when sending assign account request + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + end + + def config_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

1 0

[PATCH v9 compass-ci 1/3] assign-account: add check my info
by Luan Shengde 10 Nov '20

10 Nov '20

check my_info for the request data [why] my_info is required when assigning account. my_info can be used to config the default yaml file [how] add check my_info before handle the request my_info: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..c0753b0 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,17 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + assign_jumper_account(data) +end + +def assign_jumper_account(data) + lacked_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lacked_info.join(', ')}." + raise error_message unless lacked_info.empty? + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

1 0

[PATCH compass-ci] sched: refactor sched class for the submit job
by Ren Wen 10 Nov '20

10 Nov '20

According to scheduler.cr API to refactor sched class. Extract submit_job function from sched.cr to submit_job.cr Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 172 +---------------------------------- src/scheduler/submit_job.cr | 175 ++++++++++++++++++++++++++++++++++++ 2 files changed, 176 insertions(+), 171 deletions(-) create mode 100644 src/scheduler/submit_job.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 1b6eb8a..ba89ab9 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -13,6 +13,7 @@ require "../scheduler/jobfile_operate" require "../scheduler/redis_client" require "../scheduler/elasticsearch_client" +require "../scheduler/submit_job" require "../scheduler/find_job_boot" require "../scheduler/find_next_job_boot" require "../scheduler/close_job" @@ -51,177 +52,6 @@ class Sched @redis.hash_del("sched/host2queues", hostname) end - # get cluster config using own lkp_src cluster file, - # a hash type will be returned - def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) - lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) - cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - return YAML.parse(File.read(cluster_file_path)) - end - - def get_commit_date(job) - if (job["upstream_repo"] != "") && (job["upstream_commit"] != "") - data = JSON.parse(%({"git_repo": "#{job["upstream_repo"]}.git", - "git_command": ["git-log", "--pretty=format:%cd", "--date=unix", - "#{job["upstream_commit"]}", "-1"]})) - response = @rgc.git_command(data) - return response.body if response.status_code == 200 - end - - return nil - end - - def submit_job(env : HTTP::Server::Context) - body = env.request.body.not_nil!.gets_to_end - - job_content = JSON.parse(body) - job = Job.new(job_content, job_content["id"]?) - job["commit_date"] = get_commit_date(job) - - # it is not a cluster job if cluster field is empty or - # field's prefix is 'cs-localhost' - cluster_file = job["cluster"] - if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") - return submit_single_job(job) - else - cluster_config = get_cluster_config(cluster_file, - job.lkp_initrd_user, - job.os_arch) - return submit_cluster_job(job, cluster_config) - end - rescue ex - puts ex.inspect_with_backtrace - return [{ - "job_id" => "0", - "message" => ex.to_s, - "job_state" => "submit", - }] - end - - # return: - # success: [{"job_id" => job_id1, "message => "", "job_state" => "submit"}, ...] - # failure: [..., {"job_id" => 0, "message" => err_msg, "job_state" => "submit"}] - def submit_cluster_job(job, cluster_config) - job_messages = Array(Hash(String, String)).new - lab = job.lab - - # collect all job ids - job_ids = [] of String - - net_id = "192.168.222" - ip0 = cluster_config["ip0"]? - if ip0 - ip0 = ip0.as_i - else - ip0 = 1 - end - - # steps for each host - cluster_config["nodes"].as_h.each do |host, config| - tbox_group = host.to_s - job_id = add_task(tbox_group, lab) - - # return when job_id is '0' - # 2 Questions: - # - how to deal with the jobs added to DB prior to this loop - # - may consume job before all jobs done - return job_messages << { - "job_id" => "0", - "message" => "add task queue sched/#{tbox_group} failed", - "job_state" => "submit", - } unless job_id - - job_ids << job_id - - # add to job content when multi-test - job["testbox"] = tbox_group - job.update_tbox_group(tbox_group) - job["node_roles"] = config["roles"].as_a.join(" ") - direct_macs = config["macs"].as_a - direct_ips = [] of String - direct_macs.size.times do - raise "Host id is greater than 254, host_id: #{ip0}" if ip0 > 254 - direct_ips << "#{net_id}.#{ip0}" - ip0 += 1 - end - job["direct_macs"] = direct_macs.join(" ") - job["direct_ips"] = direct_ips.join(" ") - - response = add_job(job, job_id) - message = (response["error"]? ? response["error"]["root_cause"] : "") - job_messages << { - "job_id" => job_id, - "message" => message.to_s, - "job_state" => "submit", - "result_root" => "/srv#{job.result_root}", - } - return job_messages if response["error"]? - end - - cluster_id = job_ids[0] - - # collect all host states - cluster_state = Hash(String, Hash(String, String)).new - job_ids.each do |job_id| - cluster_state[job_id] = {"state" => ""} - # will get cluster id according to job id - @redis.hash_set("sched/id2cluster", job_id, cluster_id) - end - - @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) - - return job_messages - end - - # return: - # success: [{"job_id" => job_id, "message" => "", job_state => "submit"}] - # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] - def submit_single_job(job) - queue = job.queue - return [{ - "job_id" => "0", - "message" => "get queue failed", - "job_state" => "submit", - }] unless queue - - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - - job_id = add_task(queue, job.lab) - return [{ - "job_id" => "0", - "message" => "add task queue sched/#{queue} failed", - "job_state" => "submit", - }] unless job_id - - response = add_job(job, job_id) - message = (response["error"]? ? response["error"]["root_cause"] : "") - - return [{ - "job_id" => job_id, - "message" => message.to_s, - "job_state" => "submit", - "result_root" => "/srv#{job.result_root}", - }] - end - - # return job_id - def add_task(queue, lab) - task_desc = JSON.parse(%({"domain": "compass-ci", "lab": "#{lab}"})) - response = @task_queue.add_task("sched/#{queue}", task_desc) - JSON.parse(response[1].to_json)["id"].to_s if response[0] == 200 - end - - # add job content to es and return a response - def add_job(job, job_id) - job.update_id(job_id) - @es.set_job_content(job) - end - def auto_submit_idle_job(tbox_group) full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" extra_job_fields = [ diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr new file mode 100644 index 0000000..8871464 --- /dev/null +++ b/src/scheduler/submit_job.cr @@ -0,0 +1,175 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def submit_job(env : HTTP::Server::Context) + body = env.request.body.not_nil!.gets_to_end + + job_content = JSON.parse(body) + job = Job.new(job_content, job_content["id"]?) + job["commit_date"] = get_commit_date(job) + + # it is not a cluster job if cluster field is empty or + # field's prefix is 'cs-localhost' + cluster_file = job["cluster"] + if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") + return submit_single_job(job) + else + cluster_config = get_cluster_config(cluster_file, + job.lkp_initrd_user, + job.os_arch) + return submit_cluster_job(job, cluster_config) + end + rescue ex + puts ex.inspect_with_backtrace + return [{ + "job_id" => "0", + "message" => ex.to_s, + "job_state" => "submit", + }] + end + + # return: + # success: [{"job_id" => job_id1, "message => "", "job_state" => "submit"}, ...] + # failure: [..., {"job_id" => 0, "message" => err_msg, "job_state" => "submit"}] + def submit_cluster_job(job, cluster_config) + job_messages = Array(Hash(String, String)).new + lab = job.lab + + # collect all job ids + job_ids = [] of String + + net_id = "192.168.222" + ip0 = cluster_config["ip0"]? + if ip0 + ip0 = ip0.as_i + else + ip0 = 1 + end + + # steps for each host + cluster_config["nodes"].as_h.each do |host, config| + tbox_group = host.to_s + job_id = add_task(tbox_group, lab) + + # return when job_id is '0' + # 2 Questions: + # - how to deal with the jobs added to DB prior to this loop + # - may consume job before all jobs done + return job_messages << { + "job_id" => "0", + "message" => "add task queue sched/#{tbox_group} failed", + "job_state" => "submit", + } unless job_id + + job_ids << job_id + + # add to job content when multi-test + job["testbox"] = tbox_group + job.update_tbox_group(tbox_group) + job["node_roles"] = config["roles"].as_a.join(" ") + direct_macs = config["macs"].as_a + direct_ips = [] of String + direct_macs.size.times do + raise "Host id is greater than 254, host_id: #{ip0}" if ip0 > 254 + direct_ips << "#{net_id}.#{ip0}" + ip0 += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") + + response = add_job(job, job_id) + message = (response["error"]? ? response["error"]["root_cause"] : "") + job_messages << { + "job_id" => job_id, + "message" => message.to_s, + "job_state" => "submit", + "result_root" => "/srv#{job.result_root}", + } + return job_messages if response["error"]? + end + + cluster_id = job_ids[0] + + # collect all host states + cluster_state = Hash(String, Hash(String, String)).new + job_ids.each do |job_id| + cluster_state[job_id] = {"state" => ""} + # will get cluster id according to job id + @redis.hash_set("sched/id2cluster", job_id, cluster_id) + end + + @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) + + return job_messages + end + + # return: + # success: [{"job_id" => job_id, "message" => "", job_state => "submit"}] + # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] + def submit_single_job(job) + queue = job.queue + return [{ + "job_id" => "0", + "message" => "get queue failed", + "job_state" => "submit", + }] unless queue + + # only single job will has "idle job" and "execute rate limiter" + if job["idle_job"].empty? + queue += "#{job.get_uuid_tag}" + else + queue = "#{queue}/idle" + end + + job_id = add_task(queue, job.lab) + return [{ + "job_id" => "0", + "message" => "add task queue sched/#{queue} failed", + "job_state" => "submit", + }] unless job_id + + response = add_job(job, job_id) + message = (response["error"]? ? response["error"]["root_cause"] : "") + + return [{ + "job_id" => job_id, + "message" => message.to_s, + "job_state" => "submit", + "result_root" => "/srv#{job.result_root}", + }] + end + + # return job_id + def add_task(queue, lab) + task_desc = JSON.parse(%({"domain": "compass-ci", "lab": "#{lab}"})) + response = @task_queue.add_task("sched/#{queue}", task_desc) + JSON.parse(response[1].to_json)["id"].to_s if response[0] == 200 + end + + # add job content to es and return a response + def add_job(job, job_id) + job.update_id(job_id) + @es.set_job_content(job) + end + + # get cluster config using own lkp_src cluster file, + # a hash type will be returned + def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) + lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) + cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) + return YAML.parse(File.read(cluster_file_path)) + end + + def get_commit_date(job) + if (job["upstream_repo"] != "") && (job["upstream_commit"] != "") + data = JSON.parse(%({"git_repo": "#{job["upstream_repo"]}.git", + "git_command": ["git-log", "--pretty=format:%cd", "--date=unix", + "#{job["upstream_commit"]}", "-1"]})) + response = @rgc.git_command(data) + return response.body if response.status_code == 200 + end + + return nil + end +end -- 2.23.0

1 0

[PATCH compass-ci] sched: refactor sched class for the update job parameter
by Cao Xueliang 10 Nov '20

10 Nov '20

According to scheduler.cr API to refactor sched class. Extract update_job_parameter function from sched.cr to update_job_parameter.cr Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 31 +----------------------- src/scheduler/update_job_parameter.cr | 34 +++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 30 deletions(-) create mode 100644 src/scheduler/update_job_parameter.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 1b6eb8a..e966ae2 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -17,6 +17,7 @@ require "../scheduler/find_job_boot" require "../scheduler/find_next_job_boot" require "../scheduler/close_job" require "../scheduler/request_cluster_state" +require "../scheduler/update_job_parameter" class Sched property es @@ -236,36 +237,6 @@ class Sched extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end - def update_job_parameter(env : HTTP::Server::Context) - job_id = env.params.query["job_id"]? - if !job_id - return false - end - - # try to get report value and then update it - job_content = {} of String => String - job_content["id"] = job_id - - (%w(start_time end_time loadavg job_state)).each do |parameter| - value = env.params.query[parameter]? - if !value || value == "" - next - end - if parameter == "start_time" || parameter == "end_time" - value = Time.unix(value.to_i).to_local.to_s("%Y-%m-%d %H:%M:%S") - end - - job_content[parameter] = value - end - - @redis.update_job(job_content) - - # json log - log = job_content.dup - log["job_id"] = log.delete("id").not_nil! - return log.to_json - end - def update_tbox_wtmp(env : HTTP::Server::Context) testbox = "" hash = Hash(String, String).new diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr new file mode 100644 index 0000000..0692025 --- /dev/null +++ b/src/scheduler/update_job_parameter.cr @@ -0,0 +1,34 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def update_job_parameter(env : HTTP::Server::Context) + job_id = env.params.query["job_id"]? + if !job_id + return false + end + + # try to get report value and then update it + job_content = {} of String => String + job_content["id"] = job_id + + (%w(start_time end_time loadavg job_state)).each do |parameter| + value = env.params.query[parameter]? + if !value || value == "" + next + end + if parameter == "start_time" || parameter == "end_time" + value = Time.unix(value.to_i).to_local.to_s("%Y-%m-%d %H:%M:%S") + end + + job_content[parameter] = value + end + + @redis.update_job(job_content) + + # json log + log = job_content.dup + log["job_id"] = log.delete("id").not_nil! + return log.to_json + end +end -- 2.23.0

1 0

[PATCH compass-ci] jobfile_operate.cr: reduce if nesting levels
by Xu Xijian 10 Nov '20

10 Nov '20

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/jobfile_operate.cr | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 4ec9f96..baab515 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -58,14 +58,14 @@ module Jobfile::Operate def self.parse_one(script_lines, key, val) return false if val.as_h? - if valid_shell_variable?(key) - value = if val.as_a? - shell_escape(val.as_a) - else - shell_escape(val.to_s) - end - script_lines << "\texport #{key}=" + value if value - end + return false unless valid_shell_variable?(key) + + value = if val.as_a? + shell_escape(val.as_a) + else + shell_escape(val.to_s) + end + script_lines << "\texport #{key}=" + value if value end def self.sh_export_top_env(job_content : Hash) -- 2.23.0

2 1

[PATCH v3 compass-ci 11/11] mail-robot: start file
by Luan Shengde 10 Nov '20

10 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..3f9a1c6 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v /srv/cci/Maildir/.compass-ci:/srv/cci/Maildir/.compass-ci:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'run-mail-robot.rb'] + +system(*cmd) -- 2.23.0

2 3

【Meeting Notice】compass-ci pkg-build 包构建相关交流 Time: 2020-11-10 18:30-20:30
by Meeting Book 10 Nov '20

10 Nov '20

1 0

【Meeting Notice】compass-ci 包构建相关 Time: 2020-11-10 18:30-20:30
by Meeting Book 10 Nov '20

10 Nov '20

1 0

compass-ci 包构建交流
by openEuler conference 10 Nov '20

10 Nov '20

1 0

[PATCH compass-ci 1/3] [multi-docker] obtain the memory,cpu from the hostname
by Xiao Shenwei 10 Nov '20

10 Nov '20

hostname before: dc-1g--$HOSTNAME now : dc-1g.$HOSTNAME Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/run.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 7234dfa..9edad62 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -13,7 +13,7 @@ if [[ $hostname =~ ^(.*)-[0-9]+$ ]]; then else tbox_group=$hostname fi -host=${tbox_group%%--*} +host=${tbox_group%.*} create_yaml_variables "$LKP_SRC/hosts/${host}" -- 2.23.0

2 2

[PATCH v3 compass-ci 03/11] mail-robot: check to assign account
by Luan Shengde 10 Nov '20

10 Nov '20

check to assign account invoke ParseApplyAccountEmail parse commit url parse pubkey invoke AssignJumperAccount generate uuid apply account store account info to es send email send success email send fail email Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account.rb | 73 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 lib/assign-account.rb diff --git a/lib/assign-account.rb b/lib/assign-account.rb new file mode 100755 index 0000000..880de97 --- /dev/null +++ b/lib/assign-account.rb @@ -0,0 +1,73 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'es_client' + +# assign uuid/account +class AssignAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def send_account + my_commit_url, my_ssh_pubkey = parse_commit_url_pub_key + acct_info = apply_my_account(my_ssh_pubkey) + + @my_info['my_commit_url'] = my_commit_url + @my_info['my_login_name'] = acct_info['my_login_name'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + my_commit_url = parse_apply_account_email.parse_commit_url + my_ssh_pubkey = parse_apply_account_email.parse_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def apply_my_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info, my_ssh_pubkey) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(message) + email_message = if message.empty? + build_apply_account_email(@my_info) + else + build_apply_account_fail_email(@my_info, message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

2 2

[PATCH v3 compass-ci 02/11] mail-robot: listen new email directory
by Luan Shengde 10 Nov '20

10 Nov '20

monitor mailbox for new email handle new email file check to apply account only if email's subject match 'apply account' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..168e64b --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,44 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +# require_relative 'send-apply-account-email' +# require_relative 'send-uuid-email' +# require_relative 'send-error-email' +# require_relative 'apply-jumper-account' +# require_relative 'parse-apply-account-email' +# require_relative 'assign-account' + +# MAILDIR = '/srv/cci/Maildir/.compass-ci' + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.send_account +end -- 2.23.0

2 4

[PATCH compass-ci] kernel_version.md: add default vmlinuz in Related files example
by Xu Xijian 10 Nov '20

10 Nov '20

[why] Add default vmlinuz in Related files example, it is a softlink. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/kernel_version.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/job/kernel_version.md b/doc/job/kernel_version.md index f6a5f4d..23c132a 100644 --- a/doc/job/kernel_version.md +++ b/doc/job/kernel_version.md @@ -34,6 +34,7 @@ Related files: ├── modules-4.19.90-2003.cgz ├── modules.cgz -> modules-4.19.90-2003.cgz ├── vmlinuz-4.19.90-2003 +├── vmlinuz -> vmlinuz-4.19.90-2003 Usage example: - submit iperf.yaml testbox=vm-2p8g--$USER os=openeuler os_arch=aarch64 os_version=20.03 runtime=20 kernel_version=4.19.90-2003 -- 2.23.0

2 2

[PATCH lkp-tests] bin/lkp-setup-rootfs: fix mount error
by Li Ping 10 Nov '20

10 Nov '20

[problem]: submit -a cci-makepkg.yaml testbox=taishan200-2280-2s64p-256g--a42 cat /srv/result/cci-makepkg/taishan200-2280-2s64p-256g--a42/2020-11-05/z9.146070/output mount error(16): Device or resource busy Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) Failed to run mount /lkp/lkp/src/lib/debug.sh:12: die /lkp/lkp/src/tests/cci-makepkg:37: main [why]: submit with the option '-a' will call the define_files function and add test-user change files to job's define_files. Then define_files_aoto_pack will be triggered. When submit -a cci-makepkg.yaml, $LKP_SRC/tests/cci-makepkg will run twice, which results in mounting twice and causes mount error. Signed-off-by: Li Ping <15396232681(a)163.com> --- bin/lkp-setup-rootfs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 680027e6..83029f09 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -251,7 +251,9 @@ chmod_root_ssh while true; do set_tbox_wtmp 'running' . $job_script define_files - define_files_auto_pack $define_files + [ "$suite" != "cci-makepkg" ] && [ "$suite" != "cci-depends" ] && { + define_files_auto_pack $define_files + } if [ "$os_mount" != "initramfs" ]; then wget_install_cgz $initrd_deps wget_install_cgz $initrd_pkg -- 2.23.0

3 3

[PATCH lkp-tests] tests/*: fix double mount problem
by Wang Yong 10 Nov '20

10 Nov '20

[Why] when use submit -a and run cci-makepkg/cci-depends job, lkp will call cci-makepkg/cci-depends first and destination dir will mount twice when job is running the second time [How] check mountpoint before mount destination dir Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/cci-depends | 2 ++ tests/cci-makepkg | 2 ++ 2 files changed, 4 insertions(+) diff --git a/tests/cci-depends b/tests/cci-depends index efa9a025b..4b885dc0c 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -7,6 +7,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh +. $LKP_SRC/lib/mount.sh [ -n "$benchmark" ] || die "benchmark is empty" [ -n "$os_mount" ] || die "os_mount is empty" @@ -28,6 +29,7 @@ pack_arch=$os_arch . $LKP_SRC/distro/$DISTRO +is_mount_point ${DEPS_MNT} && umount ${DEPS_MNT} mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$DEPS_MNT $DEPS_MNT || die "Failed to run mount" umask 002 diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe2..a105f1895 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -13,6 +13,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh . $LKP_SRC/lib/env.sh +. $LKP_SRC/lib/mount.sh [ -n "$benchmark" ] || die "benchmark is empty" [ -n "$os_mount" ] || die "os_mount is empty" @@ -34,6 +35,7 @@ pack_to=${os_mount}/${os}/${os_arch}/${os_version}/${benchmark} cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" [ -n "$LKP_SERVER" ] && { + is_mount_point ${PKG_MNT} && umount ${PKG_MNT} mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } -- 2.23.0

2 1

[PATCH compass-ci 2/3] [multi-docker] support queues parameter
by Xiao Shenwei 09 Nov '20

09 Nov '20

register host2queues and requested specified queues Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 47 ++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 4e6bc2f..8e489c6 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -11,15 +11,26 @@ require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' +names = Set.new %w[ + SCHED_HOST + SCHED_PORT +] +defaults = relevant_defaults(names) +SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = defaults['SCHED_PORT'] || 3000 + def get_url(hostname) - names = Set.new %w[ - SCHED_HOST - SCHED_PORT - ] - defaults = relevant_defaults(names) - host = defaults['SCHED_HOST'] || '172.17.0.1' - port = defaults['SCHED_PORT'] || 3000 - "http://#{host}:#{port}/boot.container/hostname/#{hostname}" + "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" +end + +def set_host2queues(hostname, queues) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/set_host2queues?host=#{hostname}&queues=#{queues}'" + system cmd +end + +def del_host2queues(hostname) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/del_host2queues?host=#{hostname}'" + system cmd end def parse_response(url) @@ -67,7 +78,7 @@ def load_initrds(load_path, hash) wget_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") end -def run(hostname, load_path, hash) +def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( @@ -77,24 +88,26 @@ def run(hostname, load_path, hash) clean_dir(load_path) end -def main(hostname) +def main(hostname, queues) + set_host2queues(hostname, queues) url = get_url hostname puts url hash = parse_response url - return if hash.nil? + return del_host2queues(hostname) if hash.nil? load_path = build_load_path(hostname) load_initrds(load_path, hash) - run(hostname, load_path, hash) + start_container(hostname, load_path, hash) + del_host2queues(hostname) end -def loop_main(hostname) +def loop_main(hostname, queues) loop do begin - main(hostname) + main(hostname, queues) rescue StandardError => e puts e.backtrace - # if an exception happend, request the next time after 30 seconds + # if an exception occurs, request the next time after 30 seconds sleep 25 ensure sleep 5 @@ -109,11 +122,11 @@ def save_pid(pids) f.close end -def multi_docker(hostname, nr_container) +def multi_docker(hostname, nr_container, queues) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop_main("#{hostname}-#{i}") + loop_main("#{hostname}-#{i}", queues) end pids << pid end -- 2.23.0

1 0

[PATCH v2 compass-ci] container: fix failed to build kibana images
by Liu Yinsi 09 Nov '20

09 Nov '20

[why] when build kibana images in x86_machine, error message: [root@localhost kibana]# ./build Sending build context to Docker daemon 5.12kB Step 1/3 : FROM gagara/kibana-oss-arm64:7.6.2 7.6.2: Pulling from gagara/kibana-oss-arm64 38163f410fa0: Pull complete 69a4d016f221: Pull complete 95e6c6e7c9ca: Pull complete d13f429dd982: Pull complete 508bb3330fb2: Pull complete 9634e726f1b6: Pull complete 9c26c37850c8: Pull complete 0d0ad8467060: Pull complete 940f92726f8b: Pull complete Digest: sha256:541632b7e9780a007f8a8be82ac8853ddcebcb04a596c00500b73f77eacfbd16 Status: Downloaded newer image for gagara/kibana-oss-arm64:7.6.2 ---> f482a0472f78 Step 2/3 : MAINTAINER Wu Zhende <wuzhende666(a)163.com> ---> Running in cfa86d8ce976 Removing intermediate container cfa86d8ce976 ---> 3be6c5f24d4b Step 3/3 : RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml ---> Running in ff455f66df8b standard_init_linux.go:220: exec user process caused "exec format error" libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error" The command '/bin/sh -c sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/kibana/Dockerfile | 4 +++- container/kibana/build | 8 +++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 35802fe..6e0dba0 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -1,7 +1,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM gagara/kibana-oss-arm64:7.6.2 +ARG BASE_IMAGE + +FROM BASE_IMAGE # docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 diff --git a/container/kibana/build b/container/kibana/build index a7e4717..52d5a2a 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -3,4 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -system 'docker build -t kibana:7.6.2 .' +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', + 'x86_64' => 'kibana:7.6.2' }.freeze + +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] + +system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." -- 2.23.0

1 0

[PATCH v1 lkp-tests] tests: adapt the special version of sysbench test
by Zhang Yu 09 Nov '20

09 Nov '20

[why] Use the special version of sysbench test mysql, use different test file are required [how] Add parameters to test, and use different test file of sysbench Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 79 +++++++++++++++++++++++++++++++------------- 1 file changed, 56 insertions(+), 23 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 4ab5b453..a4d79550 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,52 +1,85 @@ #!/bin/sh -# mysql_user -# mysql_host -# mysql_port -# mysql_db -# db_driver # - oltp_test_mode # - oltp_tables_count # - oltp_table_size +# - max_requests +# - mysql_table_engine # - nr_threads +# - rand_type +# - rand_spec_pct # - runtime # - report_interval : "${mysql_user:=root}" -: "${mysql_host:=localhost}" -: "${mysql_port:=3306}" -: "${mysql_db:=test_1000}" +: "${mysql_host:=$direct_server_ips}" +: "${mysql_port:=$mysql_port}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=$mysql_password}" : "${db_driver:=mysql}" -: "${oltp_tables_count:=3}" -: "${oltp_table_size:=1000}" -: "${nr_threads:=64}" -: "${runtime:=120}" -: "${report_interval:=10}" +: "${oltp_test_mode:=complex}" +: "${oltp_tables_count:=1000}" +: "${oltp_table_size:=100000}" +: "${max_requests:=0}" +: "${mysql_table_engine:=innodb}" +: "${rand_type:=special}" +: "${rand_spec_pct:=100}" +: "${nr_threads:=256}" +: "${runtime:=600}" +: "${report_interval:=1}" -args=( +args1=( --mysql-user=$mysql_user --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db + --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver - --tables=$oltp_tables_count - --table-size=$oltp_table_size + --mysql-password=$mysql_password + --max-requests=$max_requests + --mysql-table-engine=$mysql_table_engine + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --rand-type=$rand_type + --rand-spec-pct=$rand_spec_pct --threads=$nr_threads --time=$runtime - --report-interval=$report_interval ) +args2=( + --mysql-user=$mysql_user + --mysql-password=$mysql_password + --mysql-host=$mysql_host + --mysql-port=$mysql_port + --mysql-db=$mysql_db + --threads=$nr_threads + --oltp-read-only=$oltp_read_only + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --report-interval=$report_interval + --time=$runtime + --events=$events +) +stop_firewalld() +{ + systemctl stop firewalld + iptables -F +} + run_sysbench_step() { - sysbench /usr/share/sysbench/$1 "${args[@]}" $2 + lua_script=$1 + shift + sysbench /usr/local/share/sysbench/tests/include/oltp_legacy/$lua_script "$@" + } run_sysbench_mysql() { - systemctl start mysqld - mysql -e "create database test_1000;" - run_sysbench_step oltp_common.lua prepare - run_sysbench_step oltp_read_write.lua run - run_sysbench_step oltp_common.lua cleanup + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + run_sysbench_step oltp.lua ${args2[@]} run + run_sysbench_step oltp.lua ${args2[@]} cleanup } +stop_firewalld run_sysbench_mysql + -- 2.23.0

1 0

Re: [PATCH compass-ci] container: fix failed to build kibana images
by Liu Yinsi 09 Nov '20

09 Nov '20

>> >> # docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 >> >>diff --git a/container/kibana/build b/container/kibana/build >>index a7e4717..60fdea2 100755 >>--- a/container/kibana/build >>+++ b/container/kibana/build >>@@ -3,4 +3,10 @@ >> # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. >> # frozen_string_literal: true >> >>-system 'docker build -t kibana:7.6.2 .' >>+BASE_IMAGE_DICT = {'aarch64'=>'gagara/kibana-oss-arm64:7.6.2', >>+ 'x86_64'=>'kibana:7.6.2' > >add space surround '=>' > >cat the hash in multi lines, write it as: >BASE_IMAGE_DICT = { > 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', > 'x86_64' => 'kibana:7.6.2' >} good Thanks, Yinsi > >Thanks, >Luan Shengde > >>+} >>+ >>+BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] >>+ >>+system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." >>-- >>2.23.0 >>

1 0

[PATCH v3 compass-ci 04/11] mail-robot: parse commit url and pubkey
by Luan Shengde 09 Nov '20

09 Nov '20

parse commit url and pubkey parse commit url check commit url exists check base url in upstream-repos check commit available hub: gitee.com clone the repo and check the commit exists hub: non-gitee.com check the feedback of curl command to check the commit url available parse pubkey extract the attachment for pubkey check pubkey available Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/parse-apply-account-email.rb | 142 +++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+) create mode 100755 lib/parse-apply-account-email.rb diff --git a/lib/parse-apply-account-email.rb b/lib/parse-apply-account-email.rb new file mode 100755 index 0000000..703f3f9 --- /dev/null +++ b/lib/parse-apply-account-email.rb @@ -0,0 +1,142 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check whether there is a commit url and ssh pub_key in the email +# apply account on junper server +# the entry point is parse_commit_url_pub_key +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + + # the commit url should be headed with a prefix: my oss commit + # the commit url must be in a standart format, like: + # https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = "No matched commit url found.\n" + error_message += "Ensure that you have add a right commit url with prefix 'my oss commit:'." + + raise error_message + end + + def parse_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + base_url_in_upstream_repos('/c/upstream-repos', base_url) + commit_url_availability(url, base_url) + + return url + end + + def base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = "The repo url for your commit is not in our upstream-repo list.\n" + error_message += 'Use a new one, or consulting the manager for available repo list.' + + raise error_message + end + + def commit_url_availability(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + gitee_commit(url, base_url) + else + non_gitee_commit(url) + end + end + + def gitee_commit(url, base_url) + my_gitee_commit = GiteeCommitUrl.new(@my_info, url, base_url) + my_gitee_commit.gitee_commit_index + end + + def non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_info['my_email'] + + return unless email_index.nil? + + error_message = "We can not conform the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end + + def parse_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + pub_key_exist(pub_key) + + return pub_key + end + + def pub_key_exist(pub_key) + return unless pub_key.nil? + + error_message = "No pub_key found.\n" + error_message += 'Please add the pub_key as an attachment with filename: id_rsa.pub.' + + raise error_message + end +end + +# check commit url availability for hub gitee.com +class GiteeCommitUrl + def initialize(my_info, url, base_url) + @my_info = my_info + @url = url + @base_url = base_url + end + + def gitee_commit_index + repo_dir = @url.split('/')[-3] + repo_url = [@base_url, 'git'].join('.') + commit_id = @url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] + + FileUtils.rm_rf repo_dir + + gitee_commit_exist(email_index) + end + + def gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = "We can not conform the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH v2 compass-ci 1/6] git_mirror: add webhook handler
by Li Yuanchao 09 Nov '20

09 Nov '20

webhook will send a message when user push. try to find the git_repo when get url of the repository. if found, push it to queue. if not found, it maybe an illegal request, so do nothing Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 65d6940..5332c16 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -88,10 +88,16 @@ class MirrorMain @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info + connection_init + handle_webhook + end + + def connection_init connection = Bunny.new('amqp://172.17.0.1:5672') connection.start channel = connection.create_channel @message_queue = channel.queue('new_refs') + @webhook_queue = channel.queue('web_hook') end def fork_stat_init(stat_key) @@ -287,3 +293,40 @@ class MirrorMain es_repo_update(git_repo) end end + +# main thread +class MirrorMain + def check_git_repo(git_repo, webhook_url) + return @git_info.key?(git_repo) && Array(@git_info[git_repo]['url'])[0] == webhook_url + end + # example + # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware + # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc + # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + def get_git_repo(webhook_url) + strings = webhook_url.split('/') + project = strings[-1] + fork_name = strings[-2] + + git_repo = "#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project}/#{fork_name}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project[0]}/#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + puts "webhook: #{webhook_url} is not found!" + end + + def handle_webhook + Thread.new do + @webhook_queue.subscribe(block: true) do |_delivery, _properties, webhook_url| + git_repo = get_git_repo(webhook_url) + do_push(git_repo) if git_repo + sleep(0.1) + end + end + end +end -- 2.23.0

4 5

[PATCH compass-ci] service/monitoring: fix can't delete closed ws
by Wu Zhende 09 Nov '20

09 Nov '20

Use a hash to save the mapping between "query" and "ws". When call function "add_filter_rule" will change the "query". If query={"job_id": "1"}, will change to "query={"job_id": ["1"]}". So need to use the modified query to delete the key-value pair in the hash. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 2 ++ src/monitoring/monitoring.cr | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 8e672e0..5d3bc8b 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -16,6 +16,8 @@ class Filter @hash[query] = Array(HTTP::WebSocket).new unless @hash[query]? @hash[query] << socket + + return query end private def convert_hash_value_to_array(query) diff --git a/src/monitoring/monitoring.cr b/src/monitoring/monitoring.cr index dff1902..d09c363 100644 --- a/src/monitoring/monitoring.cr +++ b/src/monitoring/monitoring.cr @@ -19,7 +19,7 @@ module Monitoring # also can be {"job_id": ["1", "2"]} query = JSON.parse(msg) if query.as_h? - filter.add_filter_rule(query, socket) + query = filter.add_filter_rule(query, socket) end end -- 2.23.0

1 0

[PATCH compass-ci] kernel_params: add kernel custom params
by Wei Jihui 09 Nov '20

09 Nov '20

[Why] user maybe use custom kernel params to setup OS input: (job yaml) kernel_custom_params: "sched_steal_node_limit=8" output: (setup log) ... ro sched_steal_node_limit=8 rdinit=/sbin/init... Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/scheduler/kernel_params.cr | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 54438b5..a558735 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -7,6 +7,10 @@ class Job return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" end + private def kernel_custom_params + return @hash["kernel_custom_params"] if @hash["kernel_custom_params"]? + end + private def set_kernel_append_root os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") @@ -30,7 +34,7 @@ class Job end private def set_kernel_params - self["kernel_params"] = " #{kernel_common_params()} #{kernel_append_root} #{kernel_console()}" + self["kernel_params"] = " #{kernel_common_params()} #{kernel_custom_params()} #{kernel_append_root} #{kernel_console()}" end end -- 2.23.0

1 0

[PATCH v3 compass-ci 10/11] mail-robot: Dockerfile
by Luan Shengde 09 Nov '20

09 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..7fdc8c8 --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v3 compass-ci 09/11] mail-robot: build file
by Luan Shengde 09 Nov '20

09 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v3 compass-ci 08/11] mail-robot: documentation for service
by Luan Shengde 09 Nov '20

09 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 89 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..ec96ca3 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,89 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + my oss commit: commit_url +attahcment + ssh pub_key + - id_rsa.pub + +## steps overview + +1. monitor + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + apply_account + invoke AssignAccount for new account + +2. assign account + init my_info + - my_email + - my_name + send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email + send mail + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account + apply_jumper_account + premise: pubkey exists + complete my_info + - my_uuid + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + - my_email + - my_name + - my_uuid + - my_commit_url + - my_login_name + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

1 0

[PATCH v3 compass-ci 07/11] mail-robot: build fail message
by Luan Shengde 09 Nov '20

09 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-fail-eamil.rb | 38 ++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 lib/assign-account-fail-eamil.rb diff --git a/lib/assign-account-fail-eamil.rb b/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..d15cd69 --- /dev/null +++ b/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following errors + + #{message} + + In order to successfully apply a account, pay attention to the following points: + + 1. mail subject + the subject should exactly: apply account + + 2. commit url + when you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + ensure you commit url exist and available to access. + + 3. ssh pubkey + you need to add a pubkey as an attachment with name 'id_rsa.pub' to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v3 compass-ci 06/11] mail-robot: build success email
by Luan Shengde 09 Nov '20

09 Nov '20

build email message for successfully assigning account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-email.rb | 52 +++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100755 lib/assign-account-email.rb diff --git a/lib/assign-account-email.rb b/lib/assign-account-email.rb new file mode 100755 index 0000000..3fe95eb --- /dev/null +++ b/lib/assign-account-email.rb @@ -0,0 +1,52 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_email(my_info) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{my_info['my_name']}, + + Thank you for joining us. + + ONE-TIME SETUP + + You can use the following info to submit jobs: + + 1) setup default config + run the following command to add the below setup to default config file + + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF + my_email: #{my_info['my_email']} + my_name: #{my_info['my_name']} + my_uuid: #{my_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + + reference to 'how to write job yaml' section to write the job yaml + you can alse reference to files in compass-ci/jobs as example. + + submit jobs, for example: + + submit jobs/iperf.yaml textbox=vm-2p8g + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v3 compass-ci 05/11] mail-robot: apply account for user
by Luan Shengde 09 Nov '20

09 Nov '20

apply account with my_info and my_ssh_pubkey my_info: - my_email - my_name - my_uuid check account exists Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/apply-jumper-account.rb | 40 +++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 lib/apply-jumper-account.rb diff --git a/lib/apply-jumper-account.rb b/lib/apply-jumper-account.rb new file mode 100755 index 0000000..7e82bb0 --- /dev/null +++ b/lib/apply-jumper-account.rb @@ -0,0 +1,40 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +# apply jumper account for user +# the enter point is apply_jumper_account +# the apply_account_info hash need include +# my_email, my_name, my_uuid, my_ssh_pubkey +class ApplyJumperAccount + def initialize(my_info, my_ssh_pubkey) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + @my_ssh_pubkey = my_ssh_pubkey + end + + def apply_jumper_account + @my_info['my_ssh_pubkey'] = @my_ssh_pubkey + + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = ' No more available jumper account.' + error_message += ' You may try again later or consulting the managet for a solution.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v3 compass-ci 01/11] mail-robot: entry point file for mail-robot service
by Luan Shengde 09 Nov '20

09 Nov '20

call mail-robot.rb parse-apply-account-email.rb assign-account.rb apply-jumper-account.rb assign-account-email.rb assign-account-fail-eamil.rb to run the mail-robot service Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/run-mail-robot.rb | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/mail-robot/run-mail-robot.rb diff --git a/container/mail-robot/run-mail-robot.rb b/container/mail-robot/run-mail-robot.rb new file mode 100755 index 0000000..6cca2d2 --- /dev/null +++ b/container/mail-robot/run-mail-robot.rb @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../../lib/assign-account-email' +require_relative '../../lib/assign-account-fail-eamil' +require_relative '../../lib/apply-jumper-account' +require_relative '../../lib/parse-apply-account-email' +require_relative '../../lib/assign-account' +require_relative '../../lib/mail-robot' + +MAILDIR = '/srv/cci/Maildir/.compass-ci' + +JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' +JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 + +monitor_new_email("#{MAILDIR}/new", "#{MAILDIR}/cur") -- 2.23.0

1 0

[PATCH compass-ci] sched: refactor sched class for the lkp cluster sync
by Ren Wen 09 Nov '20

09 Nov '20

According to scheduler.cr API to refactor sched class. Extract request_cluster_state function from sched.cr to request_cluster_state.cr. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 108 +----------------------- src/scheduler/request_cluster_state.cr | 111 +++++++++++++++++++++++++ 2 files changed, 112 insertions(+), 107 deletions(-) create mode 100644 src/scheduler/request_cluster_state.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 3709cb1..077b071 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -15,6 +15,7 @@ require "../scheduler/elasticsearch_client" require "../scheduler/find_job_boot" require "../scheduler/find_next_job_boot" +require "../scheduler/request_cluster_state" class Sched property es @@ -49,113 +50,6 @@ class Sched @redis.hash_del("sched/host2queues", hostname) end - # return: - # Hash(String, Hash(String, String)) - def get_cluster_state(cluster_id) - cluster_state = @redis.hash_get("sched/cluster_state", cluster_id) - if cluster_state - cluster_state = Hash(String, Hash(String, String)).from_json(cluster_state) - else - cluster_state = Hash(String, Hash(String, String)).new - end - return cluster_state - end - - # Update job info according to cluster id. - def update_cluster_state(cluster_id, job_id, job_info : Hash(String, String)) - cluster_state = get_cluster_state(cluster_id) - if cluster_state[job_id]? - cluster_state[job_id].merge!(job_info) - @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) - end - end - - # Return response according to different request states. - # all request states: - # wait_ready | abort | failed | finished | wait_finish | - # write_state | roles_ip - def request_cluster_state(env) - request_state = env.params.query["state"] - job_id = env.params.query["job_id"] - cluster_id = @redis.hash_get("sched/id2cluster", job_id).not_nil! - cluster_state = "" - - states = {"abort" => "abort", - "finished" => "finish", - "failed" => "abort", - "wait_ready" => "ready", - "wait_finish" => "finish"} - - case request_state - when "abort", "finished", "failed" - # update node state only - update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) - when "wait_ready" - update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) - @block_helper.block_until_finished(cluster_id) { - cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) - cluster_state == "ready" || cluster_state == "abort" - } - - return cluster_state - when "wait_finish" - update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) - while 1 - sleep(10) - cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) - break if (cluster_state == "finish" || cluster_state == "abort") - end - - return cluster_state - when "write_state" - node_roles = env.params.query["node_roles"] - node_ip = env.params.query["ip"] - direct_ips = env.params.query["direct_ips"] - direct_macs = env.params.query["direct_macs"] - - job_info = {"roles" => node_roles, - "ip" => node_ip, - "direct_ips" => direct_ips, - "direct_macs" => direct_macs} - update_cluster_state(cluster_id, job_id, job_info) - when "roles_ip" - role = "server" - role_state = get_role_state(cluster_id, role) - raise "Missing #{role} state in cluster state" unless role_state - return "server=#{role_state["ip"]}\n" \ - "direct_server_ips=#{role_state["direct_ips"]}" - end - - # show cluster state - return @redis.hash_get("sched/cluster_state", cluster_id) - end - - # get the node state of role from cluster_state - private def get_role_state(cluster_id, role) - cluster_state = get_cluster_state(cluster_id) - cluster_state.each_value do |role_state| - return role_state if role_state["roles"] == role - end - end - - # node_state: "finish" | "ready" - def sync_cluster_state(cluster_id, job_id, node_state) - cluster_state = get_cluster_state(cluster_id) - cluster_state.each_value do |host_state| - state = host_state["state"] - return "abort" if state == "abort" - end - - cluster_state.each_value do |host_state| - state = host_state["state"] - next if "#{state}" == "#{node_state}" - return "retry" - end - - # cluster state is node state when all nodes are normal - return node_state - end - # get cluster config using own lkp_src cluster file, # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr new file mode 100644 index 0000000..07ba6fd --- /dev/null +++ b/src/scheduler/request_cluster_state.cr @@ -0,0 +1,111 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + # return: + # Hash(String, Hash(String, String)) + def get_cluster_state(cluster_id) + cluster_state = @redis.hash_get("sched/cluster_state", cluster_id) + if cluster_state + cluster_state = Hash(String, Hash(String, String)).from_json(cluster_state) + else + cluster_state = Hash(String, Hash(String, String)).new + end + return cluster_state + end + + # Update job info according to cluster id. + def update_cluster_state(cluster_id, job_id, job_info : Hash(String, String)) + cluster_state = get_cluster_state(cluster_id) + if cluster_state[job_id]? + cluster_state[job_id].merge!(job_info) + @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) + end + end + + # Return response according to different request states. + # all request states: + # wait_ready | abort | failed | finished | wait_finish | + # write_state | roles_ip + def request_cluster_state(env) + request_state = env.params.query["state"] + job_id = env.params.query["job_id"] + cluster_id = @redis.hash_get("sched/id2cluster", job_id).not_nil! + cluster_state = "" + + states = {"abort" => "abort", + "finished" => "finish", + "failed" => "abort", + "wait_ready" => "ready", + "wait_finish" => "finish"} + + case request_state + when "abort", "finished", "failed" + # update node state only + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) + when "wait_ready" + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) + @block_helper.block_until_finished(cluster_id) { + cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) + cluster_state == "ready" || cluster_state == "abort" + } + + return cluster_state + when "wait_finish" + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) + while 1 + sleep(10) + cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) + break if (cluster_state == "finish" || cluster_state == "abort") + end + + return cluster_state + when "write_state" + node_roles = env.params.query["node_roles"] + node_ip = env.params.query["ip"] + direct_ips = env.params.query["direct_ips"] + direct_macs = env.params.query["direct_macs"] + + job_info = {"roles" => node_roles, + "ip" => node_ip, + "direct_ips" => direct_ips, + "direct_macs" => direct_macs} + update_cluster_state(cluster_id, job_id, job_info) + when "roles_ip" + role = "server" + role_state = get_role_state(cluster_id, role) + raise "Missing #{role} state in cluster state" unless role_state + return "server=#{role_state["ip"]}\n" \ + "direct_server_ips=#{role_state["direct_ips"]}" + end + + # show cluster state + return @redis.hash_get("sched/cluster_state", cluster_id) + end + + # get the node state of role from cluster_state + private def get_role_state(cluster_id, role) + cluster_state = get_cluster_state(cluster_id) + cluster_state.each_value do |role_state| + return role_state if role_state["roles"] == role + end + end + + # node_state: "finish" | "ready" + def sync_cluster_state(cluster_id, job_id, node_state) + cluster_state = get_cluster_state(cluster_id) + cluster_state.each_value do |host_state| + state = host_state["state"] + return "abort" if state == "abort" + end + + cluster_state.each_value do |host_state| + state = host_state["state"] + next if "#{state}" == "#{node_state}" + return "retry" + end + + # cluster state is node state when all nodes are normal + return node_state + end +end -- 2.23.0

2 2

[PATCH v2 compass-ci] sched: refactor sched class for the lkp cluster sync
by Ren Wen 09 Nov '20

09 Nov '20

According to scheduler.cr API to refactor sched class. Extract request_cluster_state function from sched.cr to request_cluster_state.cr Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 108 +----------------------- src/scheduler/request_cluster_state.cr | 111 +++++++++++++++++++++++++ 2 files changed, 112 insertions(+), 107 deletions(-) create mode 100644 src/scheduler/request_cluster_state.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 6aba6cd..1b6eb8a 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -16,6 +16,7 @@ require "../scheduler/elasticsearch_client" require "../scheduler/find_job_boot" require "../scheduler/find_next_job_boot" require "../scheduler/close_job" +require "../scheduler/request_cluster_state" class Sched property es @@ -50,113 +51,6 @@ class Sched @redis.hash_del("sched/host2queues", hostname) end - # return: - # Hash(String, Hash(String, String)) - def get_cluster_state(cluster_id) - cluster_state = @redis.hash_get("sched/cluster_state", cluster_id) - if cluster_state - cluster_state = Hash(String, Hash(String, String)).from_json(cluster_state) - else - cluster_state = Hash(String, Hash(String, String)).new - end - return cluster_state - end - - # Update job info according to cluster id. - def update_cluster_state(cluster_id, job_id, job_info : Hash(String, String)) - cluster_state = get_cluster_state(cluster_id) - if cluster_state[job_id]? - cluster_state[job_id].merge!(job_info) - @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) - end - end - - # Return response according to different request states. - # all request states: - # wait_ready | abort | failed | finished | wait_finish | - # write_state | roles_ip - def request_cluster_state(env) - request_state = env.params.query["state"] - job_id = env.params.query["job_id"] - cluster_id = @redis.hash_get("sched/id2cluster", job_id).not_nil! - cluster_state = "" - - states = {"abort" => "abort", - "finished" => "finish", - "failed" => "abort", - "wait_ready" => "ready", - "wait_finish" => "finish"} - - case request_state - when "abort", "finished", "failed" - # update node state only - update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) - when "wait_ready" - update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) - @block_helper.block_until_finished(cluster_id) { - cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) - cluster_state == "ready" || cluster_state == "abort" - } - - return cluster_state - when "wait_finish" - update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) - while 1 - sleep(10) - cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) - break if (cluster_state == "finish" || cluster_state == "abort") - end - - return cluster_state - when "write_state" - node_roles = env.params.query["node_roles"] - node_ip = env.params.query["ip"] - direct_ips = env.params.query["direct_ips"] - direct_macs = env.params.query["direct_macs"] - - job_info = {"roles" => node_roles, - "ip" => node_ip, - "direct_ips" => direct_ips, - "direct_macs" => direct_macs} - update_cluster_state(cluster_id, job_id, job_info) - when "roles_ip" - role = "server" - role_state = get_role_state(cluster_id, role) - raise "Missing #{role} state in cluster state" unless role_state - return "server=#{role_state["ip"]}\n" \ - "direct_server_ips=#{role_state["direct_ips"]}" - end - - # show cluster state - return @redis.hash_get("sched/cluster_state", cluster_id) - end - - # get the node state of role from cluster_state - private def get_role_state(cluster_id, role) - cluster_state = get_cluster_state(cluster_id) - cluster_state.each_value do |role_state| - return role_state if role_state["roles"] == role - end - end - - # node_state: "finish" | "ready" - def sync_cluster_state(cluster_id, job_id, node_state) - cluster_state = get_cluster_state(cluster_id) - cluster_state.each_value do |host_state| - state = host_state["state"] - return "abort" if state == "abort" - end - - cluster_state.each_value do |host_state| - state = host_state["state"] - next if "#{state}" == "#{node_state}" - return "retry" - end - - # cluster state is node state when all nodes are normal - return node_state - end - # get cluster config using own lkp_src cluster file, # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr new file mode 100644 index 0000000..ac6cb8e --- /dev/null +++ b/src/scheduler/request_cluster_state.cr @@ -0,0 +1,111 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + # Return response according to different request states. + # all request states: + # wait_ready | abort | failed | finished | wait_finish | + # write_state | roles_ip + def request_cluster_state(env) + request_state = env.params.query["state"] + job_id = env.params.query["job_id"] + cluster_id = @redis.hash_get("sched/id2cluster", job_id).not_nil! + cluster_state = "" + + states = {"abort" => "abort", + "finished" => "finish", + "failed" => "abort", + "wait_ready" => "ready", + "wait_finish" => "finish"} + + case request_state + when "abort", "finished", "failed" + # update node state only + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) + when "wait_ready" + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) + @block_helper.block_until_finished(cluster_id) { + cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) + cluster_state == "ready" || cluster_state == "abort" + } + + return cluster_state + when "wait_finish" + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) + while 1 + sleep(10) + cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) + break if (cluster_state == "finish" || cluster_state == "abort") + end + + return cluster_state + when "write_state" + node_roles = env.params.query["node_roles"] + node_ip = env.params.query["ip"] + direct_ips = env.params.query["direct_ips"] + direct_macs = env.params.query["direct_macs"] + + job_info = {"roles" => node_roles, + "ip" => node_ip, + "direct_ips" => direct_ips, + "direct_macs" => direct_macs} + update_cluster_state(cluster_id, job_id, job_info) + when "roles_ip" + role = "server" + role_state = get_role_state(cluster_id, role) + raise "Missing #{role} state in cluster state" unless role_state + return "server=#{role_state["ip"]}\n" \ + "direct_server_ips=#{role_state["direct_ips"]}" + end + + # show cluster state + return @redis.hash_get("sched/cluster_state", cluster_id) + end + + # node_state: "finish" | "ready" + def sync_cluster_state(cluster_id, job_id, node_state) + cluster_state = get_cluster_state(cluster_id) + cluster_state.each_value do |host_state| + state = host_state["state"] + return "abort" if state == "abort" + end + + cluster_state.each_value do |host_state| + state = host_state["state"] + next if "#{state}" == "#{node_state}" + return "retry" + end + + # cluster state is node state when all nodes are normal + return node_state + end + + # return: + # Hash(String, Hash(String, String)) + def get_cluster_state(cluster_id) + cluster_state = @redis.hash_get("sched/cluster_state", cluster_id) + if cluster_state + cluster_state = Hash(String, Hash(String, String)).from_json(cluster_state) + else + cluster_state = Hash(String, Hash(String, String)).new + end + return cluster_state + end + + # Update job info according to cluster id. + def update_cluster_state(cluster_id, job_id, job_info : Hash(String, String)) + cluster_state = get_cluster_state(cluster_id) + if cluster_state[job_id]? + cluster_state[job_id].merge!(job_info) + @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) + end + end + + # get the node state of role from cluster_state + private def get_role_state(cluster_id, role) + cluster_state = get_cluster_state(cluster_id) + cluster_state.each_value do |role_state| + return role_state if role_state["roles"] == role + end + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] monitoring/filter.cr: query key support regular match
by Wu Zhende 09 Nov '20

09 Nov '20

[Why] Can use regular match in query's keys. [Example] 1: monitor job.*= query=>{"job.*":null} if one log's key include "job", will be matched. 2: monitor .*= this can match all logs Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 88702b6..8e672e0 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,7 +56,8 @@ class Filter def match_query(query : Hash(String, JSON::Any), msg : Hash(String, JSON::Any)) query.each do |key, value| - return false unless msg.has_key?(key) + key = find_real_key(key, msg.keys) unless msg.has_key?(key) + return false unless key values = value.as_a next if values.includes?(nil) || values.includes?(msg[key]?) @@ -66,6 +67,12 @@ class Filter return true end + private def find_real_key(rule, keys) + keys.each do |key| + return key if key.to_s =~ /#{rule}/ + end + end + private def regular_match(rules, string) rules.each do |rule| return true if string =~ /#{rule}/ -- 2.23.0

2 1

[PATCH v8 lkp-tests 1/2] jobs/iozone-bs.yaml: combine multiple test parameter to single
by Lu Kaiyi 09 Nov '20

09 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml [how] combine multiple test parameter to single Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..53f1ac46 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write, read, rand_rw block_size: - 64k -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: query key support regular match
by Wu Zhende 09 Nov '20

09 Nov '20

[Why] Enables more flexible monitoring conditions. When I use "monitor job.*=", will get "query=>{"job":{"*":null}}". It's not what I want. I want "query=>{"job.*": null}". Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 56283e10..67c2389a 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -51,7 +51,7 @@ class Monitor def merge_overrides return if @overrides.empty? - revise_hash(@query, @overrides, true) + @query.merge!(@overrides) end def field_check -- 2.23.0

2 1

[PATCH v1 compass-ci] sched: refactor sched class for the close job function
by Cao Xueliang 09 Nov '20

09 Nov '20

According to scheduler.cr API to refactor sched class. Extract close_job function from sched.cr to close_job.cr Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 28 +--------------------------- src/scheduler/close_job.cr | 31 +++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+), 27 deletions(-) create mode 100644 src/scheduler/close_job.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 3709cb1..6aba6cd 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -15,6 +15,7 @@ require "../scheduler/elasticsearch_client" require "../scheduler/find_job_boot" require "../scheduler/find_next_job_boot" +require "../scheduler/close_job" class Sched property es @@ -404,33 +405,6 @@ class Sched @redis.hash_set("sched/tbox2ssh_port", testbox, ssh_port) end - def delete_access_key_file(job : Job) - File.delete(job.access_key_file) if File.exists?(job.access_key_file) - end - - def close_job(job_id : String) - job = @redis.get_job(job_id) - - delete_access_key_file(job) if job - - response = @es.set_job_content(job) - if response["_id"] == nil - # es update fail, raise exception - raise "es set job content fail!" - end - - response = @task_queue.hand_over_task( - "sched/#{job.queue}", "extract_stats", job_id - ) - if response[0] != 201 - raise "#{response}" - end - - @redis.remove_finished_job(job_id) - - return %({"job_id": "#{job_id}", "job_state": "complete"}) - end - private def query_consumable_keys(shortest_queue_name) keys = [] of String search = "sched/" + shortest_queue_name + "*" diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr new file mode 100644 index 0000000..d071d69 --- /dev/null +++ b/src/scheduler/close_job.cr @@ -0,0 +1,31 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def close_job(job_id : String) + job = @redis.get_job(job_id) + + delete_access_key_file(job) if job + + response = @es.set_job_content(job) + if response["_id"] == nil + # es update fail, raise exception + raise "es set job content fail!" + end + + response = @task_queue.hand_over_task( + "sched/#{job.queue}", "extract_stats", job_id + ) + if response[0] != 201 + raise "#{response}" + end + + @redis.remove_finished_job(job_id) + + return %({"job_id": "#{job_id}", "job_state": "complete"}) + end + + def delete_access_key_file(job : Job) + File.delete(job.access_key_file) if File.exists?(job.access_key_file) + end +end -- 2.23.0

2 1

[PATCH compass-ci] sched: refactor sched class for the close job function
by Cao Xueliang 09 Nov '20

09 Nov '20

According to scheduler.cr API to refactor sched class. Extract close_job function from sched.cr to close_job.cr Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 28 +--------------------------- src/scheduler/close_job.cr | 31 +++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+), 27 deletions(-) create mode 100644 src/scheduler/close_job.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 3709cb1..6aba6cd 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -15,6 +15,7 @@ require "../scheduler/elasticsearch_client" require "../scheduler/find_job_boot" require "../scheduler/find_next_job_boot" +require "../scheduler/close_job" class Sched property es @@ -404,33 +405,6 @@ class Sched @redis.hash_set("sched/tbox2ssh_port", testbox, ssh_port) end - def delete_access_key_file(job : Job) - File.delete(job.access_key_file) if File.exists?(job.access_key_file) - end - - def close_job(job_id : String) - job = @redis.get_job(job_id) - - delete_access_key_file(job) if job - - response = @es.set_job_content(job) - if response["_id"] == nil - # es update fail, raise exception - raise "es set job content fail!" - end - - response = @task_queue.hand_over_task( - "sched/#{job.queue}", "extract_stats", job_id - ) - if response[0] != 201 - raise "#{response}" - end - - @redis.remove_finished_job(job_id) - - return %({"job_id": "#{job_id}", "job_state": "complete"}) - end - private def query_consumable_keys(shortest_queue_name) keys = [] of String search = "sched/" + shortest_queue_name + "*" diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr new file mode 100644 index 0000000..d071d69 --- /dev/null +++ b/src/scheduler/close_job.cr @@ -0,0 +1,31 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def close_job(job_id : String) + job = @redis.get_job(job_id) + + delete_access_key_file(job) if job + + response = @es.set_job_content(job) + if response["_id"] == nil + # es update fail, raise exception + raise "es set job content fail!" + end + + response = @task_queue.hand_over_task( + "sched/#{job.queue}", "extract_stats", job_id + ) + if response[0] != 201 + raise "#{response}" + end + + @redis.remove_finished_job(job_id) + + return %({"job_id": "#{job_id}", "job_state": "complete"}) + end + + def delete_access_key_file(job : Job) + File.delete(job.access_key_file) if File.exists?(job.access_key_file) + end +end -- 2.23.0

1 1

[PATCH compass-ci] container: fix failed to build kibana images
by Liu Yinsi 09 Nov '20

09 Nov '20

[why] when build kibana images in x86_machine, error message: [root@localhost kibana]# ./build Sending build context to Docker daemon 5.12kB Step 1/3 : FROM gagara/kibana-oss-arm64:7.6.2 7.6.2: Pulling from gagara/kibana-oss-arm64 38163f410fa0: Pull complete 69a4d016f221: Pull complete 95e6c6e7c9ca: Pull complete d13f429dd982: Pull complete 508bb3330fb2: Pull complete 9634e726f1b6: Pull complete 9c26c37850c8: Pull complete 0d0ad8467060: Pull complete 940f92726f8b: Pull complete Digest: sha256:541632b7e9780a007f8a8be82ac8853ddcebcb04a596c00500b73f77eacfbd16 Status: Downloaded newer image for gagara/kibana-oss-arm64:7.6.2 ---> f482a0472f78 Step 2/3 : MAINTAINER Wu Zhende <wuzhende666(a)163.com> ---> Running in cfa86d8ce976 Removing intermediate container cfa86d8ce976 ---> 3be6c5f24d4b Step 3/3 : RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml ---> Running in ff455f66df8b standard_init_linux.go:220: exec user process caused "exec format error" libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error" The command '/bin/sh -c sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according different according to different system architecture Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/kibana/Dockerfile | 4 +++- container/kibana/build | 8 +++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 35802fe..6e0dba0 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -1,7 +1,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM gagara/kibana-oss-arm64:7.6.2 +ARG BASE_IMAGE + +FROM BASE_IMAGE # docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 diff --git a/container/kibana/build b/container/kibana/build index a7e4717..60fdea2 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -3,4 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -system 'docker build -t kibana:7.6.2 .' +BASE_IMAGE_DICT = {'aarch64'=>'gagara/kibana-oss-arm64:7.6.2', + 'x86_64'=>'kibana:7.6.2' +} + +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] + +system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." -- 2.23.0

2 1

[PATCH v3 compass-ci] container: fix failed to build archlinux images in x86_64 machine
by Liu Yinsi 09 Nov '20

09 Nov '20

[why] when build archlinux images in x86_64 machine error: error: failed retrieving file 'core.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'core.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update core (failed to retrieve some files) error: failed retrieving file 'extra.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'extra.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Resolving timed out after 10000 milliseconds error: failed to update extra (download library error) error: failed retrieving file 'community.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : The requested URL returned error:404 error: failed to update community (failed to retrieve some files) error: failed to synchronize all databases The command '/bin/sh -c pacman --needed --noprogressbar --noconfirm -Syu && pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk' returned a non-zero code: 1 because archlinux mirror not support x86_64 machine. [how] root/etc/pacman.d/mirrorlist => root/aarch64/etc/pacman.d/mirrorlist root/x86_64/etc/pacman.d/mirrorlist use ARCH=$(arch) to choose mirrorlist according to different system architecture. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/archlinux/Dockerfile | 6 +++++- container/archlinux/build | 2 +- .../archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist | 0 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist | 1 + 4 files changed, 7 insertions(+), 2 deletions(-) rename container/archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist (100%) create mode 100644 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/Dockerfile b/container/archlinux/Dockerfile index c0f05d3..1f80ae0 100644 --- a/container/archlinux/Dockerfile +++ b/container/archlinux/Dockerfile @@ -5,7 +5,11 @@ FROM lopsided/archlinux MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> -COPY root / +ARG ARCH + +COPY root/$ARCH / + RUN chmod 755 /etc + RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk diff --git a/container/archlinux/build b/container/archlinux/build index 81feda2..9749489 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,4 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t archlinux:testbed . +docker build --build-arg ARCH=$(arch) -t archlinux:testbed . diff --git a/container/archlinux/root/etc/pacman.d/mirrorlist b/container/archlinux/root/aarch64/etc/pacman.d/mirrorlist similarity index 100% rename from container/archlinux/root/etc/pacman.d/mirrorlist rename to container/archlinux/root/aarch64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist new file mode 100644 index 0000000..556fac8 --- /dev/null +++ b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist @@ -0,0 +1 @@ +Server = http://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch -- 2.23.0

1 0

[PATCH v5 compass-ci] compare values by each metrics
by Lu Weitao 09 Nov '20

09 Nov '20

compare values by each metrics based on groups matrices, and format compare result as echart data_set background: To support compare with user-defined template feature, the work-flow is: load compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch do: compare_values by each metrics ---> format/show results Signed-off-by: Lu Weitao <luweitaobe(a)163.com> Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_data_format.rb | 18 +++++++ lib/compare_matrixes.rb | 103 +++++++++++++++++++++++++++++++++++++ 2 files changed, 121 insertions(+) create mode 100644 lib/compare_data_format.rb diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb new file mode 100644 index 0000000..3d82550 --- /dev/null +++ b/lib/compare_data_format.rb @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ or GPL-2.0 +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# ---------------------------------------------------------------------------------------------------- +# format compare results for a specific format +# + +def format_for_echart(metrics_compare_results, template_params) + echart_result = {} + echart_result['title'] = template_params['title'] + echart_result['unit'] = template_params['unit'] + x_params = template_params['x_params'] + echart_result['x_name'] = x_params.join('|') if x_params + echart_result['tables'] = metrics_compare_results + + echart_result +end diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 078028a..119d42d 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -6,6 +6,7 @@ LKP_SRC ||= ENV['LKP_SRC'] || File.dirname(__dir__) require 'set' require 'json/ext' require_relative 'themes' +require_relative './compare_data_format.rb' require "#{LKP_SRC}/lib/stats" FAILURE_PATTERNS = IO.read("#{LKP_SRC}/etc/failure").split("\n") @@ -399,6 +400,108 @@ def compare_group_matrices(group_matrices, options) result_str end +# input: groups_matrices +# { +# group_key_1 => { +# dimension_1 => matrix_1, (openeuler 20.03) +# dimension_2 => matrix_2, (openeuler 20.09) +# dimension_3 => matrix_3, (centos 7.6) +# }, +# group_key_2 => {...} +# } +# +# output: compare_metrics_values +# { +# group_key_1 => { +# metric_1 => { +# 'average' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'standard_deviation' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'change' => { +# 'dimension_2 vs dimension_1' => xxx +# 'dimension_3 vs dimension_1' => xxx +# 'dimension_3 vs dimension_2' => xxx +# } +# }, +# metric_2 => {...} +# } +# } +def compare_metrics_values(groups_matrices) + metrics_compare_values = {} + groups_matrices.each do |group_key, dimensions| + metrics_compare_values[group_key] = get_metric_values(dimensions) + end + + metrics_compare_values +end + +def get_metric_values(dimensions) + metrics_values = {} + dimensions.each do |dim, matrix| + matrix.each do |metric, values| + assign_metric_values(metrics_values, dim, metric, values) + end + end + assign_metric_change(metrics_values) + + metrics_values +end + +def assign_metric_values(metrics_values, dim, metric, values) + metrics_values[metric] ||= {} + metrics_values[metric]['average'] ||= {} + metrics_values[metric]['standard_deviation'] ||= {} + metric_value = get_values(values, true) + metrics_values[metric]['average'][dim] = metric_value[:average] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] +end + +def assign_metric_change(metrics_values) + metrics_values.each do |metric, values| + metrics_values[metric]['change'] = {} + next if values['average'].size < 2 + + dimension_list = values['average'].keys + dimension_groups = get_dimensions_combination(dimension_list) + dimension_groups.each do |base_dimension, challenge_dimension| + change = get_compare_value(values['average'][base_dimension], values['average'][challenge_dimension], true) + values['change'] = { "#{challenge_dimension} vs #{base_dimension}" => change } + end + end +end + +# input: dimension_list +# eg: ['openeuler 20.03', 'debian sid', 'centos 7.6'] +# output: Array(base_dimension: String, challenge_dimension: String) +# [ +# ['openeuler 20.03', 'debian sid'], +# ['openeuler 20.03', 'centos 7.6'], +# ['debian sid', 'centos 7.6'] +# ] +def get_dimentions_combination(dimension_list) + dims = [] + dimension_list_size = dimension_list.size + (1..dimension_list_size - 1).each do |i| + (i..dimension_list_size - 1).each do |j| + dims << [dimension_list[i - 1], dimension_list[j]] + end + end + + dims +end + +def show_compare_result(metrics_compare_results, template_params) + echart_results = format_for_echart(metrics_compare_results, template_params) + print JSON.pretty_generate(echart_results) +end + # Format Fields def format_fails_runs(fails, runs) -- 2.23.0

2 2

[PATCH v8 compass-ci 3/3] assign-account: add my_info for assign account
by Luan Shengde 09 Nov '20

09 Nov '20

add my info when execute apply account command [why] when applying account, it's need to add my_info to the default yaml file: ~/.config/compass-ci/defaults/account.yaml my_info: - my_email - my_name - my_uuid [how] add my_info along with the pub_key when applying account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 94 +++++++++++--------- 1 file changed, 54 insertions(+), 40 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..686a327 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,58 +12,63 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +my_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, my_info) mail_content = Mail.read(email_file) - - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + my_info['my_email'] = mail_content.from[0] + my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end end options = OptionParser.new do |opts| - opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key_file] [--raw-email email_file]\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" opts.banner += " -e or -f is required\n" opts.banner += ' -s is optional when use -e' opts.separator '' opts.separator 'options:' - opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + my_info['my_email'] = email_address + # when apply account with email address, will get no user name + my_info['my_name'] = '' end - opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ + 'ssh pub_key file, enable password-less login') do |pub_key_file| + my_info['my_ssh_pubkey'] = File.read(pub_key_file) end - opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + init_info(email_file, my_info) end - opts.on_tail('-h|--help', 'show this message') do + opts.on_tail('-h', '--help', 'show this message') do puts opts exit end @@ -71,21 +76,24 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: Thank you for joining us. You can use the following command to login the jumper server: - login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} - account password: - #{acct_infos['passwd']} + Suggest: + If you use the password to login, change it in time. regards compass-ci @@ -94,26 +102,32 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(my_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(my_info) message = "No email address specified\n" - message += "use -e email_address add a email address\n" + message += "use -e to add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? + raise message if my_info['my_email'].nil? - acct_info = account_info($apply_info['my_ssh_pubkey']) - - message = build_message($apply_info['my_email'], acct_info) + account_info = apply_account(my_info) + # for manually assign account, there will be no my_commit_url + # but the key my_commit_url is required for es + my_info['my_commit_url'] = '' + my_info['my_login_name'] = account_info['my_login_name'] + my_info.delete 'my_ssh_pubkey' + store_account_info(my_info) + message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(my_info) -- 2.23.0

1 0

[PATCH v8 compass-ci 2/3] assign-account: add config default yaml
by Luan Shengde 09 Nov '20

09 Nov '20

when assigning account, config the default yaml [why] easier for user to config the default yaml file [how] write my_info to the default yaml file ~/.config/compass-ci/default/account.yaml my_info: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 88 ++++++++++++-------- 1 file changed, 53 insertions(+), 35 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..24d476b 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -7,17 +7,17 @@ ACCOUNT_DIR dir layout: tree -├── assigned-users -│ ├── user1 -│ ├── user2 -│ ├── user3 -│ ├── ... -├── available-users -│ ├── user11 -│ ├── user12 -│ ├── user13 -│ ├── ... -└── jumper-info +|-- assigned-users +| |-- user1 +| |-- user2 +| |-- user3 +| |-- ... +|-- available-users +| |-- user11 +| |-- user12 +| |-- user13 +| |-- ... +|-- jumper-info assigned-users: store assigned user files available-users: store available user files @@ -30,9 +30,10 @@ API: call graph: setup_jumper_account_info read_account_info - build_account_name + build_account_info read_jumper_info - setup_authorized_key + config_default_yaml + config_authorized_key the returned data for setup_jumper_account_info like: { @@ -44,6 +45,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -61,12 +64,12 @@ class AccountStorage message = 'no more available users' raise message if files.empty? - account_info = build_account_name(available_dir, files) + account_info = build_account_info(available_dir, files) return account_info end - def build_account_name(available_dir, files) + def build_account_info(available_dir, files) files.sort account_info = [] account_info.push files[0] @@ -93,35 +96,50 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end - jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + + new_account_info = { + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) - return jumper_account_info + config_authorized_key(login_name, pub_key) unless pub_key.nil? + config_default_yaml(login_name) + + return new_account_info + end + + def config_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + # my_email, my_name, my_uuid is required to config default yaml file + # they are added along with 'my_ssh_pubkey' when sending assign account request + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def config_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

1 0

[PATCH compass-ci] container/web-backend-nginx: add proxy for srv-http server
by Lu Weitao 09 Nov '20

09 Nov '20

[why] we required access api by https request [how] use web-backend-nginx server to proxy srv-http server that user can access like: https://api.compass-ci.openeuler.org:11320/result/iozone/taishan200-2288-2s… Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend-nginx/nginx.conf | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/container/web-backend-nginx/nginx.conf b/container/web-backend-nginx/nginx.conf index b461a2c..c9a4ec4 100644 --- a/container/web-backend-nginx/nginx.conf +++ b/container/web-backend-nginx/nginx.conf @@ -22,5 +22,14 @@ http { # web-backend server proxy_pass http://172.17.0.1:32767; } + + location ~ ^/(result|pub) { + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forward-For $proxy_add_x_forwarded_for; + + # srv-http server + proxy_pass http://172.17.0.1:11300; + } } } -- 2.23.0

2 3

[PATCH v3 compass-ci] sched: refactor sched class for the find job boot function
by Cao Xueliang 09 Nov '20

09 Nov '20

According to scheduler.cr API to refactor sched class. Extract find_job_boot function from sched.cr to find_job_boot.cr Extract find_next_job_boot function from sched.cr to find_next_job_boot.cr Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 237 +--------------------------- src/scheduler/find_job_boot.cr | 220 ++++++++++++++++++++++++++ src/scheduler/find_next_job_boot.cr | 14 ++ 3 files changed, 237 insertions(+), 234 deletions(-) create mode 100644 src/scheduler/find_job_boot.cr create mode 100644 src/scheduler/find_next_job_boot.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a4b12b4..6ecd95d 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -13,6 +13,9 @@ require "../scheduler/jobfile_operate" require "../scheduler/redis_client" require "../scheduler/elasticsearch_client" +require "../scheduler/find_job_boot" +require "../scheduler/find_next_job_boot" + class Sched property es property redis @@ -323,230 +326,6 @@ class Sched @es.set_job_content(job) end - private def ipxe_msg(msg) - "#!ipxe - echo ... - echo #{msg} - echo ... - reboot" - end - - private def grub_msg(msg) - "#!grub - echo ... - echo #{msg} - echo ... - reboot" - end - - private def get_boot_container(job : Job) - response = Hash(String, String).new - response["docker_image"] = "#{job.docker_image}" - response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + - JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") - response["job"] = "http://#{SCHED_HOST}:#{SCHED_PORT}/job_initrd_tmpfs/#{job.id}/job.cgz" - - return response.to_json - end - - private def get_boot_grub(job : Job) - initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" - - response = "#!grub\n\n" - response += "linux (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" - response += "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")} user=lkp" - response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job" - response += " rootovl ip=dhcp ro root=#{job.kernel_append_root}\n" - - response += "initrd (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" - response += JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/initrd.lkp") - response += " (http,#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT})" - response += JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/#{initrd_lkp_cgz}") - response += " (http,#{SCHED_HOST}:#{SCHED_PORT})/job_initrd_tmpfs/" - response += "#{job.id}/job.cgz\n" - - response += "boot\n" - - return response - end - - def touch_access_key_file(job : Job) - FileUtils.touch(job.access_key_file) - end - - def boot_content(job : Job | Nil, boot_type : String) - touch_access_key_file(job) if job - - case boot_type - when "ipxe" - return job ? get_boot_ipxe(job) : ipxe_msg("No job now") - when "grub" - return job ? get_boot_grub(job) : grub_msg("No job now") - when "container" - return job ? get_boot_container(job) : Hash(String, String).new.to_json - else - raise "Not defined boot type #{boot_type}" - end - end - - def rand_queues(queues) - return queues if queues.empty? - - queues_size = queues.size - base = Random.rand(queues_size) - temp_queues = [] of String - - (0..queues_size - 1).each do |index| - temp_queues << queues[(index + base) % queues_size] - end - - return temp_queues - end - - def get_queues(host) - queues = [] of String - - queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str - - queues_str.split(',', remove_empty: true) do |item| - queues << item.strip - end - - return rand_queues(queues) - end - - def get_job_from_queues(queues, testbox) - job = nil - - queues.each do |queue| - job = prepare_job("sched/#{queue}", testbox) - return job if job - end - - return job - end - - def get_job_boot(host, boot_type) - queues = get_queues(host) - job = get_job_from_queues(queues, host) - - if job - Jobfile::Operate.create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) - end - - return boot_content(job, boot_type) - end - - # auto submit a job to collect the host information - # grub hostname is link with ":", like "00:01:02:03:04:05" - # remind: if like with "-", last "-05" is treated as host number - # then hostname will be "sut-00-01-02-03-04" !!! - def submit_host_info_job(mac) - host = "sut-#{mac}" - set_host_mac(mac, host) - - Jobfile::Operate.auto_submit_job( - "#{ENV["LKP_SRC"]}/jobs/host-info.yaml", - "testbox: #{host}") - end - - def find_job_boot(env : HTTP::Server::Context) - value = env.params.url["value"] - boot_type = env.params.url["boot_type"] - - case boot_type - when "ipxe" - host = @redis.hash_get("sched/mac2host", normalize_mac(value)) - when "grub" - host = @redis.hash_get("sched/mac2host", normalize_mac(value)) - submit_host_info_job(value) unless host - when "container" - host = value - end - - get_job_boot(host, boot_type) - end - - def find_next_job_boot(env) - hostname = env.params.query["hostname"]? - mac = env.params.query["mac"]? - if !hostname && mac - hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) - end - - get_job_boot(hostname, "ipxe") - end - - def get_testbox_boot_content(testbox, boot_type) - job = find_job(testbox) if testbox - Jobfile::Operate.create_job_cpio(job.dump_to_json_any, - Kemal.config.public_folder) if job - - return boot_content(job, boot_type) - end - - private def find_job(testbox : String, count = 1) - tbox_group = JobHelper.match_tbox_group(testbox) - tbox = tbox_group.partition("--")[0] - - queue_list = query_consumable_keys(tbox) - - boxes = ["sched/" + testbox, - "sched/" + tbox_group, - "sched/" + tbox, - "sched/" + tbox_group + "/idle"] - boxes.each do |box| - next if queue_list.select(box).size == 0 - count.times do - job = prepare_job(box, testbox) - return job if job - - sleep(1) unless count == 1 - end - end - - # when find no job, auto submit idle job at background - spawn { auto_submit_idle_job(tbox_group) } - - return nil - end - - private def prepare_job(queue_name, testbox) - response = @task_queue.consume_task(queue_name) - job_id = JSON.parse(response[1].to_json)["id"] if response[0] == 200 - job = nil - - if job_id - begin - job = @es.get_job(job_id.to_s) - rescue ex - puts "Invalid job (id=#{job_id}) in es. Info: #{ex}" - puts ex.inspect_with_backtrace - end - end - - if job - job.update({"testbox" => testbox}) - job.set_result_root - puts %({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state": "set result root"}) - @redis.set_job(job) - end - return job - end - - private def get_idle_job(tbox_group, testbox) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - - # if there has no idle job, auto submit and get 1 - if job.nil? - auto_submit_idle_job(tbox_group) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - end - - return job - end - def auto_submit_idle_job(tbox_group) full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" extra_job_fields = [ @@ -561,16 +340,6 @@ class Sched extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end - private def get_boot_ipxe(job : Job) - response = "#!ipxe\n\n" - response += job.initrds_uri - response += job.kernel_uri - response += job.kernel_params - response += "\nboot\n" - - return response - end - def update_job_parameter(env : HTTP::Server::Context) job_id = env.params.query["job_id"]? if !job_id diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr new file mode 100644 index 0000000..b5a23c5 --- /dev/null +++ b/src/scheduler/find_job_boot.cr @@ -0,0 +1,220 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def find_job_boot(env : HTTP::Server::Context) + value = env.params.url["value"] + boot_type = env.params.url["boot_type"] + + case boot_type + when "ipxe" + host = @redis.hash_get("sched/mac2host", normalize_mac(value)) + when "grub" + host = @redis.hash_get("sched/mac2host", normalize_mac(value)) + submit_host_info_job(value) unless host + when "container" + host = value + end + + get_job_boot(host, boot_type) + end + + # auto submit a job to collect the host information + # grub hostname is link with ":", like "00:01:02:03:04:05" + # remind: if like with "-", last "-05" is treated as host number + # then hostname will be "sut-00-01-02-03-04" !!! + def submit_host_info_job(mac) + host = "sut-#{mac}" + set_host_mac(mac, host) + + Jobfile::Operate.auto_submit_job( + "#{ENV["LKP_SRC"]}/jobs/host-info.yaml", + "testbox: #{host}") + end + + def rand_queues(queues) + return queues if queues.empty? + + queues_size = queues.size + base = Random.rand(queues_size) + temp_queues = [] of String + + (0..queues_size - 1).each do |index| + temp_queues << queues[(index + base) % queues_size] + end + + return temp_queues + end + + def get_queues(host) + queues = [] of String + + queues_str = @redis.hash_get("sched/host2queues", host) + return queues unless queues_str + + queues_str.split(',', remove_empty: true) do |item| + queues << item.strip + end + + return rand_queues(queues) + end + + def get_job_from_queues(queues, testbox) + job = nil + + queues.each do |queue| + job = prepare_job("sched/#{queue}", testbox) + return job if job + end + + return job + end + + def get_job_boot(host, boot_type) + queues = get_queues(host) + job = get_job_from_queues(queues, host) + + if job + Jobfile::Operate.create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + end + + return boot_content(job, boot_type) + end + + private def ipxe_msg(msg) + "#!ipxe + echo ... + echo #{msg} + echo ... + reboot" + end + + private def grub_msg(msg) + "#!grub + echo ... + echo #{msg} + echo ... + reboot" + end + + private def get_boot_container(job : Job) + response = Hash(String, String).new + response["docker_image"] = "#{job.docker_image}" + response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") + response["job"] = "http://#{SCHED_HOST}:#{SCHED_PORT}/job_initrd_tmpfs/#{job.id}/job.cgz" + + return response.to_json + end + + private def get_boot_grub(job : Job) + initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" + + response = "#!grub\n\n" + response += "linux (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" + response += "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")} user=lkp" + response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job" + response += " rootovl ip=dhcp ro root=#{job.kernel_append_root}\n" + + response += "initrd (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" + response += JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/initrd.lkp") + response += " (http,#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT})" + response += JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/#{initrd_lkp_cgz}") + response += " (http,#{SCHED_HOST}:#{SCHED_PORT})/job_initrd_tmpfs/" + response += "#{job.id}/job.cgz\n" + + response += "boot\n" + + return response + end + + def touch_access_key_file(job : Job) + FileUtils.touch(job.access_key_file) + end + + private def get_boot_ipxe(job : Job) + response = "#!ipxe\n\n" + response += job.initrds_uri + response += job.kernel_uri + response += job.kernel_params + response += "\nboot\n" + + return response + end + + def boot_content(job : Job | Nil, boot_type : String) + touch_access_key_file(job) if job + + case boot_type + when "ipxe" + return job ? get_boot_ipxe(job) : ipxe_msg("No job now") + when "grub" + return job ? get_boot_grub(job) : grub_msg("No job now") + when "container" + return job ? get_boot_container(job) : Hash(String, String).new.to_json + else + raise "Not defined boot type #{boot_type}" + end + end + + private def find_job(testbox : String, count = 1) + tbox_group = JobHelper.match_tbox_group(testbox) + tbox = tbox_group.partition("--")[0] + + queue_list = query_consumable_keys(tbox) + + boxes = ["sched/" + testbox, + "sched/" + tbox_group, + "sched/" + tbox, + "sched/" + tbox_group + "/idle"] + boxes.each do |box| + next if queue_list.select(box).size == 0 + count.times do + job = prepare_job(box, testbox) + return job if job + + sleep(1) unless count == 1 + end + end + + # when find no job, auto submit idle job at background + spawn { auto_submit_idle_job(tbox_group) } + + return nil + end + + private def prepare_job(queue_name, testbox) + response = @task_queue.consume_task(queue_name) + job_id = JSON.parse(response[1].to_json)["id"] if response[0] == 200 + job = nil + + if job_id + begin + job = @es.get_job(job_id.to_s) + rescue ex + puts "Invalid job (id=#{job_id}) in es. Info: #{ex}" + puts ex.inspect_with_backtrace + end + end + + if job + job.update({"testbox" => testbox}) + job.set_result_root + puts %({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state": "set result root"}) + @redis.set_job(job) + end + return job + end + + private def get_idle_job(tbox_group, testbox) + job = prepare_job("sched/#{tbox_group}/idle", testbox) + + # if there has no idle job, auto submit and get 1 + if job.nil? + auto_submit_idle_job(tbox_group) + job = prepare_job("sched/#{tbox_group}/idle", testbox) + end + + return job + end +end diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr new file mode 100644 index 0000000..807a9ae --- /dev/null +++ b/src/scheduler/find_next_job_boot.cr @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def find_next_job_boot(env) + hostname = env.params.query["hostname"]? + mac = env.params.query["mac"]? + if !hostname && mac + hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) + end + + get_job_boot(hostname, "ipxe") + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] sched: refactor sched class for the find job boot function
by Cao Xueliang 09 Nov '20

09 Nov '20

According to scheduler.cr API to refactor sched class. Extract find_job_boot function from sched.cr to find_job_boot.cr Extract find_next_job_boot function from sched.cr to find_next_job_boot.cr Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 237 +--------------------------- src/scheduler/find_job_boot.cr | 221 ++++++++++++++++++++++++++ src/scheduler/find_next_job_boot.cr | 14 ++ 3 files changed, 238 insertions(+), 234 deletions(-) create mode 100644 src/scheduler/find_job_boot.cr create mode 100644 src/scheduler/find_next_job_boot.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a4b12b4..6ecd95d 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -13,6 +13,9 @@ require "../scheduler/jobfile_operate" require "../scheduler/redis_client" require "../scheduler/elasticsearch_client" +require "../scheduler/find_job_boot" +require "../scheduler/find_next_job_boot" + class Sched property es property redis @@ -323,230 +326,6 @@ class Sched @es.set_job_content(job) end - private def ipxe_msg(msg) - "#!ipxe - echo ... - echo #{msg} - echo ... - reboot" - end - - private def grub_msg(msg) - "#!grub - echo ... - echo #{msg} - echo ... - reboot" - end - - private def get_boot_container(job : Job) - response = Hash(String, String).new - response["docker_image"] = "#{job.docker_image}" - response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + - JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") - response["job"] = "http://#{SCHED_HOST}:#{SCHED_PORT}/job_initrd_tmpfs/#{job.id}/job.cgz" - - return response.to_json - end - - private def get_boot_grub(job : Job) - initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" - - response = "#!grub\n\n" - response += "linux (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" - response += "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")} user=lkp" - response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job" - response += " rootovl ip=dhcp ro root=#{job.kernel_append_root}\n" - - response += "initrd (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" - response += JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/initrd.lkp") - response += " (http,#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT})" - response += JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/#{initrd_lkp_cgz}") - response += " (http,#{SCHED_HOST}:#{SCHED_PORT})/job_initrd_tmpfs/" - response += "#{job.id}/job.cgz\n" - - response += "boot\n" - - return response - end - - def touch_access_key_file(job : Job) - FileUtils.touch(job.access_key_file) - end - - def boot_content(job : Job | Nil, boot_type : String) - touch_access_key_file(job) if job - - case boot_type - when "ipxe" - return job ? get_boot_ipxe(job) : ipxe_msg("No job now") - when "grub" - return job ? get_boot_grub(job) : grub_msg("No job now") - when "container" - return job ? get_boot_container(job) : Hash(String, String).new.to_json - else - raise "Not defined boot type #{boot_type}" - end - end - - def rand_queues(queues) - return queues if queues.empty? - - queues_size = queues.size - base = Random.rand(queues_size) - temp_queues = [] of String - - (0..queues_size - 1).each do |index| - temp_queues << queues[(index + base) % queues_size] - end - - return temp_queues - end - - def get_queues(host) - queues = [] of String - - queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str - - queues_str.split(',', remove_empty: true) do |item| - queues << item.strip - end - - return rand_queues(queues) - end - - def get_job_from_queues(queues, testbox) - job = nil - - queues.each do |queue| - job = prepare_job("sched/#{queue}", testbox) - return job if job - end - - return job - end - - def get_job_boot(host, boot_type) - queues = get_queues(host) - job = get_job_from_queues(queues, host) - - if job - Jobfile::Operate.create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) - end - - return boot_content(job, boot_type) - end - - # auto submit a job to collect the host information - # grub hostname is link with ":", like "00:01:02:03:04:05" - # remind: if like with "-", last "-05" is treated as host number - # then hostname will be "sut-00-01-02-03-04" !!! - def submit_host_info_job(mac) - host = "sut-#{mac}" - set_host_mac(mac, host) - - Jobfile::Operate.auto_submit_job( - "#{ENV["LKP_SRC"]}/jobs/host-info.yaml", - "testbox: #{host}") - end - - def find_job_boot(env : HTTP::Server::Context) - value = env.params.url["value"] - boot_type = env.params.url["boot_type"] - - case boot_type - when "ipxe" - host = @redis.hash_get("sched/mac2host", normalize_mac(value)) - when "grub" - host = @redis.hash_get("sched/mac2host", normalize_mac(value)) - submit_host_info_job(value) unless host - when "container" - host = value - end - - get_job_boot(host, boot_type) - end - - def find_next_job_boot(env) - hostname = env.params.query["hostname"]? - mac = env.params.query["mac"]? - if !hostname && mac - hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) - end - - get_job_boot(hostname, "ipxe") - end - - def get_testbox_boot_content(testbox, boot_type) - job = find_job(testbox) if testbox - Jobfile::Operate.create_job_cpio(job.dump_to_json_any, - Kemal.config.public_folder) if job - - return boot_content(job, boot_type) - end - - private def find_job(testbox : String, count = 1) - tbox_group = JobHelper.match_tbox_group(testbox) - tbox = tbox_group.partition("--")[0] - - queue_list = query_consumable_keys(tbox) - - boxes = ["sched/" + testbox, - "sched/" + tbox_group, - "sched/" + tbox, - "sched/" + tbox_group + "/idle"] - boxes.each do |box| - next if queue_list.select(box).size == 0 - count.times do - job = prepare_job(box, testbox) - return job if job - - sleep(1) unless count == 1 - end - end - - # when find no job, auto submit idle job at background - spawn { auto_submit_idle_job(tbox_group) } - - return nil - end - - private def prepare_job(queue_name, testbox) - response = @task_queue.consume_task(queue_name) - job_id = JSON.parse(response[1].to_json)["id"] if response[0] == 200 - job = nil - - if job_id - begin - job = @es.get_job(job_id.to_s) - rescue ex - puts "Invalid job (id=#{job_id}) in es. Info: #{ex}" - puts ex.inspect_with_backtrace - end - end - - if job - job.update({"testbox" => testbox}) - job.set_result_root - puts %({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state": "set result root"}) - @redis.set_job(job) - end - return job - end - - private def get_idle_job(tbox_group, testbox) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - - # if there has no idle job, auto submit and get 1 - if job.nil? - auto_submit_idle_job(tbox_group) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - end - - return job - end - def auto_submit_idle_job(tbox_group) full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" extra_job_fields = [ @@ -561,16 +340,6 @@ class Sched extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end - private def get_boot_ipxe(job : Job) - response = "#!ipxe\n\n" - response += job.initrds_uri - response += job.kernel_uri - response += job.kernel_params - response += "\nboot\n" - - return response - end - def update_job_parameter(env : HTTP::Server::Context) job_id = env.params.query["job_id"]? if !job_id diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr new file mode 100644 index 0000000..9f08288 --- /dev/null +++ b/src/scheduler/find_job_boot.cr @@ -0,0 +1,221 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def find_job_boot(env : HTTP::Server::Context) + value = env.params.url["value"] + boot_type = env.params.url["boot_type"] + + case boot_type + when "ipxe" + host = @redis.hash_get("sched/mac2host", normalize_mac(value)) + when "grub" + host = @redis.hash_get("sched/mac2host", normalize_mac(value)) + submit_host_info_job(value) unless host + when "container" + host = value + end + + get_job_boot(host, boot_type) + end + + # auto submit a job to collect the host information + # grub hostname is link with ":", like "00:01:02:03:04:05" + # remind: if like with "-", last "-05" is treated as host number + # then hostname will be "sut-00-01-02-03-04" !!! + def submit_host_info_job(mac) + host = "sut-#{mac}" + set_host_mac(mac, host) + + Jobfile::Operate.auto_submit_job( + "#{ENV["LKP_SRC"]}/jobs/host-info.yaml", + "testbox: #{host}") + end + + def rand_queues(queues) + return queues if queues.empty? + + queues_size = queues.size + base = Random.rand(queues_size) + temp_queues = [] of String + + (0..queues_size - 1).each do |index| + temp_queues << queues[(index + base) % queues_size] + end + + return temp_queues + end + + def get_queues(host) + queues = [] of String + + queues_str = @redis.hash_get("sched/host2queues", host) + return queues unless queues_str + + queues_str.split(',', remove_empty: true) do |item| + queues << item.strip + end + + return rand_queues(queues) + end + + def get_job_from_queues(queues, testbox) + job = nil + + queues.each do |queue| + job = prepare_job("sched/#{queue}", testbox) + return job if job + end + + return job + end + + def get_job_boot(host, boot_type) + queues = get_queues(host) + job = get_job_from_queues(queues, host) + + if job + Jobfile::Operate.create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + end + + return boot_content(job, boot_type) + end + + private def ipxe_msg(msg) + "#!ipxe + echo ... + echo #{msg} + echo ... + reboot" + end + + private def grub_msg(msg) + "#!grub + echo ... + echo #{msg} + echo ... + reboot" + end + + private def get_boot_container(job : Job) + response = Hash(String, String).new + response["docker_image"] = "#{job.docker_image}" + response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") + response["job"] = "http://#{SCHED_HOST}:#{SCHED_PORT}/job_initrd_tmpfs/#{job.id}/job.cgz" + + return response.to_json + end + + private def get_boot_grub(job : Job) + initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" + + response = "#!grub\n\n" + response += "linux (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" + response += "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")} user=lkp" + response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job" + response += " rootovl ip=dhcp ro root=#{job.kernel_append_root}\n" + + response += "initrd (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" + response += JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/initrd.lkp") + response += " (http,#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT})" + response += JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/#{initrd_lkp_cgz}") + response += " (http,#{SCHED_HOST}:#{SCHED_PORT})/job_initrd_tmpfs/" + response += "#{job.id}/job.cgz\n" + + response += "boot\n" + + return response + end + + def touch_access_key_file(job : Job) + FileUtils.touch(job.access_key_file) + end + + private def get_boot_ipxe(job : Job) + response = "#!ipxe\n\n" + response += job.initrds_uri + response += job.kernel_uri + response += job.kernel_params + response += "\nboot\n" + + return response + end + + def boot_content(job : Job | Nil, boot_type : String) + touch_access_key_file(job) if job + + case boot_type + when "ipxe" + return job ? get_boot_ipxe(job) : ipxe_msg("No job now") + when "grub" + return job ? get_boot_grub(job) : grub_msg("No job now") + when "container" + return job ? get_boot_container(job) : Hash(String, String).new.to_json + else + raise "Not defined boot type #{boot_type}" + end + end + + private def find_job(testbox : String, count = 1) + tbox_group = JobHelper.match_tbox_group(testbox) + tbox = tbox_group.partition("--")[0] + + queue_list = query_consumable_keys(tbox) + + boxes = ["sched/" + testbox, + "sched/" + tbox_group, + "sched/" + tbox, + "sched/" + tbox_group + "/idle"] + boxes.each do |box| + next if queue_list.select(box).size == 0 + count.times do + job = prepare_job(box, testbox) + return job if job + + sleep(1) unless count == 1 + end + end + + # when find no job, auto submit idle job at background + spawn { auto_submit_idle_job(tbox_group) } + + return nil + end + + private def prepare_job(queue_name, testbox) + response = @task_queue.consume_task(queue_name) + job_id = JSON.parse(response[1].to_json)["id"] if response[0] == 200 + job = nil + + if job_id + begin + job = @es.get_job(job_id.to_s) + rescue ex + puts "Invalid job (id=#{job_id}) in es. Info: #{ex}" + puts ex.inspect_with_backtrace + end + end + + if job + job.update({"testbox" => testbox}) + job.set_result_root + puts %({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state": "set result root"}) + @redis.set_job(job) + end + return job + end + + private def get_idle_job(tbox_group, testbox) + job = prepare_job("sched/#{tbox_group}/idle", testbox) + + # if there has no idle job, auto submit and get 1 + if job.nil? + auto_submit_idle_job(tbox_group) + job = prepare_job("sched/#{tbox_group}/idle", testbox) + end + + return job + end + +end diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr new file mode 100644 index 0000000..807a9ae --- /dev/null +++ b/src/scheduler/find_next_job_boot.cr @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def find_next_job_boot(env) + hostname = env.params.query["hostname"]? + mac = env.params.query["mac"]? + if !hostname && mac + hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) + end + + get_job_boot(hostname, "ipxe") + end +end -- 2.23.0

3 4

[PATCH v2 lkp-tests] [multi-qemu] support queues parameter
by Xiao Shenwei 09 Nov '20

09 Nov '20

define queues field on multi-qemu: example: multi-qemu-0: nr_vm: 20 tbox_group: vm-2p8g queues: vm-2p8g.taishan200-2280-2s48p-256g--a1 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/multi-qemu | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index b407251f..8d9dd38d 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -1,6 +1,7 @@ #!/bin/sh # - tbox_group # - nr_vm +# - queues # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. @@ -8,10 +9,10 @@ multi_qemu() { export CCI_SRC=/c/compass-ci - local hostname=$tbox_group--$HOSTNAME + local hostname=$tbox_group.$HOSTNAME cd "$CACHE_DIR" - $CCI_SRC/providers/multi-qemu "$hostname" "$nr_vm" + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" } multi_qemu -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] src/lib/web_backend.rb: "/compare_candidates" tbox_group regex error
by Zhang Yuhang 09 Nov '20

09 Nov '20

[error info] 1. /\d+$/ will be matched, such as "xxx123". 2. `index = "xxx123".index('--') || "xxx123".rindex('-')` - index equal nil. 3. `r = r[0, nil]` - raise a TypeError. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index c3f6bef..55c5e4a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -47,7 +47,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end -- 2.23.0

1 0

[PATCH v2 lab-z9] cluster: add cluster config file
by Zhang Yu 09 Nov '20

09 Nov '20

About config filename: "cs-s1-a102-c1": - cs : cluster - s1 : one server node - a102 : server node name - c1 : one client node Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- cluster/cs-s1-a102-c1 | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 cluster/cs-s1-a102-c1 diff --git a/cluster/cs-s1-a102-c1 b/cluster/cs-s1-a102-c1 new file mode 100644 index 0000000..af57123 --- /dev/null +++ b/cluster/cs-s1-a102-c1 @@ -0,0 +1,10 @@ +switch: Switch-P10 +ip0: 1 +nodes: + taishan200-2280-2s48p-512g--a102: + roles: [ server ] + macs: [ "44:67:47:c9:ea:37" ] + + taishan200-2280-2s48p-256g--a101: + roles: [ client ] + macs: [ "44:67:47:d7:6c:a3" ] -- 2.23.0

1 0

[PATCH v3 compass-ci] sched: simplify cluster state updating
by Ren Wen 09 Nov '20

09 Nov '20

There are three steps when updating cluster state: 1) get cluster state from redis. 2) update cluster state. 3) rewrite to redis. Before: write to redis one job info once. After: write to redis more job info once. It will save time when writing more than one job info once. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a4b12b4..bff6090 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -58,11 +58,11 @@ class Sched return cluster_state end - # get -> modify -> set - def update_cluster_state(cluster_id, job_id, property, value) + # Update job info according to cluster id. + def update_cluster_state(cluster_id, job_id, job_info : Hash(String, String)) cluster_state = get_cluster_state(cluster_id) if cluster_state[job_id]? - cluster_state[job_id].merge!({property => value}) + cluster_state[job_id].merge!(job_info) @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) end end @@ -86,9 +86,9 @@ class Sched case request_state when "abort", "finished", "failed" # update node state only - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) @block_helper.block_until_finished(cluster_id) { cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) cluster_state == "ready" || cluster_state == "abort" @@ -96,7 +96,7 @@ class Sched return cluster_state when "wait_finish" - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) @@ -110,10 +110,11 @@ class Sched direct_ips = env.params.query["direct_ips"] direct_macs = env.params.query["direct_macs"] - update_cluster_state(cluster_id, job_id, "roles", node_roles) - update_cluster_state(cluster_id, job_id, "ip", node_ip) - update_cluster_state(cluster_id, job_id, "direct_ips", direct_ips) - update_cluster_state(cluster_id, job_id, "direct_macs", direct_macs) + job_info = {"roles" => node_roles, + "ip" => node_ip, + "direct_ips" => direct_ips, + "direct_macs" => direct_macs} + update_cluster_state(cluster_id, job_id, job_info) when "roles_ip" role = "server" role_state = get_role_state(cluster_id, role) -- 2.23.0

1 0

[PATCH v7 lkp-tests 1/2] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 07 Nov '20

07 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..53f1ac46 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write, read, rand_rw block_size: - 64k -- 2.23.0

2 1

[PATCH v7 lkp-tests 2/2] tests/iozone: change the way of parsing parameter
by Lu Kaiyi 07 Nov '20

07 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] change the way of parsing parameter. small example for parsing parameter: test=" write, read, rand_rw" OLD_IFS="$IFS" IFS="," array=($test) IFS="$OLD_IFS" for ele in ${array[@]} do case $ele in "write") echo $ele ;; "read") echo $ele ;; "rand_rw") echo $ele ;; esac done output: ~/workspace/shell_learning% ./test.sh write read rand_rw if use "${array[@]}" in for loop, there will be no output. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 66 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..b6a1e206 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,52 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + # reset IFS to split $test by "," and then restore default + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write") args+=" -i 0" + ;; + "read") args+=" -i 1" + ;; + "rand_rw") args+=" -i 2" + ;; + "backwards") args+=" -i 3" + ;; + "record") args+=" -i 4" + ;; + "stride") args+=" -i 5" + ;; + "fwrite") args+=" -i 6" + ;; + "fread") args+=" -i 7" + ;; + "rand_mix") args+=" -i 8" + ;; + "pwrite") args+=" -i 9" + ;; + "pread") args+=" -i 10" + ;; + "pwritev") args+=" -i 11" + ;; + "preadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

2 1

[PATCH v2 compass-ci 07/10] container/mail-robot: Dockerfile
by Luan Shengde 07 Nov '20

07 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..4b660c1 --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER shdluan(a)163.com + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

2 4

[PATCH v2 compass-ci 10/10] doc/mail-robot.md
by Luan Shengde 07 Nov '20

07 Nov '20

documentation for mail-robot service Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 76 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..1e65d39 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,76 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + my oss commit: commit_url +attahcment + ssh pub_key + - id_rsa.pub + +## steps overview + +1. monitor + - the monitor will listen to the mailbox for new email files + handle new email file: + check to send account + email subject: apply account + +2. assign account + - check email + check commit url + check whether there is a standard commit url + check whether the repo's url in upstream-repos + check whether the commit url valid + if the hub name is gitee.com + # authentication is needed qwhen use curl to check the url + clone the repo and check the commit + if the hub name is not gitee.com + check the commit with curl + + check pub_key + check whether there is a ssh pub_key in the email file + + it will return failure email if any check end with failure + + - generate uuid + - apply account + build user_info + - my_email + - my_name + - my_uuid + - lab + - pub_key + execute command to apply account on jumper and return the account info + curl -XGET "${JUMPER_HOST}:${JUMPER_PORT}/assign_account" -d '${user_info}' + write my_email, my_name, my_uuid, lab to the defaule config file: + + - store apply infos + apply_infos: + - my_email + - my_name + - my_commit_url + - my_login_name + - my_uuid + invoke es to store the apply infos + + - send uuid + build send uuid email message with the apply_infos + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

2 2

[PATCH v7 compass-ci 3/3] assign-account: optimize answerback-email.rb
by Luan Shengde 07 Nov '20

07 Nov '20

1. disable use golbal variable apply_info [why] rubocop: Style/GlobalVars: Do not introduce global variables. [how] use local variables instead of global variables 2. fix OptionParser argument [why] when write this: opts.on('-e|--email email_address', 'appoint email address') will lead to could not identify the options ./answerback-email.rb --email xxx(a)163.com ./answerback-email.rb:72:in `<main>': invalid option: --email (OptionParser::InvalidOption) [how] write seprately like: opts.on('-e email_address', '--email email_address', 'appoint email address') 3. add my info when execute apply account command [why] when applying account, the assign-account service need to write the my info to user's default config file ~/.config/compass-ci/defaults/account.yaml my_info: - my_email - my_name - my_uuid [how] transfer the user info along with the pub_key when sending apply account request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 94 +++++++++++--------- 1 file changed, 54 insertions(+), 40 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..686a327 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,58 +12,63 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +my_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, my_info) mail_content = Mail.read(email_file) - - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + my_info['my_email'] = mail_content.from[0] + my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end end options = OptionParser.new do |opts| - opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key_file] [--raw-email email_file]\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" opts.banner += " -e or -f is required\n" opts.banner += ' -s is optional when use -e' opts.separator '' opts.separator 'options:' - opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + my_info['my_email'] = email_address + # when apply account with email address, will get no user name + my_info['my_name'] = '' end - opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ + 'ssh pub_key file, enable password-less login') do |pub_key_file| + my_info['my_ssh_pubkey'] = File.read(pub_key_file) end - opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + init_info(email_file, my_info) end - opts.on_tail('-h|--help', 'show this message') do + opts.on_tail('-h', '--help', 'show this message') do puts opts exit end @@ -71,21 +76,24 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: Thank you for joining us. You can use the following command to login the jumper server: - login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + Login command: + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} + + Account password: + #{account_info['my_password']} - account password: - #{acct_infos['passwd']} + Suggest: + If you use the password to login, change it in time. regards compass-ci @@ -94,26 +102,32 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(my_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(my_info) message = "No email address specified\n" - message += "use -e email_address add a email address\n" + message += "use -e to add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? + raise message if my_info['my_email'].nil? - acct_info = account_info($apply_info['my_ssh_pubkey']) - - message = build_message($apply_info['my_email'], acct_info) + account_info = apply_account(my_info) + # for manually assign account, there will be no my_commit_url + # but the key my_commit_url is required for es + my_info['my_commit_url'] = '' + my_info['my_login_name'] = account_info['my_login_name'] + my_info.delete 'my_ssh_pubkey' + store_account_info(my_info) + message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(my_info) -- 2.23.0

1 0

[PATCH v7 compass-ci 2/3] assign-account: add config default yaml for user
by Luan Shengde 07 Nov '20

07 Nov '20

when assign account, config the default yaml for user [why] easier for user to config the default yaml file [how] write my info to the default yaml file ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 90 ++++++++++++-------- 1 file changed, 53 insertions(+), 37 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..239f709 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -7,17 +7,17 @@ ACCOUNT_DIR dir layout: tree -├── assigned-users -│ ├── user1 -│ ├── user2 -│ ├── user3 -│ ├── ... -├── available-users -│ ├── user11 -│ ├── user12 -│ ├── user13 -│ ├── ... -└── jumper-info +|-- assigned-users +| |-- user1 +| |-- user2 +| |-- user3 +| |-- ... +|-- available-users +| |-- user11 +| |-- user12 +| |-- user13 +| |-- ... +|-- jumper-info assigned-users: store assigned user files available-users: store available user files @@ -28,11 +28,12 @@ jumper-info: store jumper server ip:port for the service API: call graph: -setup_jumper_account_info +assign_new_account read_account_info - build_account_name + build_account_info read_jumper_info - setup_authorized_key + config_default_yaml + config_authorized_key the returned data for setup_jumper_account_info like: { @@ -44,6 +45,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -61,12 +64,12 @@ class AccountStorage message = 'no more available users' raise message if files.empty? - account_info = build_account_name(available_dir, files) + account_info = build_account_info(available_dir, files) return account_info end - def build_account_name(available_dir, files) + def build_account_info(available_dir, files) files.sort account_info = [] account_info.push files[0] @@ -90,38 +93,51 @@ class AccountStorage return jumper_info end - def setup_jumper_account_info + def assign_new_account account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end - jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + + new_account_info = { + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) - return jumper_account_info + config_authorized_key(login_name, pub_key) unless pub_key.nil? + config_default_yaml(login_name) + + return new_account_info + end + + def config_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def config_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

1 0

[PATCH v7 compass-ci 1/3] assign-account: add check my info
by Luan Shengde 07 Nov '20

07 Nov '20

check required my info when assigning account [why] my info is required when assigning account used for config the default yaml file [how] add check my info before assign new account my info: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..560f7cc 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,10 +17,19 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + assign_jumper_account(data) +end + +def assign_jumper_account(data) + lacked_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lacked_info.join(', ')}." + raise error_message unless lacked_info.empty? + ref_account_info = AccountStorage.new(data) - account_info = ref_account_info.setup_jumper_account_info + account_info = ref_account_info.assign_new_account return account_info.to_json end -- 2.23.0

1 0

[PATCH v6 compass-ci 1/3] assign-account: add check for assign account
by Luan Shengde 07 Nov '20

07 Nov '20

check required my info when assigning account [why] my info is required when assigning the account and config the default yaml file [how] add check for my info before assign account my info: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index e356c18..834f666 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -7,7 +7,7 @@ require 'sinatra' require 'open3' require 'json' require 'yaml' -require_relative 'get_account_info.rb' +require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 @@ -17,8 +17,17 @@ get '/assign_account' do data = YAML.safe_load request.body.read rescue StandardError => e puts e.message + puts e.backtrace end + check_to_assign_account(data) +end + +def check_to_assign_account(data) + lacked_info = %w[my_email my_name my_uuid] - data.keys + error_message = "lack of my infos: #{lacked_info.join(', ')}." + raise error_message unless lacked_info.empty? + ref_account_info = AccountStorage.new(data) account_info = ref_account_info.setup_jumper_account_info -- 2.23.0

2 2

[PATCH compass-ci] add-software: change vmlinuz judge path
by Xu Xijian 06 Nov '20

06 Nov '20

[why] vmlinuz file is under $os_dir/boot/ now, so change the judge path Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/add-software/run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/add-software/run b/container/add-software/run index f4c6833..089efd2 100755 --- a/container/add-software/run +++ b/container/add-software/run @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -[ -L "$1/vmlinuz" -a -n "$2" ] || { +[ -L "$1/boot/vmlinuz" -a -n "$2" ] || { cat << 'EOF' Example usage: 1> ./run /os/centos/aarch64/7.6.1810 pack1 pack2 ... -- 2.23.0

1 0

[PATCH v2 compass-ci] jobfile_operate.cr: simplify if block
by Xu Xijian 06 Nov '20

06 Nov '20

[why] Somewhere there is only one line of code in if block, simply them with format "$cmd if $condition" Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/jobfile_operate.cr | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 68818ed..4ec9f96 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -56,11 +56,9 @@ module Jobfile::Operate end def self.parse_one(script_lines, key, val) - if valid_shell_variable?(key) - if val.as_h? - return false - end + return false if val.as_h? + if valid_shell_variable?(key) value = if val.as_a? shell_escape(val.as_a) else @@ -133,9 +131,7 @@ module Jobfile::Operate end # if the create job cpio failed, what to do? - if idd.match(/ERROR/) - puts idd - end + puts idd if idd.match(/ERROR/) # create result dir and copy job.sh, job.yaml and job.cgz to result dir src_dir = File.dirname(temp_yaml) dst_dir = File.join("/srv", job_content["result_root"].to_s) @@ -204,9 +200,7 @@ module Jobfile::Operate def self.auto_submit_job(job_file, override_parameter, other_parameters = nil) cmd = "#{ENV["LKP_SRC"]}/sbin/submit " - if other_parameters - cmd += other_parameters.join(" ") - end + cmd += other_parameters.join(" ") if other_parameters cmd += " -s '#{override_parameter}' #{job_file}" puts `#{cmd}` end -- 2.23.0

1 0

[PATCH compass-ci] jobfile_operate.cr: simplify if block
by Xu Xijian 06 Nov '20

06 Nov '20

[why] Somewhere there is only one line of code in if block, simply them with format "$cmd if $condition" Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/jobfile_operate.cr | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index 68818ed..80cbcf7 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -57,9 +57,7 @@ module Jobfile::Operate def self.parse_one(script_lines, key, val) if valid_shell_variable?(key) - if val.as_h? - return false - end + return false if val.as_h? value = if val.as_a? shell_escape(val.as_a) @@ -133,9 +131,7 @@ module Jobfile::Operate end # if the create job cpio failed, what to do? - if idd.match(/ERROR/) - puts idd - end + puts idd if idd.match(/ERROR/) # create result dir and copy job.sh, job.yaml and job.cgz to result dir src_dir = File.dirname(temp_yaml) dst_dir = File.join("/srv", job_content["result_root"].to_s) @@ -204,9 +200,7 @@ module Jobfile::Operate def self.auto_submit_job(job_file, override_parameter, other_parameters = nil) cmd = "#{ENV["LKP_SRC"]}/sbin/submit " - if other_parameters - cmd += other_parameters.join(" ") - end + cmd += other_parameters.join(" ") if other_parameters cmd += " -s '#{override_parameter}' #{job_file}" puts `#{cmd}` end -- 2.23.0

2 2

[PATCH v4 lkp-tests 1/2] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 06 Nov '20

06 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 PKGBUILD-1.1 contains only the changed part. % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..fbe9c432 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -888,7 +888,7 @@ update_pkgver() { exit 1 fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" - source_safe "$BUILDFILE" + source_safe "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -1604,10 +1604,14 @@ cd_safe() { source_safe() { shopt -u extglob - if ! source "$@"; then - error "$(gettext "Failed to source %s")" "$1" - exit 1 - fi + local file + for file in "$@" + do + if ! source "$file"; then + error "$(gettext "Failed to source %s")" "$file" + exit 1 + fi + done shopt -s extglob } @@ -1626,7 +1630,6 @@ merge_arch_attrs() { source_buildfile() { source_safe "$@" - if (( !SOURCEONLY )); then merge_arch_attrs fi @@ -3661,6 +3664,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +PKGBUILD_TAG=$PKGBUILD_TAG if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then @@ -3691,7 +3695,7 @@ else if [[ ${BUILDFILE:0:1} != "/" ]]; then BUILDFILE="$startdir/$BUILDFILE" fi - source_buildfile "$BUILDFILE" + source_buildfile "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

1 0

help star
by Wu Fengguang 06 Nov '20

06 Nov '20

Jihui is our code star, thanks Jihui! wfg@crystal /c/todo% bin/help-star people/*/credit/2020-11-06.md 24.5 Wu Fengguang 23.0 Wei Jihui 18.5 Cao Xueliang 14.7 Yu Chuan 10.2 Zhang Yu 10.1 Luan Shengde 8.9 Li Ping 8.6 Wang Yong 8.4 Wu Zhende 7.6 Lin Jiaxin 7.5 Cui Yili 7.5 Xu Xijian 5.9 Lu Weitao 5.6 Ren Wen 5.5 Zhang Dewan 5.5 Bai Jing 5.3 Zhang Yuhang 5.1 Du Kaitian 4.9 Liu Shaofei 4.9 Zhang Yale 3.5 Liu Yinsi 3.3 Li Yuanchao 3.0 Hu Xuejiao 2.8 Xiao Shenwei 2.2 Lu Kaiyi 1.7 Sun Yukui 1.1 Wang Chenglong 0.0 Tong Qunfeng 0.0 Shi Zhichao 0.0 Wang Shuai 0.0 Cui Xucui 0.0 Zhou Shengrui

1 1

git count
by Wu Fengguang 06 Nov '20

06 Nov '20

Zhende is our code star, thanks Zhende! wfg@crystal /c/todo% bin/git-count 9 Wu Zhende 7 Wang Chenglong 5 Cao Xueliang 5 Xu Xijian 5 Li Ping 4 Liu Yinsi 4 Zhang Yale 4 Yu Chuan 3 Bai Jing 3 Hu Xuejiao 3 Xiao Shenwei 3 Wang Yong 2 Wei Jihui 2 Zhang Yu 2 Lu Kaiyi 1 Liu Shaofei 1 Wu Fengguang 1 Cui Yili 0 Tong Qunfeng 0 Du Kaitian 0 Lu Weitao 0 Sun Yukui 0 Shi Zhichao 0 Li Yuanchao 0 Luan Shengde 0 Ren Wen 0 Zhang Yuhang 0 Lin Jiaxin 0 Zhang Dewan 0 Zhou Shengrui

1 1

review star
by Wu Fengguang 06 Nov '20

06 Nov '20

Xueliang is our review star, thanks Xueliang! wfg@crystal /c/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 141 85 185 44 Wu Fengguang 48 20 24 14 Cao Xueliang 22 15 21 11 Ren Wen 41 17 23 11 Wu Zhende 66 30 32 9 Wei Jihui 45 17 27 8 Xu Xijian 23 13 18 7 Liu Shaofei 55 12 3 7 Lu Kaiyi 45 11 18 6 Xiao Shenwei 42 11 20 5 Li Yuanchao 12 8 8 5 Cui Yili 16 11 24 4 Du Kaitian 44 5 5 4 Zhang Yale 51 8 8 3 Li Ping 37 5 9 3 Wang Chenglong 31 7 24 3 Yu Chuan 24 12 11 3 Zhang Yu 45 10 21 3 Hu Xuejiao 16 9 17 3 Wang Yong 20 10 10 3 Lin Jiaxin 34 15 32 3 Zhang Yuhang 86 24 61 3 Luan Shengde 31 8 18 3 Liu Yinsi 21 6 2 2 Bai Jing 2 2 3 0 Zhang Dewan 11 11 7 0 Sun Yukui 58 2 2 0 Lu Weitao 1 1 2 0 Tong Qunfeng

1 0

[PATCH compass-ci 3/3] src/lib/web_backend.rb: add date range limit feature
by Zhang Yuhang 06 Nov '20

06 Nov '20

[why] The "start_time" field need limit by date. example: GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 33 ++++++++++++++++++++++++++++++- src/lib/web_backend.rb | 13 ++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index fcc765f..96c2347 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -31,8 +31,39 @@ get '/compare' do compare(params) end -# GET /get_jobs?upstream_repo=AvxToNeon/AvxToNeon&page_size=12&page_num=3 +# GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 # upstream_repo: +# - regex search +# > optional +# +# os: +# > optional +# +# os_version: +# > optional +# +# os_arch: +# > optional +# +# suite: +# > optional +# +# category: +# > optional +# +# testbox: +# > optional +# +# job_state: +# > optional +# +# id: +# > optional +# +# start_date: +# > optional +# +# end_date: # > optional # # page_size: diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 55c5e4a..25b99ea 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -260,6 +260,17 @@ def get_jobs_result(result) jobs end +def get_job_query_range(condition_fields) + range = { start_time: {} } + start_date = condition_fields[:start_date] + end_date = condition_fields[:end_date] + + range[:start_time][:gte] = "#{start_date} 00:00:00" if start_date + range[:start_time][:lte] = "#{end_date} 23:59:59" if end_date + + { range: range } +end + def search_job(condition_fields, page_size, page_num) must = [] FIELDS.each do |field| @@ -272,6 +283,8 @@ def search_job(condition_fields, page_size, page_num) { term: { field => value } } end end + range = get_job_query_range(condition_fields) + must << range if range[:range][:start_time] result, total = es_search(must, page_size, page_num * page_size) return get_jobs_result(result), total end -- 2.23.0

1 0

[PATCH compass-ci 2/3] src/lib/web_backend.rb: "/compare_candidates" tbox_group regex error
by Zhang Yuhang 06 Nov '20

06 Nov '20

[error info] 1. /\d+$/ will be matched, such as "xxx123". 2. `index = "xxx123".index('--') || "xxx123".rindex('-')` - index equal nil. 3. `r = r[0, nil]` - raise a TypeError. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index c3f6bef..55c5e4a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -47,7 +47,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end -- 2.23.0

1 0

[PATCH compass-ci 1/3] sbin/es-repo-mapping.sh: create script that generates "repo" mapping
by Zhang Yuhang 06 Nov '20

06 Nov '20

[why] We need a script to create "repo" mapping in es. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-repo-mapping.sh | 47 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 sbin/es-repo-mapping.sh diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh new file mode 100755 index 0000000..4605051 --- /dev/null +++ b/sbin/es-repo-mapping.sh @@ -0,0 +1,47 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Check if "repo" index already exists. +status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) +[ $status_code -eq 200 ] && echo '"repo" index already exists.' && exit + +# Create "repo" index. +echo 'Start to create "repo" index.' +curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' +{ + "mappings": { + "_doc": { + "properties": { + "git_repo": { + "type": "keyword" + }, + "pkgbuild_repo": { + "type": "keyword" + }, + "url": { + "type": "keyword" + }, + "fetch_time": { + "type": "keyword" + }, + "new_refs_time": { + "type": "keyword" + }, + "offset_fetch": { + "type": "long" + }, + "offset_new_refs": { + "type": "long" + }, + "priority": { + "type": "long" + }, + "queued": { + "type": "boolean" + } + } + } + } +}' +[ $? -ne 0 ] && echo 'Failed to create "repo" index.' -- 2.23.0

1 0

[PATCH v2 compass-ci] container: fix failed to build archlinux images in x86_64 machine
by Liu Yinsi 06 Nov '20

06 Nov '20

[why] when build archlinux images in x86_64 machine error: error: failed retrieving file 'core.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'core.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update core (failed to retrieve some files) error: failed retrieving file 'extra.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'extra.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Resolving timed out after 10000 milliseconds error: failed to update extra (download library error) error: failed retrieving file 'community.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : The requested URL returned error:404 error: failed to update community (failed to retrieve some files) error: failed to synchronize all databases The command '/bin/sh -c pacman --needed --noprogressbar --noconfirm -Syu && pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk' returned a non-zero code: 1 because archlinux mirror not support x86_64 machine. [how] root/etc/pacman.d/mirrorlist => root/aarch64/etc/pacman.d/mirrorlist root/x86_64/etc/pacman.d/mirrorlist use ARCH=$(arch) to choose mirrorlist according to different system architecture. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/archlinux/Dockerfile | 6 +++++- container/archlinux/build | 2 +- .../archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist | 0 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist | 1 + 4 files changed, 7 insertions(+), 2 deletions(-) rename container/archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist (100%) create mode 100644 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/Dockerfile b/container/archlinux/Dockerfile index c0f05d3..1f80ae0 100644 --- a/container/archlinux/Dockerfile +++ b/container/archlinux/Dockerfile @@ -5,7 +5,11 @@ FROM lopsided/archlinux MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> -COPY root / +ARG ARCH + +COPY root/$ARCH / + RUN chmod 755 /etc + RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk diff --git a/container/archlinux/build b/container/archlinux/build index 81feda2..9749489 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,4 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t archlinux:testbed . +docker build --build-arg ARCH=$(arch) -t archlinux:testbed . diff --git a/container/archlinux/root/etc/pacman.d/mirrorlist b/container/archlinux/root/aarch64/etc/pacman.d/mirrorlist similarity index 100% rename from container/archlinux/root/etc/pacman.d/mirrorlist rename to container/archlinux/root/aarch64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist new file mode 100644 index 0000000..556fac8 --- /dev/null +++ b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist @@ -0,0 +1 @@ +Server = http://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch -- 2.23.0

1 0

[PATCH compass-ci] container: fix failed to build archlinux images in x86_64 machine
by Liu Yinsi 06 Nov '20

06 Nov '20

[why] when build archlinux images in x86_64 machine error: error: failed retrieving file 'core.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'core.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update core (failed to retrieve some files) error: failed retrieving file 'extra.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'extra.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Resolving timed out after 10000 milliseconds error: failed to update extra (download library error) error: failed retrieving file 'community.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : The requested URL returned error:404 error: failed to update community (failed to retrieve some files) error: failed to synchronize all databases The command '/bin/sh -c pacman --needed --noprogressbar --noconfirm -Syu && pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk' returned a non-zero code: 1 becaue archlinux mirror not support x86_64 machine. [how] root/etc/pacman.d/mirrorlist => root/aarch64/etc/pacman.d/mirrorlist root/x86_64/etc/pacman.d/mirrorlist use arch=$(arch) to choose mirrorlist according to different system architecture. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/archlinux/Dockerfile | 6 +++++- container/archlinux/build | 2 +- .../archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist | 0 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist | 1 + 4 files changed, 7 insertions(+), 2 deletions(-) rename container/archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist (100%) create mode 100644 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/Dockerfile b/container/archlinux/Dockerfile index c0f05d3..1f80ae0 100644 --- a/container/archlinux/Dockerfile +++ b/container/archlinux/Dockerfile @@ -5,7 +5,11 @@ FROM lopsided/archlinux MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> -COPY root / +ARG ARCH + +COPY root/$ARCH / + RUN chmod 755 /etc + RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk diff --git a/container/archlinux/build b/container/archlinux/build index 81feda2..a1eced4 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,4 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t archlinux:testbed . +docker build --build-arg arch=$(arch) -t archlinux:testbed . diff --git a/container/archlinux/root/etc/pacman.d/mirrorlist b/container/archlinux/root/aarch64/etc/pacman.d/mirrorlist similarity index 100% rename from container/archlinux/root/etc/pacman.d/mirrorlist rename to container/archlinux/root/aarch64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist new file mode 100644 index 0000000..556fac8 --- /dev/null +++ b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist @@ -0,0 +1 @@ +Server = http://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch -- 2.23.0

3 4

[PATCH v4 compass-ci] multi-qemu: refactor code and add queues parameter
by Xiao Shenwei 06 Nov '20

06 Nov '20

Usage: multi-qemu -n -c -q -n, --name HOSTNAME_PREFIX format: $tbox_group.$HOSTNAME -c, --count count how many VM do you need -q, --queues queues separated by "," -h, --help show this message example: ./multi-qemu -n vm-2p8g.taishan200-2280-2s48p-256g--a1 \ -c 20 \ -q vm-2p8g.taishan200-2280-2s48p-256g--a1,vm-2p8g.aarch64 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-qemu | 96 +++++++++++++++++++++++++++++++++----------- 1 file changed, 72 insertions(+), 24 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 23e0451..c729c8e 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -4,46 +4,94 @@ # frozen_string_literal: true require 'fileutils' +require 'optparse' -PWD = Dir.pwd +opt = {} +options = OptionParser.new do |opts| + opts.banner = 'Usage: multi-qemu -n -c -q' + + opts.separator '' + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'format: $tbox_group.$HOSTNAME') do |name| + opt['hostname_prefix'] = name + end + + opts.on('-c count', '--count count', 'how many VM do you need') do |num| + opt['nr_vm'] = num + end + + opts.on('-q queues', '--queues queues', 'separated by ","') do |queues| + opt['queues'] = queues + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.size.zero? + puts options + exit 1 +end + +options.parse!(ARGV) # Run multiple QEMU in parallel -HOSTNAME = ARGV[0] || "vm-2p8g--#{ENV['USER']}" -NR_VM = ARGV[1] || 1 +PWD = Dir.pwd +HOSTNAME = opt['hostname_prefix'] || "vm-2p8g.#{ENV['HOSTNAME']}" +NR_VM = opt['nr_vm'] || 1 +QUEUES = opt['queues'] || "#{ENV['HOSTNAME']}.#{RUBY_PLATFORM.split('-')[0]}" +LOG_DIR = '/srv/cci/serial/logs' -def run(seqno) - loop do - start_time = Time.new - hostname = "#{HOSTNAME}-#{seqno}" - log_file = "/srv/cci/serial/logs/#{hostname}" +def main(hostname) + start_time = record_runtime_log(hostname) + start_qemu(hostname) + record_runtime_log(hostname, start_time: start_time, is_start: false) +end +def record_runtime_log(hostname, start_time: Time.new, is_start: true) + log_file = "#{LOG_DIR}/#{hostname}" + if is_start File.open(log_file, 'w') do |f| # fluentd refresh time is 1s # let fluentd to monitor this file first sleep(2) f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting QEMU" end + return start_time + end - pwd_hostname = File.join(PWD, hostname) - FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) - FileUtils.cd(pwd_hostname) - system( - { 'hostname' => hostname }, - ENV['CCI_SRC'] + '/providers/qemu.sh' - ) - - duration = ((Time.new - start_time) / 60).round(2) - File.open(log_file, 'a') do |f| - f.puts "\nTotal QEMU duration: #{duration} minutes" - end + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal QEMU duration: #{duration} minutes" + end +end + +def start_qemu(hostname) + pwd_hostname = File.join(PWD, hostname) + FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) + FileUtils.cd(pwd_hostname) + system( + { 'hostname' => hostname, 'queues' => QUEUES }, + ENV['CCI_SRC'] + '/providers/qemu.sh' + ) +end - # sleep 5s is for fluentd to collect it's log - sleep(5) +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.backtrace + # if an exception occurs, request the next time after 30 seconds + sleep 25 + ensure + sleep 5 + end end end def save_pid(arr) - FileUtils.rm('pid') if File.exist?('pid') f = File.new('pid', 'a') arr.each do |i| f.puts(i) @@ -55,7 +103,7 @@ def multiqemu pids = [] NR_VM.to_i.times do |i| pid = Process.fork do - run i + loop_main("#{HOSTNAME}-#{i}") end pids << pid end -- 2.23.0

1 0

Re: [PATCH v3 lkp-tests 1/2] sbin/makepkg: support applying patch for PKGBUILD
by Ren Wen 06 Nov '20

06 Nov '20

>+ if ! source "$file"; then Builtin `source` returns the status of the last command executed in FILENAME; Is this what you want? Thanks, RenWen >+ error "$(gettext "Failed to source %s")" "$1" >+ exit 1 >+ fi >+ done > shopt -s extglob > } > >@@ -1626,7 +1630,6 @@ merge_arch_attrs() { > > source_buildfile() { > source_safe "$@" >- > if (( !SOURCEONLY )); then > merge_arch_attrs > fi >@@ -3661,6 +3664,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} > GPGKEY=${_GPGKEY:-$GPGKEY} > PACKAGER=${_PACKAGER:-$PACKAGER} > CARCH=${_CARCH:-$CARCH} >+PKGBUILD_TAG=$PKGBUILD_TAG > > if (( INFAKEROOT )); then > if [[ -z $FAKEROOTKEY ]]; then >@@ -3691,7 +3695,7 @@ else > if [[ ${BUILDFILE:0:1} != "/" ]]; then > BUILDFILE="$startdir/$BUILDFILE" > fi >- source_buildfile "$BUILDFILE" >+ source_buildfile "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" > fi > > # set defaults if they weren't specified in buildfile >-- >2.23.0 >

1 0

[PATCH v3 lkp-tests 1/2] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 06 Nov '20

06 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 PKGBUILD-1.1 contains only the changed part. % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..dc8471b9 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -888,7 +888,7 @@ update_pkgver() { exit 1 fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" - source_safe "$BUILDFILE" + source_safe "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -1604,10 +1604,14 @@ cd_safe() { source_safe() { shopt -u extglob - if ! source "$@"; then - error "$(gettext "Failed to source %s")" "$1" - exit 1 - fi + local file + for file in "$@" + do + if ! source "$file"; then + error "$(gettext "Failed to source %s")" "$1" + exit 1 + fi + done shopt -s extglob } @@ -1626,7 +1630,6 @@ merge_arch_attrs() { source_buildfile() { source_safe "$@" - if (( !SOURCEONLY )); then merge_arch_attrs fi @@ -3661,6 +3664,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +PKGBUILD_TAG=$PKGBUILD_TAG if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then @@ -3691,7 +3695,7 @@ else if [[ ${BUILDFILE:0:1} != "/" ]]; then BUILDFILE="$startdir/$BUILDFILE" fi - source_buildfile "$BUILDFILE" + source_buildfile "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

3 5

[PATCH v3 compass-ci] multi-qemu: refactor code and add queues parameter
by Xiao Shenwei 06 Nov '20

06 Nov '20

Usage: multi-qemu -n -c -q -n, --name HOSTNAME_PREFIX format: $tbox_group.$HOSTNAME -c, --count count how many VM do you need -q, --queues queues separated by "," -h, --help show this message example: ./multi-qemu -n vm-2p8g.taishan200-2280-2s48p-256g--a1 \ -c 20 \ -q vm-2p8g.taishan200-2280-2s48p-256g--a1,vm-2p8g.aarch64 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-qemu | 95 +++++++++++++++++++++++++++++++++----------- 1 file changed, 71 insertions(+), 24 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 23e0451..bb85d41 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -4,46 +4,93 @@ # frozen_string_literal: true require 'fileutils' +require 'optparse' -PWD = Dir.pwd +opt = {} +options = OptionParser.new do |opts| + opts.banner = 'Usage: multi-qemu -n -c -q' + + opts.separator '' + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'format: $tbox_group.$HOSTNAME') do |name| + opt['hostname_prefix'] = name + end + + opts.on('-c count', '--count count', 'how many VM do you need') do |num| + opt['nr_vm'] = num + end + + opts.on('-q queues', '--queues queues', 'separated by ","') do |queues| + opt['queues'] = queues + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.size.zero? + puts options + exit 1 +end + +options.parse!(ARGV) # Run multiple QEMU in parallel -HOSTNAME = ARGV[0] || "vm-2p8g--#{ENV['USER']}" -NR_VM = ARGV[1] || 1 +PWD = Dir.pwd +HOSTNAME = opt['hostname_prefix'] || "vm-2p8g.#{ENV['HOSTNAME']}" +NR_VM = opt['nr_vm'] || 1 +QUEUES = opt['queues'] || "#{ENV['HOSTNAME']}.#{RUBY_PLATFORM.split('-')[0]}" +LOG_DIR = '/srv/cci/serial/logs' -def run(seqno) - loop do - start_time = Time.new - hostname = "#{HOSTNAME}-#{seqno}" - log_file = "/srv/cci/serial/logs/#{hostname}" +def main(hostname) + start_time = record_runtime_log(hostname) + start_qemu(hostname) + record_runtime_log(hostname, start_time: start_time, is_start: false) +end +def record_runtime_log(hostname, start_time: Time.new, is_start: true) + log_file = "#{LOG_DIR}/#{hostname}" + if is_start File.open(log_file, 'w') do |f| # fluentd refresh time is 1s # let fluentd to monitor this file first sleep(2) f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting QEMU" end + return start_time + end + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal QEMU duration: #{duration} minutes" + end +end - pwd_hostname = File.join(PWD, hostname) - FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) - FileUtils.cd(pwd_hostname) - system( - { 'hostname' => hostname }, - ENV['CCI_SRC'] + '/providers/qemu.sh' - ) - - duration = ((Time.new - start_time) / 60).round(2) - File.open(log_file, 'a') do |f| - f.puts "\nTotal QEMU duration: #{duration} minutes" - end +def start_qemu(hostname) + pwd_hostname = File.join(PWD, hostname) + FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) + FileUtils.cd(pwd_hostname) + system( + { 'hostname' => hostname, 'queues' => QUEUES }, + ENV['CCI_SRC'] + '/providers/qemu.sh' + ) +end - # sleep 5s is for fluentd to collect it's log - sleep(5) +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.backtrace + # if an exception occurs, request the next time after 30 seconds + sleep 25 + ensure + sleep 5 + end end end def save_pid(arr) - FileUtils.rm('pid') if File.exist?('pid') f = File.new('pid', 'a') arr.each do |i| f.puts(i) @@ -55,7 +102,7 @@ def multiqemu pids = [] NR_VM.to_i.times do |i| pid = Process.fork do - run i + loop_main("#{HOSTNAME}-#{i}") end pids << pid end -- 2.23.0

2 2

[PATCH lkp-tests 2/3] iso2rootfs: add step: config dns resolver of rootfs
by Yu Chuan 06 Nov '20

06 Nov '20

[Addition] Use config_rootfs as a function to maintain scalability. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index f864188a8b02..bf842108ae02 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -339,6 +339,22 @@ run_qcow2rootfs() ./run "${qcow2_path}" "${ROOTFS_DES_DIR}" } + +############ config rootfs ############ +config_dns_resolver() +{ + local resolv_conf_path="${ROOTFS_DES_DIR}/etc/resolv.conf" + cat <<-EOF > "${resolv_conf_path}" + nameserver 114.114.114.114 + nameserver 8.8.8.8 + EOF +} + +config_rootfs() +{ + config_dns_resolver +} + ############ test rootfs ############ get_qemu_efi_fd() { @@ -499,6 +515,8 @@ main() config_qcow2rootfs run_qcow2rootfs + config_rootfs + test_rootfs [ -z ${test_yaml} ] || submit_test_job_yaml -- 2.23.0

2 2

[PATCH v2 compass-ci] area: refactor multi-qemu to support queues parameter
by Xiao Shenwei 06 Nov '20

06 Nov '20

Usage: multi-qemu -n -c -q -n, --name HOSTNAME_PREFIX format: $tbox_group.$HOSTNAME -c, --count count how many VM do you need -q, --queues queues separated by "," -h, --help show this message example: ./multi-qemu -n vm-2p8g.taishan200-2280-2s48p-256g--a1 \ -c 20 \ -q vm-2p8g.taishan200-2280-2s48p-256g--a1,vm-2p8g.aarch64 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/multi-qemu | 95 +++++++++++++++++++++++++++++++++----------- 1 file changed, 71 insertions(+), 24 deletions(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index 23e0451..045160f 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -4,46 +4,93 @@ # frozen_string_literal: true require 'fileutils' +require 'optparse' -PWD = Dir.pwd +opt = {} +options = OptionParser.new do |opts| + opts.banner = 'Usage: multi-qemu -n -c -q' + + opts.separator '' + opts.on('-n HOSTNAME_PREFIX', '--name HOSTNAME_PREFIX', 'format: $tbox_group.$HOSTNAME') do |name| + opt['hostname_prefix'] = name + end + + opts.on('-c count', '--count count', 'how many VM do you need') do |num| + opt['nr_vm'] = num + end + + opts.on('-q queues', '--queues queues', 'separated by ","') do |queues| + opt['queues'] = queues + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.size.zero? + puts options + exit 1 +end + +options.parse!(ARGV) # Run multiple QEMU in parallel -HOSTNAME = ARGV[0] || "vm-2p8g--#{ENV['USER']}" -NR_VM = ARGV[1] || 1 +PWD = Dir.pwd +HOSTNAME = opt['hostname_prefix'] || "vm-2p8g.#{ENV['HOSTNAME']}" +NR_VM = opt['nr_vm'] || 1 +QUEUES = opt['queues'] || "#{ENV['HOSTNAME']}.#{RUBY_PLATFORM.split('-')[0]}" +LOG_DIR = '/srv/cci/serial/logs' -def run(seqno) - loop do - start_time = Time.new - hostname = "#{HOSTNAME}-#{seqno}" - log_file = "/srv/cci/serial/logs/#{hostname}" +def main(hostname) + start_time = record_runtime_log(hostname) + start_qemu(hostname) + record_runtime_log(hostname, start_time: start_time, is_start: false) +end +def record_runtime_log(hostname, start_time: Time.new, is_start: true) + log_file = "#{LOG_DIR}/#{hostname}" + if is_start File.open(log_file, 'w') do |f| # fluentd refresh time is 1s # let fluentd to monitor this file first sleep(2) f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting QEMU" end + return start_time + end + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal QEMU duration: #{duration} minutes" + end +end - pwd_hostname = File.join(PWD, hostname) - FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) - FileUtils.cd(pwd_hostname) - system( - { 'hostname' => hostname }, - ENV['CCI_SRC'] + '/providers/qemu.sh' - ) - - duration = ((Time.new - start_time) / 60).round(2) - File.open(log_file, 'a') do |f| - f.puts "\nTotal QEMU duration: #{duration} minutes" - end +def start_qemu(hostname) + pwd_hostname = File.join(PWD, hostname) + FileUtils.mkdir_p(pwd_hostname) unless File.exist?(pwd_hostname) + FileUtils.cd(pwd_hostname) + system( + { 'hostname' => hostname, 'queues' => QUEUES }, + ENV['CCI_SRC'] + '/providers/qemu.sh' + ) +end - # sleep 5s is for fluentd to collect it's log - sleep(5) +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.backtrace + # if an exception occurs, request the next time after 30 seconds + sleep 25 + ensure + sleep 5 + end end end def save_pid(arr) - FileUtils.rm('pid') if File.exist?('pid') f = File.new('pid', 'a') arr.each do |i| f.puts(i) @@ -55,7 +102,7 @@ def multiqemu pids = [] NR_VM.to_i.times do |i| pid = Process.fork do - run i + loop_main("#{HOSTNAME}-#{i}") end pids << pid end -- 2.23.0

4 8

[PATCH v5 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 06 Nov '20

06 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..f902fb0e --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,35 @@ +#!/bin/sh +# - runtime +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is an old version, It is both a benchmark and a stress test for the Linux kernel scheduler. +## It's main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes) which communicate via either sockets or pipes and time how long +## it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' + +[ -n "$runtime" ] || runtime=600 +[ -n "$iterations" ] || iterations=10 +[ -n "$loops" ] || loops=100 +[ -n "$nr_threads" ] || nr_threads=1 + +run_hackbench() +{ + # args: groups number, mode [process/thread], loops, ipc [pipe/socket] + hackbench_path=/lkp/benchmarks/hackbench-git/hackbench/hackbench + if [ ! -x /lkp/benchmarks/hackbench-git/hackbench/hackbench ]; then + echo "The hackbench executable file cannot be found." && exit 1 + fi + + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

3 4

[PATCH v6 compass-ci 3/3] assign-account: optimize answerback-email.rb
by Luan Shengde 06 Nov '20

06 Nov '20

1. disable use golbal variable apply_info [why] rubocop: Style/GlobalVars: Do not introduce global variables. [how] use local variables instead of global variables 2. fix OptionParser argument [why] when write this: opts.on('-e|--email email_address', 'appoint email address') will lead to could not identify the options ./answerback-email.rb --email xxx(a)163.com ./answerback-email.rb:72:in `<main>': invalid option: --email (OptionParser::InvalidOption) [how] write seprately like: opts.on('-e email_address', '--email email_address', 'appoint email address') 3. add transfer user info when execute apply account command [why] when applying account, the assign-account service need to write the my info to user's default config file ~/.config/compass-ci/defaults/account.yaml my_info: - my_email - my_name - my_uuid [how] transfer the user info along with the pub_key when sending apply account request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 87 +++++++++++--------- 1 file changed, 49 insertions(+), 38 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index bb8e809..cc49903 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -12,58 +12,63 @@ require 'mail' require 'set' require 'optparse' require_relative '../defconfig' +require_relative '../../lib/es_client' names = Set.new %w[ JUMPER_HOST JUMPER_PORT - SEND_MAIL_HOST_INTERNET - SEND_MAIL_PORT_INTERNET + SEND_MAIL_HOST + SEND_MAIL_PORT ] defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST_INTERNET'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT_INTERNET'] || 11312 +SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 -$apply_info = { +my_info = { 'my_email' => nil, + 'my_name' => nil, + 'my_uuid' => %x(uuidgen).chomp, 'my_ssh_pubkey' => nil } -def init_info(email_file) +def init_info(email_file, my_info) mail_content = Mail.read(email_file) - - $apply_info['my_email'] = mail_content.from[0] - $apply_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' - mail_content.part[1].body.decoded.gsub(/\r|\n/, '') - end - - $apply_info + my_info['my_email'] = mail_content.from[0] + my_info['my_name'] = mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + my_info['my_ssh_pubkey'] = if mail_content.part[1].filename == 'id_rsa.pub' + mail_content.part[1].body.decoded + end end options = OptionParser.new do |opts| - opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key_file] [--raw-email email_file]\n" + opts.banner = 'Usage: answerback-mail.rb [-e|--email email] ' + opts.banner += "[-s|--ssh-pubkey pub_key_file] [-f|--raw-email email_file]\n" opts.banner += " -e or -f is required\n" opts.banner += ' -s is optional when use -e' opts.separator '' opts.separator 'options:' - opts.on('-e|--email email_address', 'appoint email address') do |email_address| - $apply_info['my_email'] = email_address + opts.on('-e email_address', '--email email_address', 'appoint email address') do |email_address| + my_info['my_email'] = email_address + # when apply account with email address, will get no user name + my_info['my_name'] = '' end - opts.on('-s|--ssh-pubkey pub_key_file', 'ssh pub_key file, enable password-less login') do |pub_key_file| - $apply_info['my_ssh_pubkey'] = File.read(pub_key_file) + opts.on('-s pub_key_file', '--ssh-pubkey pub_key_file', \ + 'ssh pub_key file, enable password-less login') do |pub_key_file| + my_info['my_ssh_pubkey'] = File.read(pub_key_file) end - opts.on('-f|--raw-email email_file', 'email file') do |email_file| - init_info(email_file) + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + init_info(email_file, my_info) end - opts.on_tail('-h|--help', 'show this message') do + opts.on_tail('-h', '--help', 'show this message') do puts opts exit end @@ -71,10 +76,10 @@ end options.parse!(ARGV) -def build_message(email, acct_infos) +def build_message(email, account_info) message = <<~EMAIL_MESSAGE To: #{email} - Subject: jumper account is ready + Subject: [compass-ci] jumper account is ready Dear user: @@ -82,10 +87,10 @@ def build_message(email, acct_infos) You can use the following command to login the jumper server: login command: - ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} + ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} account password: - #{acct_infos['passwd']} + #{account_info['my_password']} regards compass-ci @@ -94,26 +99,32 @@ def build_message(email, acct_infos) return message end -def account_info(pub_key) - account_info_str = if pub_key.nil? - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') - else - %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - end +def apply_account(my_info) + account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{my_info.to_json}') JSON.parse account_info_str end -def send_account +def send_account(my_info) message = "No email address specified\n" - message += "use -e email_address add a email address\n" + message += "use -e to add a email address\n" message += 'or use -f to add a email file' - raise message if $apply_info['my_email'].nil? - - acct_info = account_info($apply_info['my_ssh_pubkey']) + raise message if my_info['my_email'].nil? - message = build_message($apply_info['my_email'], acct_info) + account_info = apply_account(my_info) + # for manually assign account, there will be no my_commit_url + # but the key is required for es + my_info['my_commit_url'] = '' + my_info['my_login_name'] = account_info['my_login_name'] + my_info.delete 'my_ssh_pubkey' + store_account_info(my_info) + message = build_message(my_info['my_email'], account_info) %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -send_account +def store_account_info(my_info) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(my_info['my_email'], my_info) +end + +send_account(my_info) -- 2.23.0

1 0

[PATCH v6 compass-ci 2/3] get_account_info: add config default yaml file
by Luan Shengde 06 Nov '20

06 Nov '20

when assign account, config the default yaml for user [why] easier for user to config the default yaml file [how] write received my info to the default yaml file ~/.config/compass-ci/default/account.yaml include: - my_email - my_name - my_uuid Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 73 ++++++++++++-------- 1 file changed, 44 insertions(+), 29 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 2f93d5b..87d2452 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -7,17 +7,17 @@ ACCOUNT_DIR dir layout: tree -├── assigned-users -│ ├── user1 -│ ├── user2 -│ ├── user3 -│ ├── ... -├── available-users -│ ├── user11 -│ ├── user12 -│ ├── user13 -│ ├── ... -└── jumper-info +|-- assigned-users +| |-- user1 +| |-- user2 +| |-- user3 +| |-- ... +|-- available-users +| |-- user11 +| |-- user12 +| |-- user13 +| |-- ... +|-- jumper-info assigned-users: store assigned user files available-users: store available user files @@ -44,6 +44,8 @@ the returned data for setup_jumper_account_info like: =end +require 'fileutils' + # get jumper and account info class AccountStorage ACCOUNT_DIR = '/opt/account_data/' @@ -93,35 +95,48 @@ class AccountStorage def setup_jumper_account_info account_info = read_account_info jumper_info = read_jumper_info - pub_key = @data['pub_key'] unless @data.nil? - - jumper_ip = jumper_info[0].chomp - jumper_port = jumper_info[1].chomp - account = account_info[0] - passwd = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + + login_name = account_info[0] + password = if pub_key.nil? + account_info[1] + else + 'Use pub_key to login' + end + jumper_account_info = { - 'account' => account, - 'passwd' => passwd, - 'jumper_ip' => jumper_ip, - 'jumper_port' => jumper_port + 'my_login_name' => login_name, + 'my_password' => password, + 'jumper_host' => jumper_info[0].chomp, + 'jumper_port' => jumper_info[1].chomp } - setup_authorized_key(account, pub_key) + setup_authorized_key(login_name, pub_key) unless pub_key.nil? + setup_default_yaml(login_name) + return jumper_account_info end - def setup_authorized_key(account, pub_key) - ssh_dir = File.join('/home/', account, '.ssh') + def setup_default_yaml(login_name) + default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') + FileUtils.mkdir_p default_yaml_dir + + File.open("#{default_yaml_dir}/account.yaml", 'a') do |file| + file.puts "my_email: #{@data['my_email']}" + file.puts "my_name: #{@data['my_name']}" + file.puts "my_uuid: #{@data['my_uuid']}" + end + %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") + end + + def setup_authorized_key(login_name, pub_key) + ssh_dir = File.join('/home/', login_name, '.ssh') Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') f.puts pub_key f.close File.chmod 0o600, 'authorized_keys' - %x(chown -R #{account}:#{account} #{ssh_dir}) + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) end end -- 2.23.0

1 0

[PATCH lkp-tests 3/3] iso2rootfs: solve the problem of multiple vmlinuz failures during test
by Yu Chuan 06 Nov '20

06 Nov '20

[Why] We use compass-ci/container/qcow2rootfs to generate rootfs from qcow2, but compass-ci updated the code, qcow2rootfs create a vmlinuz soft link at {rootfs_dir}/boot. As a result, when getting var(kernel_path), the vmlinuz soft link is also included, and our var(kernel_path) gets two identical vmlinuz. Error msg: + sed -i 's|KERNEL|"{rootfs_dir}/boot/vmlinuz-4.19.90-2010.2.0.0046.oe1.aarch64 {rootfs_dir}/boot/vmlinuz-4.19.90-2010.2.0.0046.oe1.aarch64"|g' {i2r_src}/env/iso2rootfs/kvm.sh sed: -e expression #1, char 133: unterminated `s' command [How] 1. When filtering vmlinuz: Before: realpath {rootfs_dir}/boot/vmlinuz* | grep -v rescue After: realpath {rootfs_dir}/boot/vmlinuz-* | grep -v rescue 2. Add a judgement, if the kernel num of result rootfs is not equal to 1, exit. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index bf842108ae02..2329715ef34b 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -371,8 +371,12 @@ test_rootfs() log_info "starting test rootfs ..." local test_script="${CGZ_PATH}/kvm.sh" - local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz*|grep -v rescue) + local kernel_path=$(realpath "${ROOTFS_DES_DIR}"/boot/vmlinuz-*|grep -v rescue) local initrd_lkp_path=$(realpath "${ROOTFS_DES_DIR}"/boot/initramfs.lkp*) + + local kernel_num=$(echo "$kernel_path" | wc -l) + [ "$kernel_num" == "1" ] || die "result rootfs have multiple vmlinuz, num: $kernel_num" + local root_path case ${rootfs_protocol} in "nfs") -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 06 Nov '20

06 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe..1faa1e9f 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -37,6 +37,23 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +check_pkgfile() +{ + PKGBUILD_TAG="-$PKGBUILD_TAG" + PKGBUILD_TAG_FILE="PKGBUILD$PKGBUILD_TAG" + + get_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 @@ -69,6 +86,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && check_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

1 0

[PATCH v2 compass-ci] sched: simplify cluster state updating
by Ren Wen 06 Nov '20

06 Nov '20

There are three steps when updating cluster state: 1) get cluster state from redis. 2) update cluster state. 3) rewrite to redis. Before: write to redis one job info once. After: write to redis more than one job info once. It will save time when writing more than one job info once. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a4b12b4..bff6090 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -58,11 +58,11 @@ class Sched return cluster_state end - # get -> modify -> set - def update_cluster_state(cluster_id, job_id, property, value) + # Update job info according to cluster id. + def update_cluster_state(cluster_id, job_id, job_info : Hash(String, String)) cluster_state = get_cluster_state(cluster_id) if cluster_state[job_id]? - cluster_state[job_id].merge!({property => value}) + cluster_state[job_id].merge!(job_info) @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) end end @@ -86,9 +86,9 @@ class Sched case request_state when "abort", "finished", "failed" # update node state only - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) @block_helper.block_until_finished(cluster_id) { cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) cluster_state == "ready" || cluster_state == "abort" @@ -96,7 +96,7 @@ class Sched return cluster_state when "wait_finish" - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) @@ -110,10 +110,11 @@ class Sched direct_ips = env.params.query["direct_ips"] direct_macs = env.params.query["direct_macs"] - update_cluster_state(cluster_id, job_id, "roles", node_roles) - update_cluster_state(cluster_id, job_id, "ip", node_ip) - update_cluster_state(cluster_id, job_id, "direct_ips", direct_ips) - update_cluster_state(cluster_id, job_id, "direct_macs", direct_macs) + job_info = {"roles" => node_roles, + "ip" => node_ip, + "direct_ips" => direct_ips, + "direct_macs" => direct_macs} + update_cluster_state(cluster_id, job_id, job_info) when "roles_ip" role = "server" role_state = get_role_state(cluster_id, role) -- 2.23.0

3 4

[PATCH v6 lkp-tests] tests/build-pkg: change location for *.cgz file
by Liu Shaofei 06 Nov '20

06 Nov '20

[why] the path of *.cgz file generated by build-pkg script cannot be obtained during test. So the path of cgz file need to be changed. [how] below is example for openeuler: - path: /srv/initrd/build-pkg/initramfs/openeuler/aarch64/20.03/adl-git/ - filename: . ├── 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz ├── df0584e46db9e7b412626deab10b923504745b2c.cgz ├── latest.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz ├── v1.0.cgz -> 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz └── v2.0.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz 0 directories, 5 files Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..ad8d4843 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,15 +23,21 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + # the same image is mounted to cifs and nfs, the generated cgz files + # are stored in the nfs directory. + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount $LKP_SERVER:$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } @@ -75,12 +83,27 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 +} + +create_softlink() +{ + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" } check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

4 4

[PATCH v3 compass-ci 6/6] container/webhook: add scripts for webhook
by Li Yuanchao 06 Nov '20

06 Nov '20

webhook.js will listen to webhook message from 'gitee.com' or 'github.com' if catch 'push' message, call push_hook.rb to send message to git-mirror through message queue Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/root/push_hook.rb | 15 +++++++++++++++ container/webhook/root/webhook.js | 22 ++++++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/webhook/root/push_hook.rb create mode 100755 container/webhook/root/webhook.js diff --git a/container/webhook/root/push_hook.rb b/container/webhook/root/push_hook.rb new file mode 100755 index 0000000..e467ccd --- /dev/null +++ b/container/webhook/root/push_hook.rb @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'bunny' + +connection = Bunny.new('amqp://172.17.0.1:5672') +connection.start +channel = connection.create_channel + +queue = channel.queue('web_hook') +message = ARGV[0] +queue.publish(message) +connection.close diff --git a/container/webhook/root/webhook.js b/container/webhook/root/webhook.js new file mode 100755 index 0000000..a98576c --- /dev/null +++ b/container/webhook/root/webhook.js @@ -0,0 +1,22 @@ +const http = require('http') +const spawn = require('child_process').spawn +const createHandler = require('git-webhook-handler') +// if users want to use webhook, they need to config webhook on their code warehouse('gitee.com' or 'github.com'), +// and use url: http://183.134.196.212:11301/webhook, secret: webhook@git. +const handler = createHandler({ path: '/webhook', secret: 'webhook@git' }) + +handler.on('error', function(err){ + console.error('Error:', err.message) +}) + +handler.on('push', function(event){ + console.log(event.payload.repository.url) + spawn('ruby', ['push_hook.rb', event.payload.repository.url]) +}) + +http.createServer(function(req, res){ + handler(req, res, function(err){ + res.statusCode = 404 + res.end('no such location') + }) +}).listen(11301) -- 2.23.0

1 0

[PATCH v3 compass-ci 5/6] container/webhook: start-depends
by Li Yuanchao 06 Nov '20

06 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start-depends | 1 + 1 file changed, 1 insertion(+) create mode 100755 container/webhook/start-depends diff --git a/container/webhook/start-depends b/container/webhook/start-depends new file mode 100755 index 0000000..5232abf --- /dev/null +++ b/container/webhook/start-depends @@ -0,0 +1 @@ +rabbitmq -- 2.23.0

1 0

[PATCH v3 compass-ci 4/6] container/webhook: start
by Li Yuanchao 06 Nov '20

06 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 container/webhook/start diff --git a/container/webhook/start b/container/webhook/start new file mode 100755 index 0000000..373396a --- /dev/null +++ b/container/webhook/start @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "webhook" + +cmd = %w[ + docker run + --restart=always + --name webhook + -d + -p 11301:11301 + node:webhook +] + +cmd += ['sh', '-c', 'node webhook.js'] +system(*cmd) -- 2.23.0

1 0

[PATCH v3 compass-ci 3/6] container/webhook: build
by Li Yuanchao 06 Nov '20

06 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/build | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/webhook/build diff --git a/container/webhook/build b/container/webhook/build new file mode 100755 index 0000000..bb5c0a7 --- /dev/null +++ b/container/webhook/build @@ -0,0 +1,6 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t node:webhook . + -- 2.23.0

1 0

[PATCH v3 compass-ci 2/6] container/webhook: Dockerfile
by Li Yuanchao 06 Nov '20

06 Nov '20

add container webhook Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/Dockerfile | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 container/webhook/Dockerfile diff --git a/container/webhook/Dockerfile b/container/webhook/Dockerfile new file mode 100644 index 0000000..9299bb6 --- /dev/null +++ b/container/webhook/Dockerfile @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM node:10.21.0-alpine +MAINTAINER Li Yuanchao <lyc163mail(a)163.com> + +ADD root / + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk upgrade && \ + apk add --no-cache git && \ + apk add npm ruby-dev + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc + +RUN npm install git-webhook-handler && \ + gem install bunny + -- 2.23.0

1 0

[PATCH v3 compass-ci 1/6] git_mirror: add webhook handler
by Li Yuanchao 06 Nov '20

06 Nov '20

webhook will send a message when user push. try to find the git_repo when get url of the repository. if found, push it to queue. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 65d6940..9d6a45d 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -88,10 +88,16 @@ class MirrorMain @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info + connection_init + handle_webhook + end + + def connection_init connection = Bunny.new('amqp://172.17.0.1:5672') connection.start channel = connection.create_channel @message_queue = channel.queue('new_refs') + @webhook_queue = channel.queue('web_hook') end def fork_stat_init(stat_key) @@ -287,3 +293,39 @@ class MirrorMain es_repo_update(git_repo) end end + +# main thread +class MirrorMain + def check_git_repo(git_repo, webhook_url) + return @git_info.key?(git_repo) && Array(@git_info[git_repo]['url'])[0] == webhook_url + end + + # example + # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware + # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc + # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + def get_git_repo(webhook_url) + fork_name, project = webhook_url.split('/')[-2, 2] + + git_repo = "#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project}/#{fork_name}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project[0]}/#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + puts "webhook: #{webhook_url} is not found!" + end + + def handle_webhook + Thread.new do + @webhook_queue.subscribe(block: true) do |_delivery, _properties, webhook_url| + git_repo = get_git_repo(webhook_url) + do_push(git_repo) if git_repo + sleep(0.1) + end + end + end +end -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests: add hackbench-git test script
by Zhang Yale 06 Nov '20

06 Nov '20

[Why] hackbench-git is an old version required by the customer. The hackbench-git usage: hackbench [-pipe] <num groups> [process|thread] [loops] So need to add new test script for hackbench-git. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/hackbench-git | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 tests/hackbench-git diff --git a/tests/hackbench-git b/tests/hackbench-git new file mode 100755 index 00000000..cd133d9e --- /dev/null +++ b/tests/hackbench-git @@ -0,0 +1,37 @@ +#!/bin/sh +# - runtime +# - nr_threads +# - mode +# - ipc +# - iterations +# - loops + +## Hackbench-git is an old version, It is both a benchmark and a stress test for the Linux kernel scheduler. +## It's main job is to create a specified number of schedulable entities pairs. +## (either threads or traditional processes) which communicate via either sockets or pipes and time how long +## it takes for each pair to send data back and forth. + +[ "$ipc" = 'pipe' ] && ipc_option='-pipe' + +[ -n "$runtime" ] || runtime=600 +[ -n "$iterations" ] || iterations=10 +[ -n "$loops" ] || loops=100 +[ -n "$nr_threads" ] || nr_threads=1 + +run_hackbench() +{ + # args: groups number, mode [process/thread], times + local hackbench_path + if [ -x /lkp/benchmarks/hackbench-git/hackbench/hackbench ]; then + hackbench_path=/lkp/benchmarks/hackbench-git/hackbench/hackbench + else + echo "The hackbench executable file cannot be found." && exit 1 + fi + + echo "--------$ipc $mode num=$nr_threads--------" + for i in $(seq 1 $iterations); do + log_cmd $hackbench_path $ipc_option $nr_threads $mode $loops + done +} + +run_hackbench -- 2.23.0

2 4

[PATCH v1 lab-z9] cluster: add cluster config file
by Zhang Yu 06 Nov '20

06 Nov '20

About config filename: "cs-s1-a102-c1": - cs : cluster - s1 : one server node - a102 : server node name - c1 : one client node Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- cluster/cs-s1-a102-c1 | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 cluster/cs-s1-a102-c1 diff --git a/cluster/cs-s1-a102-c1 b/cluster/cs-s1-a102-c1 new file mode 100644 index 0000000..af57123 --- /dev/null +++ b/cluster/cs-s1-a102-c1 @@ -0,0 +1,10 @@ +switch: Switch-P10 +ip0: 1 +nodes: + taishan200-2280-2s48p-512g--a102: + roles: [ server ] + macs: [ "44:67:47:c9:ea:37" ] + + taishan200-2280-2s48p-256g--a101: + roles: [ client ] + macs: [ "44:67:47:d7:6c:a3" ] -- 2.23.0

1 0

[PATCH compass-ci 1/6] git_mirror: add webhook handler
by Li Yuanchao 06 Nov '20

06 Nov '20

webhook will send a message when user push. try to find the git_repo when get url of the repository. if found, push it to queue. if not found, it maybe an illegal request, do nothing Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 65d6940..11cf3f1 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -88,10 +88,16 @@ class MirrorMain @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info + connection_init + handle_webhook + end + + def connection_init connection = Bunny.new('amqp://172.17.0.1:5672') connection.start channel = connection.create_channel @message_queue = channel.queue('new_refs') + @webhook_queue = channel.queue('web_hook') end def fork_stat_init(stat_key) @@ -287,3 +293,35 @@ class MirrorMain es_repo_update(git_repo) end end + +# main thread +class MirrorMain + def check_git_repo(git_repo, webhook_url) + return @git_info.key?(git_repo) && Array(@git_info[git_repo]['url'])[0] == webhook_url + end + + def get_git_repo(webhook_url) + strings = webhook_url.split('/') + project = strings[-1] + fork_name = strings[-2] + + git_repo = "#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project}/#{fork_name}" + return git_repo if check_git_repo(git_repo, webhook_url) + + git_repo = "#{project[0]}/#{project}/#{project}" + return git_repo if check_git_repo(git_repo, webhook_url) + end + + def handle_webhook + Thread.new do + @webhook_queue.subscribe(block: true) do |_delivery, _properties, webhook_url| + git_repo = get_git_repo(webhook_url) + do_push(git_repo) if git_repo + sleep(0.1) + end + end + end +end -- 2.23.0

4 7

[PATCH v6 lkp-tests 1/2] jobs/iozone-bs.yaml: combine iozone's multiple -i parameter to single
by Lu Kaiyi 06 Nov '20

06 Nov '20

[why] avoid explosion of parameter for iozone-bs.yaml Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone-bs.yaml | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/jobs/iozone-bs.yaml b/jobs/iozone-bs.yaml index e2cd9f48..026b97e2 100644 --- a/jobs/iozone-bs.yaml +++ b/jobs/iozone-bs.yaml @@ -2,9 +2,7 @@ suite: iozone category: benchmark file_size: 4g -write_rewrite: true -read_reread: true -random_read_write: true +test: write,read,rand_rw block_size: - 64k -- 2.23.0

2 2

[PATCH lkp-tests] pkg: add nginx PKGBUILD
by Wei Jihui 06 Nov '20

06 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/nginx/PKGBUILD | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100644 pkg/nginx/PKGBUILD diff --git a/pkg/nginx/PKGBUILD b/pkg/nginx/PKGBUILD new file mode 100644 index 00000000..00b30118 --- /dev/null +++ b/pkg/nginx/PKGBUILD @@ -0,0 +1,23 @@ +pkgname=nginx +pkgver=1.14 +pkgrel=2 +arch=('i386' 'x86_64') +url="https://nginx.org" +license=('BSD') +source=("https://nginx.org/download/nginx-${pkgver}.${pkgrel}.tar.gz") +md5sums=('SKIP') + +build() +{ + benchmark_path="/lkp/benchmarks/${pkgname}" + mkdir -p "${benchmark_path}" + cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + ./configure --with-http_ssl_module --prefix="${benchmark_path}" + make +} + +package() +{ + cd "${srcdir}/${pkgname}-${pkgver}.${pkgrel}" + make install +} -- 2.23.0

1 0

[PATCH v6 lkp-tests 2/2] tests/iozone: change the way of parsing parameter
by Lu Kaiyi 06 Nov '20

06 Nov '20

[why] iozone-bs.yaml has combined multiple parameter to single, so, iozone tool also need change the way of parsing parameter. [how] change the way of parsing parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 66 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 38 insertions(+), 28 deletions(-) diff --git a/tests/iozone b/tests/iozone index 88a92a18..b6a1e206 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,42 +1,52 @@ #!/bin/sh # - block_size # - file_size -# - write_rewrite -# - read_reread -# - random_read_write -# - read_backwards -# - rewrite_record -# - stride_read -# - fwrite_refwrite -# - fread_refread -# - random_mix -# - pwrite_repwrite -# - pread_repread -# - pwritev_repwritev -# - preadv_repreadv +# - test ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. . $LKP_SRC/lib/reproduce-log.sh - args="iozone" if [ -n "$block_size" ]; then args+=" -r $block_size" - [ -n "$file_size" ] && args+=" -s $file_size" - [ -n "$write_rewrite" ] && args+=" -i 0" - [ -n "$read_reread" ] && args+=" -i 1" - [ -n "$random_read_write" ] && args+=" -i 2" - [ -n "$read_backwards" ] && args+=" -i 3" - [ -n "$rewrite_record" ] && args+=" -i 4" - [ -n "$stride_read" ] && args+=" -i 5" - [ -n "$fwrite_refwrite" ] && args+=" -i 6" - [ -n "$fread_refread" ] && args+=" -i 7" - [ -n "$random_mix" ] && args+=" -i 8" - [ -n "$pwrite_repwrite" ] && args+=" -i 9" - [ -n "$pread_repread" ] && args+=" -i 10" - [ -n "$pwritev_repwritev" ] && args+=" -i 11" - [ -n "$preadv_repreadv" ] && args+=" -i 12" + [ -n "$file_size" ] && args+=" -s $file_size" + OLD_IFS="$IFS" + # reset IFS to split $test by "," and then restore default + IFS="," + array=($test) + IFS="$OLD_IFS" + for ele in ${array[@]} + do + case $ele in + "write") args+=" -i 0" + ;; + "read") args+=" -i 1" + ;; + "rand_rw") args+=" -i 2" + ;; + "backwards") args+=" -i 3" + ;; + "record") args+=" -i 4" + ;; + "stride") args+=" -i 5" + ;; + "fwrite") args+=" -i 6" + ;; + "fread") args+=" -i 7" + ;; + "rand_mix") args+=" -i 8" + ;; + "pwrite") args+=" -i 9" + ;; + "pread") args+=" -i 10" + ;; + "pwritev") args+=" -i 11" + ;; + "preadv") args+=" -i 12" + ;; + esac + done else args+=" -a" fi -- 2.23.0

1 0

[PATCH v2 compass-ci 3/6] container/webhook: build
by Li Yuanchao 06 Nov '20

06 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/build | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/webhook/build diff --git a/container/webhook/build b/container/webhook/build new file mode 100755 index 0000000..bb5c0a7 --- /dev/null +++ b/container/webhook/build @@ -0,0 +1,6 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t node:webhook . + -- 2.23.0

3 3

[PATCH v6 compass-ci 1/2] auto group jobs with user-defined template
by Lu Weitao 06 Nov '20

06 Nov '20

auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow is: load compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch do: auto group jobs_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 122 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 121 insertions(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..40191a9 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,123 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + jobs_list = extract_jobs_list(jobs_list) + get_group_by_template(jobs_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + new_job = get_new_job(job, metrics) + next if new_job.empty? + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + value = find_param_in_job(job, param) + group_key_list << value if value + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +def find_param_in_job(job, param) + return job[param] if job.key?(param) + + job.each_value do |v| + return v[param] if v.is_a?(Hash) && v.key?(param) + end +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job(job, metrics) + return {} unless job['stats'] + + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + + new_job +end -- 2.23.0

2 2

[PATCH lkp-tests] add the queue parameter for multi-qemu job
by Xiao Shenwei 06 Nov '20

06 Nov '20

usage: on multi-qemu.yaml multi-qemu-0: nr_vm: 1 tbox_group: vm-2p8g queues: vm-2p8g.taishan200-2280-2s48p-256g--a1 Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/multi-qemu | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/daemon/multi-qemu b/daemon/multi-qemu index b407251f..8d9dd38d 100755 --- a/daemon/multi-qemu +++ b/daemon/multi-qemu @@ -1,6 +1,7 @@ #!/bin/sh # - tbox_group # - nr_vm +# - queues # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. @@ -8,10 +9,10 @@ multi_qemu() { export CCI_SRC=/c/compass-ci - local hostname=$tbox_group--$HOSTNAME + local hostname=$tbox_group.$HOSTNAME cd "$CACHE_DIR" - $CCI_SRC/providers/multi-qemu "$hostname" "$nr_vm" + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" } multi_qemu -- 2.23.0

3 3

[PATCH v7 lkp-tests] tests/build-pkg: change location for *.cgz file
by Liu Shaofei 06 Nov '20

06 Nov '20

[why] the path of *.cgz file generated by build-pkg script cannot be obtained during test. So the path of cgz file need to be changed. [how] Example for openeuler is as below: - path: /srv/initrd/build-pkg/initramfs/openeuler/aarch64/20.03/adl-git/ - filename: . ├── 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz ├── df0584e46db9e7b412626deab10b923504745b2c.cgz ├── latest.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz ├── v1.0.cgz -> 92c89f904a14d4bfc25abcd2c3d5c2da35a589d6.cgz └── v2.0.cgz -> df0584e46db9e7b412626deab10b923504745b2c.cgz 0 directories, 5 files Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index b10b1497..ad8d4843 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -2,10 +2,12 @@ # - os # - os_arch # - os_version +# - os_mount # - pkgbuild_repo # - upstream_repo # - upstream_url # - upstream_commit +# - upstream_tag ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, @@ -21,15 +23,21 @@ check_vars() [ -n "$os_version" ] || die "os_version is empty" [ -n "$pkgbuild_repo" ] || die "pkgbuild_repo is empty" [ -n "$upstream_commit" ] || die "upstream_commit is empty" + [ -n "$os_mount" ] || die "os_mount is empty" } mount_dest() { + # the same image is mounted to cifs and nfs, the generated cgz files + # are stored in the nfs directory. + [[ "$os_mount" = "cifs" ]] && os_mount="nfs" + pack_to=${os_mount}/${os}/${os_arch}/${os_version} + PKG_MNT=/initrd/build-pkg mkdir -p "$PKG_MNT" [ -n "$LKP_SERVER" ] && { - mount $LKP_SERVER:$PKG_MNT $PKG_MNT || die "Failed to run mount" + mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } } @@ -75,12 +83,27 @@ build_source_pkg() sed -i "s|^source=.*|${upstream_source}|g" PKGBUILD } - cgz_name="$PKG_MNT/${pkgname}/${upstream_commit}.cgz" + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ - $LKP_SRC/sbin/makepkg -A --check --config $LKP_SRC/etc/makepkg.conf 2>&1 + $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 +} + +create_softlink() +{ + local soft_path="$PKG_MNT/${pack_to}/${pkgname}" + local bm_name=$(basename $(realpath ${cgz_name})) + + if [ -n "$upstream_tag" ]; then + echo "create soft link: ${upstream_tag}.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/${upstream_tag}.cgz" + fi + + echo "create soft link: latest.cgz -> ${bm_name}" + ln -sf "${bm_name}" "${soft_path}/latest.cgz" } check_vars mount_dest request_pkg build_source_pkg +create_softlink -- 2.23.0

1 0

[PATCH v3 compass-ci] sbin/es-repo-mapping.sh: create script that generates "repo" mapping
by Zhang Yuhang 06 Nov '20

06 Nov '20

[why] We need a script to create "repo" mapping in es. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- sbin/es-repo-mapping.sh | 47 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 sbin/es-repo-mapping.sh diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh new file mode 100755 index 0000000..4605051 --- /dev/null +++ b/sbin/es-repo-mapping.sh @@ -0,0 +1,47 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Check if "repo" index already exists. +status_code=$(curl -sIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) +[ $status_code -eq 200 ] && echo '"repo" index already exists.' && exit + +# Create "repo" index. +echo 'Start to create "repo" index.' +curl -H 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' +{ + "mappings": { + "_doc": { + "properties": { + "git_repo": { + "type": "keyword" + }, + "pkgbuild_repo": { + "type": "keyword" + }, + "url": { + "type": "keyword" + }, + "fetch_time": { + "type": "keyword" + }, + "new_refs_time": { + "type": "keyword" + }, + "offset_fetch": { + "type": "long" + }, + "offset_new_refs": { + "type": "long" + }, + "priority": { + "type": "long" + }, + "queued": { + "type": "boolean" + } + } + } + } +}' +[ $? -ne 0 ] && echo 'Failed to create "repo" index.' -- 2.23.0

1 0

[PATCH v3 lkp-tests] add testcase for sysbench-threads-git
by Hu Xuejiao 06 Nov '20

06 Nov '20

[why] old version package needs to debug by new testcase Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/sysbench-threads-git | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 tests/sysbench-threads-git diff --git a/tests/sysbench-threads-git b/tests/sysbench-threads-git new file mode 100755 index 00000000..df1bb5cf --- /dev/null +++ b/tests/sysbench-threads-git @@ -0,0 +1,18 @@ +#!/bin/sh +# - nr_threads +# - thread_yields +# - thread_locks + +. "$LKP_SRC/lib/reproduce-log.sh" + +: "${nr_threads:=2}" +: "${thread_yields:=100}" +: "${thread_locks:=2}" + +args=( + --threads=$nr_threads + --thread-yields=$thread_yields + --thread-locks=$thread_locks +) + +log_cmd sysbench --test=threads "${args[@]}" run -- 2.23.0

2 1

[PATCH v2 compass-ci 2/6] container/webhook: Dockerfile
by Li Yuanchao 06 Nov '20

06 Nov '20

add container webhook Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/Dockerfile | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 container/webhook/Dockerfile diff --git a/container/webhook/Dockerfile b/container/webhook/Dockerfile new file mode 100644 index 0000000..61d7f07 --- /dev/null +++ b/container/webhook/Dockerfile @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM node:10.21.0-alpine +MAINTAINER Li Yuanchao <lyc163mail(a)163.com> + +ADD root / + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk upgrade && \ + apk add --no-cache git && \ + apk add npm ruby-dev + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc + +RUN npm install git-webhook-handler &&\ + gem install bunny + -- 2.23.0

2 1

[PATCH lkp-tests] lib/job.rb: change the way get hosts file
by Li Ping 06 Nov '20

06 Nov '20

[problem]: submit test.yaml testbox=vm-2p8g hosts_file not exist: /home/lys/lkp-tests/hosts/vm-2p8g, maybe need check testbox field submit test.yaml, got job_id=crystal.114424 but there is no need to puts above error message, if we have got the hosts file from lab-z9. And even not got hosts file, submit job also will success. submit test.yaml testbox=vm-2p8g.hjhjgg hosts_file not exist: /home/lys/lkp-tests/hosts/vm-2p8g.hjhjgg, maybe need check testbox field submit test.yaml, got job_id=crystal.114428 --- lib/job.rb | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 9fc7b17f..01719c2e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,20 +334,14 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + return lab_hosts_file if lab_hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.file?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + raise ArgumentError, "hosts file not exist: #{hosts_file_name}, maybe need check testbox field" end def include_files -- 2.23.0

1 0

[PATCH compass-ci] sched: simplify cluster state updating
by Ren Wen 06 Nov '20

06 Nov '20

Three steps when updating cluster state: 1) get cluster state from redis. 2) update cluster state. 3) rewrite to redis. Before: write to redis one job info once. After: write to redis more job infos once. It will save time when writing more than one job info once. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a4b12b4..72f4e0b 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -58,11 +58,11 @@ class Sched return cluster_state end - # get -> modify -> set - def update_cluster_state(cluster_id, job_id, property, value) + # Update job's infos according to cluster id. + def update_cluster_state(cluster_id, job_id, job_infos : Hash(String, String)) cluster_state = get_cluster_state(cluster_id) if cluster_state[job_id]? - cluster_state[job_id].merge!({property => value}) + cluster_state[job_id].merge!(job_infos) @redis.hash_set("sched/cluster_state", cluster_id, cluster_state.to_json) end end @@ -86,9 +86,9 @@ class Sched case request_state when "abort", "finished", "failed" # update node state only - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) when "wait_ready" - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) @block_helper.block_until_finished(cluster_id) { cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) cluster_state == "ready" || cluster_state == "abort" @@ -96,7 +96,7 @@ class Sched return cluster_state when "wait_finish" - update_cluster_state(cluster_id, job_id, "state", states[request_state]) + update_cluster_state(cluster_id, job_id, {"state" => states[request_state]}) while 1 sleep(10) cluster_state = sync_cluster_state(cluster_id, job_id, states[request_state]) @@ -110,10 +110,11 @@ class Sched direct_ips = env.params.query["direct_ips"] direct_macs = env.params.query["direct_macs"] - update_cluster_state(cluster_id, job_id, "roles", node_roles) - update_cluster_state(cluster_id, job_id, "ip", node_ip) - update_cluster_state(cluster_id, job_id, "direct_ips", direct_ips) - update_cluster_state(cluster_id, job_id, "direct_macs", direct_macs) + job_infos = {"roles" => node_roles, + "ip" => node_ip, + "direct_ips" => direct_ips, + "direct_macs" => direct_macs} + update_cluster_state(cluster_id, job_id, job_infos) when "roles_ip" role = "server" role_state = get_role_state(cluster_id, role) -- 2.23.0

2 2

[PATCH lkp-tests] cluster: add a cluster config file
by Zhang Yu 06 Nov '20

06 Nov '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- cluster/cs-lkp-hsw-ep5 | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/cluster/cs-lkp-hsw-ep5 b/cluster/cs-lkp-hsw-ep5 index 448b418a..ba91b92e 100644 --- a/cluster/cs-lkp-hsw-ep5 +++ b/cluster/cs-lkp-hsw-ep5 @@ -1,7 +1,10 @@ -lkp-hsw-ep5: - roles: [ server ] - macs: [ "ec:f4:bb:cb:7b:92" ] +switch: switch01 +ip0: 1 +nodes: + lkp-hsw-ep5: + roles: [ server ] + macs: [ "ec:f4:bb:cb:7b:92" ] -lkp-hsw-ep2: - roles: [ client ] - macs: [ "ec:f4:bb:cb:54:92" ] + lkp-hsw-ep2: + roles: [ client ] + macs: [ "ec:f4:bb:cb:54:92" ] -- 2.23.0

2 2

[PATCH v4 compass-ci 2/2] add template compare mode
by Lu Weitao 05 Nov '20

05 Nov '20

add template compare mode in sbin/compare(command-line tool) by: - load and parse compare_template.yaml - call lib/es_query.rb, query jobs data set from ES - call lib/matrix.rb, convert jobs data set to group matrices - call lib/compare_matrixes.rb, use groups matrices to compare background: To support compare with user-defined template feature, the work-flow is: load compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results example: compare -t/--template compare_template.yaml, the yaml like: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s output:{ "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "bs|test_size", "tables": { "4k|1G": { "fio.write_iops": { "average": { "openeuler 20.03": 308298.5524761111, "debian sid": 220289.07563 }, "standard_deviation": { "openeuler 20.03": null, "debian sid": null }, "change": { "debian sid vs openeuler 20.03": -28.5 } } }, "16k|1G": {...}, ... } } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++ sbin/compare | 10 +++++++- 2 files changed, 76 insertions(+), 1 deletion(-) diff --git a/lib/compare.rb b/lib/compare.rb index 2837edc..645f682 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -6,6 +6,7 @@ require_relative './es_query.rb' require_relative './matrix2.rb' require_relative './compare_matrixes.rb' require_relative './constants.rb' +require 'yaml' # ------------------------------------------------------------------------------------------- # compare_matrices_list @@ -66,3 +67,69 @@ def create_groups_matrices_list(conditions, dims) query_results = es.multi_field_query(conditions) combine_group_query_data(query_results, dims) end + +# ------------------------------------------------------------------------------------------- +# compare with user-defined compare_template.yaml +# compare_temlpate.yaml sample: +# compare_metrics: +# - fio.write_iops +# - fio.read_iops +# filter: +# suite: +# - fio-basic +# os_arch: +# - aarch64 +# - x86 +# compare_dimensions: +# - os: debian +# os_version: sid +# - os: openeuler +# os_version: 20.03 +# x_params: +# - bs +# - test_size +# title: Hackbench Performance Testing +# unit: KB/s +# + +def compare_by_template(template) + template_params = load_template(template) + groups_matrices = create_groups_matrices(template_params) + compare_results = compare_metrics_values(groups_matrices) + show_compare_result(compare_results, template_params) +end + +def load_template(template) + unless File.file?(template) + warn 'template does not exist' + exit + end + YAML.load_file(template) +end + +# input: template_params: Hash +# eg: +# { +# "compare_metrics"=>["fio.write_iops", "fio.read_iops"], +# "filter"=>[ +# {"suite"=>["fio-bisic"]}, +# {"os_arch"=>["aarch_64"]} +# ], +# "compare_dimensions"=>[ +# {"os"=>"openeuler", "os_version"=>20.03}, +# {"os"=>"centos", "os_version"=>7.6} +# ], +# "x_params"=>["block_size", "package_size"], +# "title"=>"Hackbench Performance Testing", +# "unit"=>"KB/s" +# } +def create_groups_matrices(template_params) + es = ESQuery.new + query_results = es.multi_field_query(template_params['filter']) + combine_group_jobs_list( + query_results, + template_params['x_params'], + template_params['compare_dimensions'], + template_params['compare_metrics'] + ) +end diff --git a/sbin/compare b/sbin/compare index cf3980c..b13682e 100755 --- a/sbin/compare +++ b/sbin/compare @@ -6,6 +6,7 @@ # Usage: # compare "conditions_1" "conditions_2" ... -c "common_conditions" # compare "conditions" -d "dimensions" +# compare -t template.yaml # Eg: # compare "id=6000,6001" "id=7000,7001" # compare "commit=a12d232e" "commit=b3bacc31" @@ -21,6 +22,7 @@ is_group = false dimensions = nil colorful = nil options = {} +template = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: compare "conditions" ... [option]' @@ -45,6 +47,10 @@ opt_parser = OptionParser.new do |opts| colorful = color end + opts.on('-t', '--template template', 'compare with user-defined template') do |t| + template = t + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -60,7 +66,9 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful -if is_group +if template + compare_by_template(template) +elsif is_group compare_group(argv, dimensions, options) else compare_matrices_list(argv, common_conditions, options) -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] compare values by each metrics
by Lu Weitao 05 Nov '20

05 Nov '20

compare values by each metrics based on groups matrices, and format compare result as echart data_set background: To support compare with user-defined template feature, the work-flow is: load compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch do: compare_values by each metrics ---> format/show results Signed-off-by: Lu Weitao <luweitaobe(a)163.com> Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_formatter.rb | 18 +++++++ lib/compare_matrixes.rb | 103 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 121 insertions(+) create mode 100644 lib/compare_formatter.rb diff --git a/lib/compare_formatter.rb b/lib/compare_formatter.rb new file mode 100644 index 0000000..3d82550 --- /dev/null +++ b/lib/compare_formatter.rb @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ or GPL-2.0 +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# ---------------------------------------------------------------------------------------------------- +# format compare results for a specific format +# + +def format_for_echart(metrics_compare_results, template_params) + echart_result = {} + echart_result['title'] = template_params['title'] + echart_result['unit'] = template_params['unit'] + x_params = template_params['x_params'] + echart_result['x_name'] = x_params.join('|') if x_params + echart_result['tables'] = metrics_compare_results + + echart_result +end diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 078028a..027c178 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -6,6 +6,7 @@ LKP_SRC ||= ENV['LKP_SRC'] || File.dirname(__dir__) require 'set' require 'json/ext' require_relative 'themes' +require_relative './compare_formatter.rb' require "#{LKP_SRC}/lib/stats" FAILURE_PATTERNS = IO.read("#{LKP_SRC}/etc/failure").split("\n") @@ -399,6 +400,108 @@ def compare_group_matrices(group_matrices, options) result_str end +# input: groups_matrices +# { +# group_key_1 => { +# dimension_1 => matrix_1, (openeuler 20.03) +# dimension_2 => matrix_2, (openeuler 20.09) +# dimension_3 => matrix_3, (centos 7.6) +# }, +# group_key_2 => {...} +# } +# +# output: compare_metrics_values +# { +# group_key_1 => { +# metric_1 => { +# 'average' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'standard_deviation' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'change' => { +# 'dimension_2 vs dimension_1' => xxx +# 'dimension_3 vs dimension_1' => xxx +# 'dimension_3 vs dimension_2' => xxx +# } +# }, +# metric_2 => {...} +# } +# } +def compare_metrics_values(groups_matrices) + metrics_compare_values = {} + groups_matrices.each do |group_key, dimensions| + metrics_compare_values[group_key] = get_metric_values(dimensions) + end + + metrics_compare_values +end + +def get_metric_values(dimensions) + metrics_values = {} + dimensions.each do |dim, matrix| + matrix.each do |metric, values| + assign_metric_values(metrics_values, dim, metric, values) + end + end + assign_metric_change(metrics_values) + + metrics_values +end + +def assign_metric_values(metrics_values, dim, metric, values) + metrics_values[metric] ||= {} + metrics_values[metric]['average'] ||= {} + metrics_values[metric]['standard_deviation'] ||= {} + metric_value = get_values(values, true) + metrics_values[metric]['average'][dim] = metric_value[:average] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] +end + +def assign_metric_change(metrics_values) + metrics_values.each do |metric, values| + metrics_values[metric]['change'] = {} + next if values['average'].size < 2 + + dimension_list = values['average'].keys + change_titles = get_change_titles(dimension_list) + change_titles.each do |base_dimension, challenge_dimension| + change = get_compare_value(values['average'][base_dimension], values['average'][challenge_dimension], true) + values['change'] = { "#{challenge_dimension} vs #{base_dimension}" => change } + end + end +end + +# input: dimension_list +# eg: ['openeuler 20.03', 'debian sid', 'centos 7.6'] +# output: Array(base_dimension: String, challenge_dimension: String) +# [ +# ['openeuler 20.03', 'debian sid'], +# ['openeuler 20.03', 'centos 7.6'], +# ['debian sid', 'centos 7.6'] +# ] +def get_change_titles(dimension_list) + dims = [] + dimension_list_size = dimension_list.size + (1..dimension_list_size - 1).each do |i| + (i..dimension_list_size - 1).each do |j| + dims << [dimension_list[i - 1], dimension_list[j]] + end + end + + dims +end + +def show_compare_result(metrics_compare_results, template_params) + echart_results = format_for_echart(metrics_compare_results, template_params) + print JSON.pretty_generate(echart_results) +end + # Format Fields def format_fails_runs(fails, runs) -- 2.23.0

1 0

[PATCH v6 compass-ci 2/2] create matrices-groups according to group jobs
by Lu Weitao 05 Nov '20

05 Nov '20

create matrices-groups according to auto group job_list background: To support compare with user-defined template feature, the work-flow is: load compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch do: create groups_matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 05f96dc..26f07fa 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -120,3 +120,28 @@ def combine_group_query_data(query_data, dims) end groups end + +# input: +# 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() +# 2. params from user's template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# return: group_matrix of Hash(String, Hash(String, matrix)) +def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) + job_list = query_data['hits']['hits'] + groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) + groups.each do |group_key, dims| + dims.each do |dim_key, jobs| + groups[group_key][dim_key] = create_matrix(jobs) + end + end + + groups +end -- 2.23.0

1 0

[PATCH v5 compass-ci 1/2] auto group jobs with user-defined template
by Lu Weitao 05 Nov '20

05 Nov '20

auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow is: load compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch do: auto group jobs_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..cc2c314 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + jobs = extract_jobs_list(jobs_list) + get_group_by_template(jobs, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + new_job = get_new_job(job, metrics) + next if new_job.empty? + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + value = find_param_in_job(job, param) + group_key_list << value if value + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +def find_param_in_job(job, param) + return job[param] if job.key?(param) + + job.each_value do |v| + return v[param] if v.is_a?(Hash) && v.key?(param) + end +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job(job, metrics) + return {} unless job['stats'] + + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0

3 4

[PATCH compass-ci] monitoring/filter.cr: query key support regular match
by Wu Zhende 05 Nov '20

05 Nov '20

[Why] Can use regular match in query's keys. [Example] 1: monitor job.*= query=>{"job.*":null} if one log's key include "job", will be matched. 2: monitor .*= this can match all logs Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 88702b6..8e672e0 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,7 +56,8 @@ class Filter def match_query(query : Hash(String, JSON::Any), msg : Hash(String, JSON::Any)) query.each do |key, value| - return false unless msg.has_key?(key) + key = find_real_key(key, msg.keys) unless msg.has_key?(key) + return false unless key values = value.as_a next if values.includes?(nil) || values.includes?(msg[key]?) @@ -66,6 +67,12 @@ class Filter return true end + private def find_real_key(rule, keys) + keys.each do |key| + return key if key.to_s =~ /#{rule}/ + end + end + private def regular_match(rules, string) rules.each do |rule| return true if string =~ /#{rule}/ -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] auto group jobs with user-defined template
by Lu Weitao 05 Nov '20

05 Nov '20

auto group jobs(query form ES) with user-defined template background: To support compare with user-defined template feature, the work-flow are: load_compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch is the step: auto group jobs_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 120 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 119 insertions(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..97827cd 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -2,7 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -# Exammple: +# Example: # Input: jobs_list. The results of ES query. # # eg: [ jobs1, jobs2, ...... ] @@ -113,3 +113,121 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto_group_by_template: auto group job_list by user's template +# Example: +# Input: +# 1. jobs_list. +# 2. params from user's template that include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4K|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'centos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list) + get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + new_job = get_new_job(job, metrics) + next if new_job.empty? + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + value = find_param_in_job(job, param) + group_key_list << value if value + end + return nil if group_key_list.size < group_params.size || group_key_list.empty? + + group_key_list.join('|') +end + +def find_param_in_job(job, param) + return job[param] if job.key?(param) + + job.each_value do |v| + return v[param] if v.is_a?(Hash) && v.key?(param) + end +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + dimensions.each do |dim| + dim.each do |key, value| + if job[key] == value + dimension_list << value + end + end + return nil if !dimension_list.empty? && dimension_list.size < dim.size + end + return nil if dimension_list.empty? + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job(job, metrics) + return {} unless job['stats'] + + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0

3 8

[PATCH v2 compass-ci 6/6] container/webhook: add scripts for webhook
by Li Yuanchao 05 Nov '20

05 Nov '20

webhook.js will listen to webhook message from 'gitee.com' or 'github.com' if catch 'push' message, call push_hook.rb to send message to git-mirror through message queue Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/root/push_hook.rb | 15 +++++++++++++++ container/webhook/root/webhook.js | 22 ++++++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/webhook/root/push_hook.rb create mode 100755 container/webhook/root/webhook.js diff --git a/container/webhook/root/push_hook.rb b/container/webhook/root/push_hook.rb new file mode 100755 index 0000000..e467ccd --- /dev/null +++ b/container/webhook/root/push_hook.rb @@ -0,0 +1,15 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'bunny' + +connection = Bunny.new('amqp://172.17.0.1:5672') +connection.start +channel = connection.create_channel + +queue = channel.queue('web_hook') +message = ARGV[0] +queue.publish(message) +connection.close diff --git a/container/webhook/root/webhook.js b/container/webhook/root/webhook.js new file mode 100755 index 0000000..a98576c --- /dev/null +++ b/container/webhook/root/webhook.js @@ -0,0 +1,22 @@ +const http = require('http') +const spawn = require('child_process').spawn +const createHandler = require('git-webhook-handler') +// if users want to use webhook, they need to config webhook on their code warehouse('gitee.com' or 'github.com'), +// and use url: http://183.134.196.212:11301/webhook, secret: webhook@git. +const handler = createHandler({ path: '/webhook', secret: 'webhook@git' }) + +handler.on('error', function(err){ + console.error('Error:', err.message) +}) + +handler.on('push', function(event){ + console.log(event.payload.repository.url) + spawn('ruby', ['push_hook.rb', event.payload.repository.url]) +}) + +http.createServer(function(req, res){ + handler(req, res, function(err){ + res.statusCode = 404 + res.end('no such location') + }) +}).listen(11301) -- 2.23.0

1 0

[PATCH v2 compass-ci 5/6] container/webhook: start-depends
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start-depends | 2 ++ 1 file changed, 2 insertions(+) create mode 100755 container/webhook/start-depends diff --git a/container/webhook/start-depends b/container/webhook/start-depends new file mode 100755 index 0000000..38fc696 --- /dev/null +++ b/container/webhook/start-depends @@ -0,0 +1,2 @@ +rabbitmq +es -- 2.23.0

1 0

[PATCH v2 compass-ci 4/6] container/webhook: start
by Li Yuanchao 05 Nov '20

05 Nov '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/start | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100755 container/webhook/start diff --git a/container/webhook/start b/container/webhook/start new file mode 100755 index 0000000..373396a --- /dev/null +++ b/container/webhook/start @@ -0,0 +1,20 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "webhook" + +cmd = %w[ + docker run + --restart=always + --name webhook + -d + -p 11301:11301 + node:webhook +] + +cmd += ['sh', '-c', 'node webhook.js'] +system(*cmd) -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 05 Nov '20

05 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe..1faa1e9f 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -37,6 +37,23 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +check_pkgfile() +{ + PKGBUILD_TAG="-$PKGBUILD_TAG" + PKGBUILD_TAG_FILE="PKGBUILD$PKGBUILD_TAG" + + get_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 @@ -69,6 +86,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && check_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

1 0

[PATCH compass-ci] job.cr: only initramfs needs modules.cgz and headers.cgz
by Xu Xijian 05 Nov '20

05 Nov '20

[why] Only initramfs needs modules.cgz and headers.cgz. [how] Judge if it boots with initramfs before setting the values of linux_modules_initrd and linux_headers_initrd. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 6309809..07873ab 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -302,8 +302,10 @@ class Job boot_dir = "#{SRV_OS}/#{os_dir}/boot" suffix = "-#{kernel_version}" if self["kernel_version"]? self["linux_vmlinuz_path"] = File.real_path("#{boot_dir}/vmlinuz#{suffix}") - self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") - self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") + if "#{os_mount}" == "initramfs" + self["linux_modules_initrd"] = File.real_path("#{boot_dir}/modules#{suffix}.cgz") + self["linux_headers_initrd"] = File.real_path("#{boot_dir}/headers#{suffix}.cgz") + end end private def set_kernel_uri -- 2.23.0

1 0

[PATCH lkp-tests] tests: add test script for libmicro
by Bai Jing 05 Nov '20

05 Nov '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/libmicro | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100755 tests/libmicro diff --git a/tests/libmicro b/tests/libmicro new file mode 100755 index 00000000..105bb9ad --- /dev/null +++ b/tests/libmicro @@ -0,0 +1,8 @@ +#!/bin/sh +# - runtime + +cd $BENCHMARK_ROOT/libMicro || exit + +[ -n "$runtime" ] || runtime=300 + +log_cmd ./bench || exit -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/job.rb: change the way get hosts file
by Li Ping 05 Nov '20

05 Nov '20

[why] we can get hosts file from the repo lab-z9 or get hosts file from lkp-tests/hosts before: get hosts file from lab-z9, but the hosts file not exist in lkp-tests/hosts the job will puts the error message: hosts_file not exist ... but there is no need to puts such error message, if we have got the hosts file from lab-z9 --- lib/job.rb | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 995b38bf..6b619b79 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -334,20 +334,14 @@ class Job end def get_hosts_file + lab_hosts_file = get_lab_hosts_file + return lab_hosts_file if lab_hosts_file + hosts_file_name = @job['tbox_group'].split('--')[0] hosts_file = "#{LKP_SRC}/hosts/#{hosts_file_name}" + return hosts_file if File.exist?(hosts_file) - lab_hosts_file = get_lab_hosts_file - if lab_hosts_file - hosts_file = lab_hosts_file - end - - if File.file?(hosts_file) - hosts_file - else - puts("hosts_file not exist: #{hosts_file}, maybe need check testbox field") - nil - end + raise ArgumentError, "hosts file not exist: #{hosts_file_name}, maybe need check testbox field" end def include_files -- 2.23.0

2 2

[PATCH compass-ci] sched: refactor sched class for the find job boot function
by Cao Xueliang 05 Nov '20

05 Nov '20

[how] According to scheduler.cr API to refactor sched class. Extract find_job_boot function from sched.cr to find_job_boot.cr Extract find_next_job_boot function from sched.cr to find_next_job_boot.cr Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 237 +--------------------------- src/scheduler/find_job_boot.cr | 218 +++++++++++++++++++++++++ src/scheduler/find_next_job_boot.cr | 11 ++ 3 files changed, 232 insertions(+), 234 deletions(-) create mode 100644 src/scheduler/find_job_boot.cr create mode 100644 src/scheduler/find_next_job_boot.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a4b12b4..6ecd95d 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -13,6 +13,9 @@ require "../scheduler/jobfile_operate" require "../scheduler/redis_client" require "../scheduler/elasticsearch_client" +require "../scheduler/find_job_boot" +require "../scheduler/find_next_job_boot" + class Sched property es property redis @@ -323,230 +326,6 @@ class Sched @es.set_job_content(job) end - private def ipxe_msg(msg) - "#!ipxe - echo ... - echo #{msg} - echo ... - reboot" - end - - private def grub_msg(msg) - "#!grub - echo ... - echo #{msg} - echo ... - reboot" - end - - private def get_boot_container(job : Job) - response = Hash(String, String).new - response["docker_image"] = "#{job.docker_image}" - response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + - JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") - response["job"] = "http://#{SCHED_HOST}:#{SCHED_PORT}/job_initrd_tmpfs/#{job.id}/job.cgz" - - return response.to_json - end - - private def get_boot_grub(job : Job) - initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" - - response = "#!grub\n\n" - response += "linux (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" - response += "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")} user=lkp" - response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job" - response += " rootovl ip=dhcp ro root=#{job.kernel_append_root}\n" - - response += "initrd (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" - response += JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/initrd.lkp") - response += " (http,#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT})" - response += JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/#{initrd_lkp_cgz}") - response += " (http,#{SCHED_HOST}:#{SCHED_PORT})/job_initrd_tmpfs/" - response += "#{job.id}/job.cgz\n" - - response += "boot\n" - - return response - end - - def touch_access_key_file(job : Job) - FileUtils.touch(job.access_key_file) - end - - def boot_content(job : Job | Nil, boot_type : String) - touch_access_key_file(job) if job - - case boot_type - when "ipxe" - return job ? get_boot_ipxe(job) : ipxe_msg("No job now") - when "grub" - return job ? get_boot_grub(job) : grub_msg("No job now") - when "container" - return job ? get_boot_container(job) : Hash(String, String).new.to_json - else - raise "Not defined boot type #{boot_type}" - end - end - - def rand_queues(queues) - return queues if queues.empty? - - queues_size = queues.size - base = Random.rand(queues_size) - temp_queues = [] of String - - (0..queues_size - 1).each do |index| - temp_queues << queues[(index + base) % queues_size] - end - - return temp_queues - end - - def get_queues(host) - queues = [] of String - - queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str - - queues_str.split(',', remove_empty: true) do |item| - queues << item.strip - end - - return rand_queues(queues) - end - - def get_job_from_queues(queues, testbox) - job = nil - - queues.each do |queue| - job = prepare_job("sched/#{queue}", testbox) - return job if job - end - - return job - end - - def get_job_boot(host, boot_type) - queues = get_queues(host) - job = get_job_from_queues(queues, host) - - if job - Jobfile::Operate.create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) - end - - return boot_content(job, boot_type) - end - - # auto submit a job to collect the host information - # grub hostname is link with ":", like "00:01:02:03:04:05" - # remind: if like with "-", last "-05" is treated as host number - # then hostname will be "sut-00-01-02-03-04" !!! - def submit_host_info_job(mac) - host = "sut-#{mac}" - set_host_mac(mac, host) - - Jobfile::Operate.auto_submit_job( - "#{ENV["LKP_SRC"]}/jobs/host-info.yaml", - "testbox: #{host}") - end - - def find_job_boot(env : HTTP::Server::Context) - value = env.params.url["value"] - boot_type = env.params.url["boot_type"] - - case boot_type - when "ipxe" - host = @redis.hash_get("sched/mac2host", normalize_mac(value)) - when "grub" - host = @redis.hash_get("sched/mac2host", normalize_mac(value)) - submit_host_info_job(value) unless host - when "container" - host = value - end - - get_job_boot(host, boot_type) - end - - def find_next_job_boot(env) - hostname = env.params.query["hostname"]? - mac = env.params.query["mac"]? - if !hostname && mac - hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) - end - - get_job_boot(hostname, "ipxe") - end - - def get_testbox_boot_content(testbox, boot_type) - job = find_job(testbox) if testbox - Jobfile::Operate.create_job_cpio(job.dump_to_json_any, - Kemal.config.public_folder) if job - - return boot_content(job, boot_type) - end - - private def find_job(testbox : String, count = 1) - tbox_group = JobHelper.match_tbox_group(testbox) - tbox = tbox_group.partition("--")[0] - - queue_list = query_consumable_keys(tbox) - - boxes = ["sched/" + testbox, - "sched/" + tbox_group, - "sched/" + tbox, - "sched/" + tbox_group + "/idle"] - boxes.each do |box| - next if queue_list.select(box).size == 0 - count.times do - job = prepare_job(box, testbox) - return job if job - - sleep(1) unless count == 1 - end - end - - # when find no job, auto submit idle job at background - spawn { auto_submit_idle_job(tbox_group) } - - return nil - end - - private def prepare_job(queue_name, testbox) - response = @task_queue.consume_task(queue_name) - job_id = JSON.parse(response[1].to_json)["id"] if response[0] == 200 - job = nil - - if job_id - begin - job = @es.get_job(job_id.to_s) - rescue ex - puts "Invalid job (id=#{job_id}) in es. Info: #{ex}" - puts ex.inspect_with_backtrace - end - end - - if job - job.update({"testbox" => testbox}) - job.set_result_root - puts %({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state": "set result root"}) - @redis.set_job(job) - end - return job - end - - private def get_idle_job(tbox_group, testbox) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - - # if there has no idle job, auto submit and get 1 - if job.nil? - auto_submit_idle_job(tbox_group) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - end - - return job - end - def auto_submit_idle_job(tbox_group) full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" extra_job_fields = [ @@ -561,16 +340,6 @@ class Sched extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end - private def get_boot_ipxe(job : Job) - response = "#!ipxe\n\n" - response += job.initrds_uri - response += job.kernel_uri - response += job.kernel_params - response += "\nboot\n" - - return response - end - def update_job_parameter(env : HTTP::Server::Context) job_id = env.params.query["job_id"]? if !job_id diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr new file mode 100644 index 0000000..5276e2a --- /dev/null +++ b/src/scheduler/find_job_boot.cr @@ -0,0 +1,218 @@ +class Sched + def find_job_boot(env : HTTP::Server::Context) + value = env.params.url["value"] + boot_type = env.params.url["boot_type"] + + case boot_type + when "ipxe" + host = @redis.hash_get("sched/mac2host", normalize_mac(value)) + when "grub" + host = @redis.hash_get("sched/mac2host", normalize_mac(value)) + submit_host_info_job(value) unless host + when "container" + host = value + end + + get_job_boot(host, boot_type) + end + + # auto submit a job to collect the host information + # grub hostname is link with ":", like "00:01:02:03:04:05" + # remind: if like with "-", last "-05" is treated as host number + # then hostname will be "sut-00-01-02-03-04" !!! + def submit_host_info_job(mac) + host = "sut-#{mac}" + set_host_mac(mac, host) + + Jobfile::Operate.auto_submit_job( + "#{ENV["LKP_SRC"]}/jobs/host-info.yaml", + "testbox: #{host}") + end + + def rand_queues(queues) + return queues if queues.empty? + + queues_size = queues.size + base = Random.rand(queues_size) + temp_queues = [] of String + + (0..queues_size - 1).each do |index| + temp_queues << queues[(index + base) % queues_size] + end + + return temp_queues + end + + def get_queues(host) + queues = [] of String + + queues_str = @redis.hash_get("sched/host2queues", host) + return queues unless queues_str + + queues_str.split(',', remove_empty: true) do |item| + queues << item.strip + end + + return rand_queues(queues) + end + + def get_job_from_queues(queues, testbox) + job = nil + + queues.each do |queue| + job = prepare_job("sched/#{queue}", testbox) + return job if job + end + + return job + end + + def get_job_boot(host, boot_type) + queues = get_queues(host) + job = get_job_from_queues(queues, host) + + if job + Jobfile::Operate.create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + end + + return boot_content(job, boot_type) + end + + private def ipxe_msg(msg) + "#!ipxe + echo ... + echo #{msg} + echo ... + reboot" + end + + private def grub_msg(msg) + "#!grub + echo ... + echo #{msg} + echo ... + reboot" + end + + private def get_boot_container(job : Job) + response = Hash(String, String).new + response["docker_image"] = "#{job.docker_image}" + response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") + response["job"] = "http://#{SCHED_HOST}:#{SCHED_PORT}/job_initrd_tmpfs/#{job.id}/job.cgz" + + return response.to_json + end + + private def get_boot_grub(job : Job) + initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" + + response = "#!grub\n\n" + response += "linux (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" + response += "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")} user=lkp" + response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job" + response += " rootovl ip=dhcp ro root=#{job.kernel_append_root}\n" + + response += "initrd (http,#{OS_HTTP_HOST}:#{OS_HTTP_PORT})" + response += JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/initrd.lkp") + response += " (http,#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT})" + response += JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/#{initrd_lkp_cgz}") + response += " (http,#{SCHED_HOST}:#{SCHED_PORT})/job_initrd_tmpfs/" + response += "#{job.id}/job.cgz\n" + + response += "boot\n" + + return response + end + + def touch_access_key_file(job : Job) + FileUtils.touch(job.access_key_file) + end + + private def get_boot_ipxe(job : Job) + response = "#!ipxe\n\n" + response += job.initrds_uri + response += job.kernel_uri + response += job.kernel_params + response += "\nboot\n" + + return response + end + + def boot_content(job : Job | Nil, boot_type : String) + touch_access_key_file(job) if job + + case boot_type + when "ipxe" + return job ? get_boot_ipxe(job) : ipxe_msg("No job now") + when "grub" + return job ? get_boot_grub(job) : grub_msg("No job now") + when "container" + return job ? get_boot_container(job) : Hash(String, String).new.to_json + else + raise "Not defined boot type #{boot_type}" + end + end + + private def find_job(testbox : String, count = 1) + tbox_group = JobHelper.match_tbox_group(testbox) + tbox = tbox_group.partition("--")[0] + + queue_list = query_consumable_keys(tbox) + + boxes = ["sched/" + testbox, + "sched/" + tbox_group, + "sched/" + tbox, + "sched/" + tbox_group + "/idle"] + boxes.each do |box| + next if queue_list.select(box).size == 0 + count.times do + job = prepare_job(box, testbox) + return job if job + + sleep(1) unless count == 1 + end + end + + # when find no job, auto submit idle job at background + spawn { auto_submit_idle_job(tbox_group) } + + return nil + end + + private def prepare_job(queue_name, testbox) + response = @task_queue.consume_task(queue_name) + job_id = JSON.parse(response[1].to_json)["id"] if response[0] == 200 + job = nil + + if job_id + begin + job = @es.get_job(job_id.to_s) + rescue ex + puts "Invalid job (id=#{job_id}) in es. Info: #{ex}" + puts ex.inspect_with_backtrace + end + end + + if job + job.update({"testbox" => testbox}) + job.set_result_root + puts %({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state": "set result root"}) + @redis.set_job(job) + end + return job + end + + private def get_idle_job(tbox_group, testbox) + job = prepare_job("sched/#{tbox_group}/idle", testbox) + + # if there has no idle job, auto submit and get 1 + if job.nil? + auto_submit_idle_job(tbox_group) + job = prepare_job("sched/#{tbox_group}/idle", testbox) + end + + return job + end + +end diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr new file mode 100644 index 0000000..674c763 --- /dev/null +++ b/src/scheduler/find_next_job_boot.cr @@ -0,0 +1,11 @@ +class Sched + def find_next_job_boot(env) + hostname = env.params.query["hostname"]? + mac = env.params.query["mac"]? + if !hostname && mac + hostname = @redis.hash_get("sched/mac2host", normalize_mac(mac)) + end + + get_job_boot(hostname, "ipxe") + end +end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] add template compare mode
by Lu Weitao 05 Nov '20

05 Nov '20

add template compare mode in sbin/compare(command-line tool) by: - load and parse compare_template.yaml - call lib/es_query.rb, query jobs data set from ES - call lib/matrix.rb, convert jobs data set to group matrices - call lib/compare_matrixes.rb, use groups matrices to compare background: To support compare with user-defined template feature, the work-flow is: load compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results example: compare -t/--template compare_template.yaml, the yaml like: compare_metrics: - fio.write_iops - fio.read_iops filter: suite: - fio-basic os_arch: - aarch64 - x86 compare_dimensions: - os: debian os_version: sid - os: openeuler os_version: 20.03 x_params: - bs - test_size title: Hackbench Performance Testing unit: KB/s output:{ "title": "Hackbench Performance Testing", "unit": "KB/s", "x_name": "bs|test_size", "tables": { "4k|1G": { "fio.write_iops": { "average": { "openeuler 20.03": 308298.5524761111, "debian sid": 220289.07563 }, "standard_deviation": { "openeuler 20.03": null, "debian sid": null }, "change": { "debian sid vs openeuler 20.03": -28.5 } } }, "16k|1G": {...}, ... } } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++ sbin/compare | 10 +++++++- 2 files changed, 76 insertions(+), 1 deletion(-) diff --git a/lib/compare.rb b/lib/compare.rb index 2837edc..645f682 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -6,6 +6,7 @@ require_relative './es_query.rb' require_relative './matrix2.rb' require_relative './compare_matrixes.rb' require_relative './constants.rb' +require 'yaml' # ------------------------------------------------------------------------------------------- # compare_matrices_list @@ -66,3 +67,69 @@ def create_groups_matrices_list(conditions, dims) query_results = es.multi_field_query(conditions) combine_group_query_data(query_results, dims) end + +# ------------------------------------------------------------------------------------------- +# compare with user-defined compare_template.yaml +# compare_temlpate.yaml sample: +# compare_metrics: +# - fio.write_iops +# - fio.read_iops +# filter: +# suite: +# - fio-basic +# os_arch: +# - aarch64 +# - x86 +# compare_dimensions: +# - os: debian +# os_version: sid +# - os: openeuler +# os_version: 20.03 +# x_params: +# - bs +# - test_size +# title: Hackbench Performance Testing +# unit: KB/s +# + +def compare_by_template(template) + template_params = load_template(template) + groups_matrices = create_groups_matrices(template_params) + compare_results = compare_metrics_values(groups_matrices) + show_compare_result(compare_results, template_params) +end + +def load_template(template) + unless File.file?(template) + warn 'template does not exist' + exit + end + YAML.load_file(template) +end + +# input: template_params: Hash +# eg: +# { +# "compare_metrics"=>["fio.write_iops", "fio.read_iops"], +# "filter"=>[ +# {"suite"=>["fio-bisic"]}, +# {"os_arch"=>["aarch_64"]} +# ], +# "compare_dimensions"=>[ +# {"os"=>"openeuler", "os_version"=>20.03}, +# {"os"=>"centos", "os_version"=>7.6} +# ], +# "x_params"=>["block_size", "package_size"], +# "title"=>"Hackbench Performance Testing", +# "unit"=>"KB/s" +# } +def create_groups_matrices(template_params) + es = ESQuery.new + query_results = es.multi_field_query(template_params['filter']) + combine_group_jobs_list( + query_results, + template_params['x_params'], + template_params['compare_dimensions'], + template_params['compare_metrics'] + ) +end diff --git a/sbin/compare b/sbin/compare index cf3980c..b13682e 100755 --- a/sbin/compare +++ b/sbin/compare @@ -6,6 +6,7 @@ # Usage: # compare "conditions_1" "conditions_2" ... -c "common_conditions" # compare "conditions" -d "dimensions" +# compare -t template.yaml # Eg: # compare "id=6000,6001" "id=7000,7001" # compare "commit=a12d232e" "commit=b3bacc31" @@ -21,6 +22,7 @@ is_group = false dimensions = nil colorful = nil options = {} +template = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: compare "conditions" ... [option]' @@ -45,6 +47,10 @@ opt_parser = OptionParser.new do |opts| colorful = color end + opts.on('-t', '--template template', 'compare with user-defined template') do |t| + template = t + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -60,7 +66,9 @@ opt_parser.parse!(argv) options = { theme: colorful } if colorful -if is_group +if template + compare_by_template(template) +elsif is_group compare_group(argv, dimensions, options) else compare_matrices_list(argv, common_conditions, options) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] compare values by each metrics
by Lu Weitao 05 Nov '20

05 Nov '20

compare values by each metrics based on groups matrices, and format compare result as echart data_set background: To support compare with user-defined template feature, the work-flow is: load compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch do: compare_values by each metrics ---> format/show results Signed-off-by: Lu Weitao <luweitaobe(a)163.com> Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare_formatter.rb | 16 +++++++ lib/compare_matrixes.rb | 100 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 116 insertions(+) create mode 100644 lib/compare_formatter.rb diff --git a/lib/compare_formatter.rb b/lib/compare_formatter.rb new file mode 100644 index 0000000..526ad8a --- /dev/null +++ b/lib/compare_formatter.rb @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ or GPL-2.0 +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +# ---------------------------------------------------------------------------------------------------- +# format compare results for a specific format +# + +def format_for_echart(metrics_compare_results, template_params) + echart_result = {} + echart_result['title'] = template_params['title'] || nil + echart_result['unit'] = template_params['unit'] || nil + echart_result['x_name'] = template_params['x_params'].join('|') || nil + echart_result['tables'] = metrics_compare_results + echart_result +end diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 078028a..6b101fd 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -6,6 +6,7 @@ LKP_SRC ||= ENV['LKP_SRC'] || File.dirname(__dir__) require 'set' require 'json/ext' require_relative 'themes' +require_relative './compare_formatter.rb' require "#{LKP_SRC}/lib/stats" FAILURE_PATTERNS = IO.read("#{LKP_SRC}/etc/failure").split("\n") @@ -399,6 +400,105 @@ def compare_group_matrices(group_matrices, options) result_str end +# input: groups_matrices +# { +# group_key_1 => { +# dimension_1 => matrix_1, (openeuler 20.03) +# dimension_2 => matrix_2, (openeuler 20.09) +# dimension_3 => matrix_3, (centos 7.6) +# }, +# group_key_2 => {...} +# } +# +# output: compare_metrics_values +# { +# group_key_1 => { +# metric_1 => { +# 'average' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'standard_deviation' => { +# 'dimension_1' => xxx +# 'dimension_2' => xxx +# 'dimension_3' => xxx +# }, +# 'change' => { +# 'dimension_2 vs dimension_1' => xxx +# 'dimension_3 vs dimension_1' => xxx +# 'dimension_3 vs dimension_2' => xxx +# } +# }, +# metric_2 => {...} +# } +# } +def compare_metrics_values(groups_matrices) + metrics_compare_values = {} + groups_matrices.each do |group_key, dimensions| + metrics_compare_values[group_key] = get_metric_values(dimensions) + end + metrics_compare_values +end + +def get_metric_values(dimensions) + metrics_values = {} + dimensions.each do |dim, matrix| + matrix.each do |metric, values| + assign_metric_values(metrics_values, dim, metric, values) + end + end + assign_metric_change(metrics_values) + metrics_values +end + +def assign_metric_values(metrics_values, dim, metric, values) + metrics_values[metric] ||= {} + metrics_values[metric]['average'] ||= {} + metrics_values[metric]['standard_deviation'] ||= {} + metric_value = get_values(values, true) + metrics_values[metric]['average'][dim] = metric_value[:average] + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] +end + +def assign_metric_change(metrics_values) + metrics_values.each do |metric, values| + metrics_values[metric]['change'] = {} + next if values['average'].size < 2 + + dimension_list = values['average'].keys + change_titles = get_change_titles(dimension_list) + change_titles.each do |base_dimension, challenge_dimension| + change = get_compare_value(values['average'][base_dimension], values['average'][challenge_dimension], true) + values['change'] = { "#{challenge_dimension} vs #{base_dimension}" => change } + end + end +end + +# input: dimension_list +# eg: ['openeuler 20.03', 'debian sid', 'centos 7.6'] +# output: Array(base_dimension: String, challenge_dimension: String) +# [ +# ['openeuler 20.03', 'debian sid'], +# ['openeuler 20.03', 'centos 7.6'], +# ['debian sid', 'centos 7.6'] +# ] +def get_change_titles(dimension_list) + dims = [] + dimension_list_size = dimension_list.size + (1..dimension_list_size - 1).each do |i| + (i..dimension_list_size - 1).each do |j| + dims << [dimension_list[i - 1], dimension_list[j]] + end + end + dims +end + +def show_compare_result(metrics_compare_results, template_params) + echart_results = format_for_echart(metrics_compare_results, template_params) + print JSON.pretty_generate(echart_results) +end + # Format Fields def format_fails_runs(fails, runs) -- 2.23.0

1 0

[PATCH v5 compass-ci 2/2] create matrices-groups according to group jobs
by Lu Weitao 05 Nov '20

05 Nov '20

create matrices-groups according to auto group job_list background: To support compare with user-defined template feature, the work-flow is: load compare_template.yaml --> query_results(ES) ---> auto group jobs_list ---> create groups_matrices ---> compare_values by each metrics ---> format/show results current patch do: create groups_matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 05f96dc..b38b2fb 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -120,3 +120,27 @@ def combine_group_query_data(query_data, dims) end groups end + +# input: +# 1. query results(job_list) from es_query that will be auto group by auto_group_by_template() +# 2. params from user's template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# return: group_matrix of Hash(String, Hash(String, matrix)) +def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) + job_list = query_data['hits']['hits'] + groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) + groups.each do |group_key, dims| + dims.each do |dim_key, jobs| + groups[group_key][dim_key] = create_matrix(jobs) + end + end + groups +end -- 2.23.0

1 0

[PATCH lkp-tests] jobs: add multi-device test parameters
by Zhang Yu 05 Nov '20

05 Nov '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- jobs/sysbench-mysql.yaml | 31 ++++++++++++++++++++++++------- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml index c060fc2a..097acad4 100644 --- a/jobs/sysbench-mysql.yaml +++ b/jobs/sysbench-mysql.yaml @@ -1,10 +1,27 @@ suite: sysbench-mysql category: benchmark -sysbench-mysql: - oltp_test_mode: complex - oltp_tables_count: 3 - oltp_table_size: 1000 - nr_threads: 64 - runtime: 120 - report_interval: 10 +cluster: cs-lkp-hsw-ep5 + +if role server: + mysql: + +if role client: + sysbench-mysql: + oltp_test_mode: complex + oltp_tables_count: 1000 + oltp_table_size: 100000 + nr_threads: 256 + runtime: 600 + report_interval: 1 + mysql_host: + mysql_port: + mysql_password: + mysql_user: + mysql_db: sysbench_1 + max_requests: 0 + mysql_table_engine: innodb + rand_type: special + rand_spec_pct: 100 + oltp_read_only: off + events: 2000000000 -- 2.23.0

3 4

[PATCH lkp-tests] job.rb: fix add non-top-level program's parameters to pp
by Wei Jihui 05 Nov '20

05 Nov '20

[Why] some programs in job yaml are not at top level, like if role server: iperf: runtime: 300 in this case, we can not add "runtime: 300" to: pp: iperf: [How] when traverse job to find program's parameters, in this case, it is iperf, then find it at "if role server:"'s keys, not job's top-level keys. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 995b38bf..9fc7b17f 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -576,10 +576,10 @@ class Job @job[comment_to_symbol('auto generated by add_pp()')] = nil @job["pp"] = Hash.new() init_program_options() - for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, p_args| - k = hash_key_re_string(@job, p_n, @job['pp'].keys) - if k && !(a)job[k].is_a?(Hash) - @job['pp'][k] = @job[k] + for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, _p_args| + k = hash_key_re_string(h, p_n, @job['pp'].keys) + if k && h[k] && !h[k].is_a?(Hash) + @job['pp'][k] = h[k] next end options_hash = @referenced_programs[p_n] -- 2.23.0

2 1

[PATCH lkp-tests] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 05 Nov '20

05 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 9551ebfe..bb20a845 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -37,6 +37,21 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +get_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +[ -n "$PKGBUILD_TAG" ] && { + PKGBUILD_TAG="-$PKGBUILD_TAG" + PKGBUILD_TAG_FILE="PKGBUILD$PKGBUILD_TAG" + get_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 -- 2.23.0

2 2