- Compass-ci - mailweb.openeuler.org

[PATCH v5 compass-ci 05/12] mail-robot: check commit url for gitee
by Luan Shengde 12 Nov '20

12 Nov '20

GiteeCommitUrlCheck check commit url availability for hub gitee - clone the repo - check commit log for the commit usage: reference parse-apply-account-email.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/gitee-commit-url-check.rb | 44 +++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100755 lib/gitee-commit-url-check.rb diff --git a/lib/gitee-commit-url-check.rb b/lib/gitee-commit-url-check.rb new file mode 100755 index 0000000..a9439ad --- /dev/null +++ b/lib/gitee-commit-url-check.rb @@ -0,0 +1,44 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check commit url availability for hub gitee.com +# gitee_commit_index +# clone the repo +# check commit available +class GiteeCommitUrlCheck + def initialize(my_info, url, base_url) + @my_info = my_info + @url = url + @base_url = base_url + end + + def gitee_commit_index + repo_dir = @url.split('/')[-3] + repo_url = [@base_url, 'git'].join('.') + commit_id = @url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] + + FileUtils.rm_rf repo_dir + + gitee_commit_exist(email_index) + end + + def gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = "We can not conform whether the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH v5 compass-ci 04/12] mail-robot: parse apply account email
by Luan Shengde 12 Nov '20

12 Nov '20

ParseApplyAccountEmail parse commit url and pubkey parse commit url check commit url exists check base url in upstream-repos check commit available hub: gitee.com call gitee-commit-url-check to check commit available hub: non-gitee.com check the feedback of curl command to check the commit url available parse pubkey attachment file name: id_rsa.pub usage: reference assign-account.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/parse-apply-account-email.rb | 115 +++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100755 lib/parse-apply-account-email.rb diff --git a/lib/parse-apply-account-email.rb b/lib/parse-apply-account-email.rb new file mode 100755 index 0000000..37063e0 --- /dev/null +++ b/lib/parse-apply-account-email.rb @@ -0,0 +1,115 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# parse mail_content +# parse_commit_url +# extract_commit_url +# check commit url exists +# base_url_in_upstream_repos +# check base url in upstream-repos +# commit_url_availability +# check commit available +# parse pub_key +# check pub_key exists +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + + # the commit url should be headed with a prefix: my oss commit + # the commit url must be in a standart format, example: + # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = "No matched commit url found.\n" + error_message += "Ensure that you have add a right commit url with prefix 'my oss commit:'." + + raise error_message + end + + def parse_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + base_url_in_upstream_repos('/c/upstream-repos', base_url) + commit_url_availability(url, base_url) + + return url + end + + def base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = "The repo url for your commit is not in our upstream-repo list.\n" + error_message += 'Use a new one, or consulting the manager for available repo list.' + + raise error_message + end + + def commit_url_availability(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + gitee_commit(url, base_url) + else + non_gitee_commit(url) + end + end + + def gitee_commit(url, base_url) + my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) + my_gitee_commit.gitee_commit_index + end + + def non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_info['my_email'] + + return unless email_index.nil? + + error_message = "We can not confirm the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end + + def parse_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + pub_key_exist(pub_key) + + return pub_key + end + + def pub_key_exist(pub_key) + return unless pub_key.nil? + + error_message = "No pub_key found.\n" + error_message += 'Please add the pub_key as an attachment with filename: id_rsa.pub.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH compass-ci 4/4] assign-account: add /etc/hostname mapping
by Luan Shengde 12 Nov '20

12 Nov '20

add /etc/hostname mapping for assign-account [why] it's need to get the jumper hostname when generate the rsa private/public key when execute ssh-keygen with -C option in container ssh-keygen -f /home/$login_name/.ssh/id_rsa -N '' -C $login_name@$hostname [how] add file mapping for file /etc/hostname Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/start | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/assign-account/start b/container/assign-account/start index 2290982..379dbd3 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -14,6 +14,8 @@ cmd=( -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v /srv/account-info:/opt/account_data/:rw + -v /etc/hostname:/etc/hostname:ro + -v /etc/localtime:/etc/localtime:ro -v /home:/home:rw -p 29999:29999 assign-account -- 2.23.0

2 2

[PATCH v2 lkp-tests] tests: adapt the special version of sysbench test
by Zhang Yu 12 Nov '20

12 Nov '20

[why] Use the special version of sysbench test mysql, use different test file are required. [how] Add parameters to test, and use different test file of sysbench Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/sysbench-mysql | 88 ++++++++++++++++++++++++++++++++------------ 1 file changed, 65 insertions(+), 23 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 4ab5b453..db9c57e1 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,52 +1,94 @@ #!/bin/sh -# mysql_user -# mysql_host -# mysql_port -# mysql_db -# db_driver # - oltp_test_mode # - oltp_tables_count # - oltp_table_size +# - max_requests +# - mysql_table_engine # - nr_threads +# - rand_type +# - rand_spec_pct # - runtime # - report_interval +# - oltp_read_only +# - events +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +# db_driver : "${mysql_user:=root}" -: "${mysql_host:=localhost}" -: "${mysql_port:=3306}" -: "${mysql_db:=test_1000}" +: "${mysql_host:=$direct_server_ips}" +: "${mysql_port:=$mysql_port}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=$mysql_password}" : "${db_driver:=mysql}" -: "${oltp_tables_count:=3}" -: "${oltp_table_size:=1000}" -: "${nr_threads:=64}" -: "${runtime:=120}" -: "${report_interval:=10}" +: "${oltp_test_mode:=complex}" +: "${oltp_tables_count:=1000}" +: "${oltp_table_size:=100000}" +: "${max_requests:=0}" +: "${mysql_table_engine:=innodb}" +: "${rand_type:=special}" +: "${rand_spec_pct:=100}" +: "${nr_threads:=256}" +: "${runtime:=600}" +: "${report_interval:=1}" +: "${oltp_read_only:=off}" +: "${events:=2000000000}" -args=( +args1=( --mysql-user=$mysql_user --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db + --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver - --tables=$oltp_tables_count - --table-size=$oltp_table_size + --mysql-password=$mysql_password + --max-requests=$max_requests + --mysql-table-engine=$mysql_table_engine + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --rand-type=$rand_type + --rand-spec-pct=$rand_spec_pct --threads=$nr_threads --time=$runtime - --report-interval=$report_interval ) +args2=( + --mysql-user=$mysql_user + --mysql-password=$mysql_password + --mysql-host=$mysql_host + --mysql-port=$mysql_port + --mysql-db=$mysql_db + --threads=$nr_threads + --oltp-read-only=$oltp_read_only + --oltp-table-size=$oltp_table_size + --oltp-tables-count=$oltp_tables_count + --report-interval=$report_interval + --time=$runtime + --events=$events +) +stop_firewalld() +{ + systemctl stop firewalld + iptables -F +} + run_sysbench_step() { - sysbench /usr/share/sysbench/$1 "${args[@]}" $2 + lua_script=$1 + shift + sysbench /usr/local/share/sysbench/tests/include/oltp_legacy/$lua_script "$@" } run_sysbench_mysql() { - systemctl start mysqld - mysql -e "create database test_1000;" - run_sysbench_step oltp_common.lua prepare - run_sysbench_step oltp_read_write.lua run - run_sysbench_step oltp_common.lua cleanup + run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + run_sysbench_step oltp.lua ${args2[@]} run + run_sysbench_step oltp.lua ${args2[@]} cleanup } +stop_firewalld run_sysbench_mysql + -- 2.23.0

2 2

[PATCH v2 compass-ci] support queues parameter
by Xiao Shenwei 12 Nov '20

12 Nov '20

register host2queues and requested specified queues Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 47 ++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 4e6bc2f..8e489c6 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -11,15 +11,26 @@ require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' +names = Set.new %w[ + SCHED_HOST + SCHED_PORT +] +defaults = relevant_defaults(names) +SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = defaults['SCHED_PORT'] || 3000 + def get_url(hostname) - names = Set.new %w[ - SCHED_HOST - SCHED_PORT - ] - defaults = relevant_defaults(names) - host = defaults['SCHED_HOST'] || '172.17.0.1' - port = defaults['SCHED_PORT'] || 3000 - "http://#{host}:#{port}/boot.container/hostname/#{hostname}" + "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" +end + +def set_host2queues(hostname, queues) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/set_host2queues?host=#{hostname}&queues=#{queues}'" + system cmd +end + +def del_host2queues(hostname) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/del_host2queues?host=#{hostname}'" + system cmd end def parse_response(url) @@ -67,7 +78,7 @@ def load_initrds(load_path, hash) wget_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") end -def run(hostname, load_path, hash) +def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( @@ -77,24 +88,26 @@ def run(hostname, load_path, hash) clean_dir(load_path) end -def main(hostname) +def main(hostname, queues) + set_host2queues(hostname, queues) url = get_url hostname puts url hash = parse_response url - return if hash.nil? + return del_host2queues(hostname) if hash.nil? load_path = build_load_path(hostname) load_initrds(load_path, hash) - run(hostname, load_path, hash) + start_container(hostname, load_path, hash) + del_host2queues(hostname) end -def loop_main(hostname) +def loop_main(hostname, queues) loop do begin - main(hostname) + main(hostname, queues) rescue StandardError => e puts e.backtrace - # if an exception happend, request the next time after 30 seconds + # if an exception occurs, request the next time after 30 seconds sleep 25 ensure sleep 5 @@ -109,11 +122,11 @@ def save_pid(pids) f.close end -def multi_docker(hostname, nr_container) +def multi_docker(hostname, nr_container, queues) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop_main("#{hostname}-#{i}") + loop_main("#{hostname}-#{i}", queues) end pids << pid end -- 2.23.0

2 2

[PATCH compass-ci 2/4] assign-account: add -r|--rsa-key option
by Luan Shengde 12 Nov '20

12 Nov '20

add -r|--rsa-key option for answerback-email [why] enable generate rsa private/public key on jumper server enable one click login whit the public key [how] add -r|--rsa-key option to enable generate rsa private/public key Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 686a327..6d8e3de 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -32,7 +32,8 @@ my_info = { 'my_email' => nil, 'my_name' => nil, 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => nil + 'my_ssh_pubkey' => nil, + 'my_jumper_pubkey' => false } def init_info(email_file, my_info) @@ -68,6 +69,10 @@ options = OptionParser.new do |opts| init_info(email_file, my_info) end + opts.on('-r', '--rsa-key', 'generate jumper ras public/private key and return pubkey') do + my_info['my_jumper_pubkey'] = true + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -114,10 +119,12 @@ def send_account(my_info) raise message if my_info['my_email'].nil? account_info = apply_account(my_info) + # for manually assign account, there will be no my_commit_url # but the key my_commit_url is required for es my_info['my_commit_url'] = '' my_info['my_login_name'] = account_info['my_login_name'] + my_info['my_jumper_pubkey'] = account_info['my_jumper_pubkey'] my_info.delete 'my_ssh_pubkey' store_account_info(my_info) message = build_message(my_info['my_email'], account_info) -- 2.23.0

2 2

[PATCH compass-ci 1/4] assign-account: Dockerfile add install ssh
by Luan Shengde 12 Nov '20

12 Nov '20

add install ssh for assign-account [why] it's need to generate rsa private/public key for user need install ssh for ssh-keygen command [how] add install ssh Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/Dockerfile | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/container/assign-account/Dockerfile b/container/assign-account/Dockerfile index f84b5e6..c1d360e 100644 --- a/container/assign-account/Dockerfile +++ b/container/assign-account/Dockerfile @@ -2,15 +2,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER luanshd0525(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ruby-sinatra ruby-json curl && \ - ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ - echo 'Asia/Shanghai' > /etc/timezone + apt-get install -y ssh ruby-sinatra ruby-json curl + COPY assign-account.rb /usr/local/bin/ COPY get_account_info.rb /usr/local/bin/ -- 2.23.0

2 2

[PATCH compass-ci] support queues parameter
by Xiao Shenwei 12 Nov '20

12 Nov '20

register host2queues and requested specified queues Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 47 ++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 4e6bc2f..8e489c6 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -11,15 +11,26 @@ require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' +names = Set.new %w[ + SCHED_HOST + SCHED_PORT +] +defaults = relevant_defaults(names) +SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = defaults['SCHED_PORT'] || 3000 + def get_url(hostname) - names = Set.new %w[ - SCHED_HOST - SCHED_PORT - ] - defaults = relevant_defaults(names) - host = defaults['SCHED_HOST'] || '172.17.0.1' - port = defaults['SCHED_PORT'] || 3000 - "http://#{host}:#{port}/boot.container/hostname/#{hostname}" + "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" +end + +def set_host2queues(hostname, queues) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/set_host2queues?host=#{hostname}&queues=#{queues}'" + system cmd +end + +def del_host2queues(hostname) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/del_host2queues?host=#{hostname}'" + system cmd end def parse_response(url) @@ -67,7 +78,7 @@ def load_initrds(load_path, hash) wget_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") end -def run(hostname, load_path, hash) +def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( @@ -77,24 +88,26 @@ def run(hostname, load_path, hash) clean_dir(load_path) end -def main(hostname) +def main(hostname, queues) + set_host2queues(hostname, queues) url = get_url hostname puts url hash = parse_response url - return if hash.nil? + return del_host2queues(hostname) if hash.nil? load_path = build_load_path(hostname) load_initrds(load_path, hash) - run(hostname, load_path, hash) + start_container(hostname, load_path, hash) + del_host2queues(hostname) end -def loop_main(hostname) +def loop_main(hostname, queues) loop do begin - main(hostname) + main(hostname, queues) rescue StandardError => e puts e.backtrace - # if an exception happend, request the next time after 30 seconds + # if an exception occurs, request the next time after 30 seconds sleep 25 ensure sleep 5 @@ -109,11 +122,11 @@ def save_pid(pids) f.close end -def multi_docker(hostname, nr_container) +def multi_docker(hostname, nr_container, queues) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop_main("#{hostname}-#{i}") + loop_main("#{hostname}-#{i}", queues) end pids << pid end -- 2.23.0

2 2

[PATCH v2 lab-z9] hosts: fix testbox included yaml error
by Wei Jihui 11 Nov '20

11 Nov '20

[Why] submit job with some physical machine's textbox will error: Included yaml file not found: 'hosts/taishan200-2280-2s48p-256g' (RuntimeError) [How] lkp-tests just included files that find $LKP_SRC and testbox's directory, so delete hosts is ok. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a103 | 2 +- hosts/taishan200-2280-2s48p-256g--a104 | 2 +- hosts/taishan200-2280-2s48p-256g--a106 | 2 +- hosts/taishan200-2280-2s48p-256g--a107 | 2 +- hosts/taishan200-2280-2s48p-512g--a105 | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/hosts/taishan200-2280-2s48p-256g--a103 b/hosts/taishan200-2280-2s48p-256g--a103 index 304c6fc..9fcc718 100644 --- a/hosts/taishan200-2280-2s48p-256g--a103 +++ b/hosts/taishan200-2280-2s48p-256g--a103 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bdc69 diff --git a/hosts/taishan200-2280-2s48p-256g--a104 b/hosts/taishan200-2280-2s48p-256g--a104 index a8a2789..41cdbb0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a104 +++ b/hosts/taishan200-2280-2s48p-256g--a104 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8892431 diff --git a/hosts/taishan200-2280-2s48p-256g--a106 b/hosts/taishan200-2280-2s48p-256g--a106 index ad587b3..95b2f05 100644 --- a/hosts/taishan200-2280-2s48p-256g--a106 +++ b/hosts/taishan200-2280-2s48p-256g--a106 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b883bf99 diff --git a/hosts/taishan200-2280-2s48p-256g--a107 b/hosts/taishan200-2280-2s48p-256g--a107 index fba1937..31b8dc6 100644 --- a/hosts/taishan200-2280-2s48p-256g--a107 +++ b/hosts/taishan200-2280-2s48p-256g--a107 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8917c39 diff --git a/hosts/taishan200-2280-2s48p-512g--a105 b/hosts/taishan200-2280-2s48p-512g--a105 index 096b5b0..66a0aca 100644 --- a/hosts/taishan200-2280-2s48p-512g--a105 +++ b/hosts/taishan200-2280-2s48p-512g--a105 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-512g +<<: taishan200-2280-2s48p-512g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b891945d -- 2.23.0

2 1

[PATCH v2 compass-ci] /lib: keep suite result in top for compare output
by Lu Kaiyi 11 Nov '20

11 Nov '20

[why] Currently, the output result of compare default order by alphabet. If there have too many lines in result, it will be difficult for user to find the lines that starts with $suite in metric column. [how] Divide the lines into two parts before print according to whether the line start with $suite or not, then combine output result into a whole to highlight the important info. [example] command: compare "os=openeuler" -d "tbox_group" in z9 one of group compare output result. suites_list:["iso2rootfs", "build-pkg", "deploy-cci"] Before: taishan200-2280-2s64p-256g--a101 vm-hi1620-2p8g--roota15 vm-hi1620-2p8g--roota14 taishan200-2280-2s64p-256g--a6 metric -------------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------ %stddev change %stddev change %stddev change %stddev \ | \ | \ | \ 493.41 ± 108% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.boot 47114.17 ± 109% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.idle 11.57 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.kernel_boot 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time.max 0.00 0 54815.06 ± 124% 0 52113.21 ± 173% 0 0.00 build-pkg.time.file_system_inputs 0.00 0 3730.76 ± 362% 0 9382.32 ± 552% 0 0.00 build-pkg.time.involuntary_context_switches 0.00 0 7.12 ± 365% 0 6.78 ± 379% 0 0.00 build-pkg.time.major_page_faults 0.00 0 35358.13 ± 194% 0 42519.61 ± 208% 0 0.00 build-pkg.time.maximum_resident_set_size 0.00 0 56001.11 ± 360% 0 160421.51 ± 553% 0 0.00 build-pkg.time.minor_page_faults 0.00 0 65536.00 0 65536.00 0 0.00 build-pkg.time.page_size 0.00 0 7.58 ± 755% 0 40.12 ± 903% 0 0.00 build-pkg.time.user_time 0.00 0 16269.71 ± 318% 0 29779.86 ± 394% 0 0.00 build-pkg.time.voluntary_context_switches 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time.max ... 5579.67 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU95.IPI1:Function_call_interrupts 3152478.83 ± 215% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.IPI0:Rescheduling_interrupts 570272.33 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.IPI1:Function_call_interrupts 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time.max 73066.67 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.file_system_inputs 124.50 ± 217% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.involuntary_context_switches 4.33 ± 172% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.major_page_faults 34816.00 ± 140% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.maximum_resident_set_size 23354.67 ± 202% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.minor_page_faults 65536.00 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.page_size 9.50 ± 68% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.percent_of_cpu_this_job_got 4.27 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.system_time 5.20 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.user_time 172904.17 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.voluntary_context_switches 2651512.00 ± 157% -61.9% 1011304.90 ± 25% -60.9% 1037595.96 ± 25% +582.1% 18086008.00 meminfo.Active 2253381.67 ± 176% -69.2% 694564.75 ± 35% -68.1% 718592.59 ± 35% +668.1% 17308404.00 meminfo.Active(anon) 466201.00 ± 223% -100.0% 0.00 -100.0% 0.00 +2509.3% 12164514.00 meminfo.AnonHugePages 1328043.33 ± 148% -50.7% 655286.32 ± 36% -49.3% 673334.38 ± 34% +966.0% 14157259.00 meminfo.AnonPages 0.00 0 524032.00 0 524032.00 0 520576.00 meminfo.CmaFree ... After: taishan200-2280-2s64p-256g--a101 vm-hi1620-2p8g--roota15 vm-hi1620-2p8g--roota14 taishan200-2280-2s64p-256g--a6 metric -------------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------ %stddev change %stddev change %stddev change %stddev \ | \ | \ | \ 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time.max 0.00 0 54815.06 ± 124% 0 52113.21 ± 173% 0 0.00 build-pkg.time.file_system_inputs 0.00 0 3730.76 ± 362% 0 9382.32 ± 552% 0 0.00 build-pkg.time.involuntary_context_switches 0.00 0 7.12 ± 365% 0 6.78 ± 379% 0 0.00 build-pkg.time.major_page_faults 0.00 0 35358.13 ± 194% 0 42519.61 ± 208% 0 0.00 build-pkg.time.maximum_resident_set_size 0.00 0 56001.11 ± 360% 0 160421.51 ± 553% 0 0.00 build-pkg.time.minor_page_faults 0.00 0 65536.00 0 65536.00 0 0.00 build-pkg.time.page_size 0.00 0 7.58 ± 755% 0 40.12 ± 903% 0 0.00 build-pkg.time.user_time 0.00 0 16269.71 ± 318% 0 29779.86 ± 394% 0 0.00 build-pkg.time.voluntary_context_switches 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time.max 0.00 0 0.00 0 0.00 0 1.00 deploy-cci.time.exit_status 0.00 0 0.00 0 0.00 0 503552.00 deploy-cci.time.file_system_inputs 0.00 0 0.00 0 0.00 0 3925.00 deploy-cci.time.involuntary_context_switches 0.00 0 0.00 0 0.00 0 28.00 deploy-cci.time.major_page_faults 0.00 0 0.00 0 0.00 0 271168.00 deploy-cci.time.maximum_resident_set_size 0.00 0 0.00 0 0.00 0 2280195.00 deploy-cci.time.minor_page_faults 0.00 0 0.00 0 0.00 0 65536.00 deploy-cci.time.page_size 0.00 0 0.00 0 0.00 0 3.00 deploy-cci.time.percent_of_cpu_this_job_got 0.00 0 0.00 0 0.00 0 17.61 deploy-cci.time.system_time 0.00 0 0.00 0 0.00 0 176.09 deploy-cci.time.user_time 0.00 0 0.00 0 0.00 0 384342.00 deploy-cci.time.voluntary_context_switches 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time.max 73066.67 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.file_system_inputs 124.50 ± 217% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.involuntary_context_switches 4.33 ± 172% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.major_page_faults 34816.00 ± 140% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.maximum_resident_set_size 23354.67 ± 202% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.minor_page_faults 65536.00 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.page_size 9.50 ± 68% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.percent_of_cpu_this_job_got 4.27 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.system_time 5.20 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.user_time 172904.17 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.voluntary_context_switches 493.41 ± 108% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.boot 47114.17 ± 109% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.idle 11.57 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.kernel_boot 1073.50 ± 217% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 1003392.50 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 612284.17 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.4:GICv3.26.Level.arch_timer 8151.83 ± 213% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU0.4:GICv3.26.Level.arch_timer 165405.33 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU0.IPI0:Rescheduling_interrupts 5886.83 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU0.IPI1:Function_call_interrupts 12477.50 ± 214% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU1.4:GICv3.26.Level.arch_timer ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 15 +++++++++------ lib/compare_matrixes.rb | 33 ++++++++++++++++++++++----------- lib/matrix2.rb | 15 +++++++++++---- 3 files changed, 42 insertions(+), 21 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..3db5169 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suites_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suites_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,15 @@ end def create_matrices_list(conditions) matrices_list = [] + suites_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suites_list.concat(suites) end - matrices_list + return matrices_list, suites_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +61,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..c1cee10 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suites_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suites_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -807,26 +808,36 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suites_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suites_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suites_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suites_list, matrixes_titles, success, theme) return '' if values.empty? + suites_list.uniq! result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suites_list.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end + + result_str += suite_str + common_str result_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..479d280 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,20 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end groups.delete(group_key) if value.size < 2 + suites_list << suite_list unless value.size < 2 end - groups + + return groups, suites_list end # input: @@ -139,7 +146,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], = create_matrix(jobs) end end -- 2.23.0

3 2

[PATCH v5 compass-ci 12/12] mail-robot: start file
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..eab1f12 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +defaults['CCI_SRC'] = '/c/compass-ci' +defaults['MAILDIR'] = '/srv/cci/Maildir/.compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'run-mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v5 compass-ci 11/12] mail-robot: build file
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v5 compass-ci 10/12] mail-robot: Dockerfile
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v5 compass-ci 09/12] mail-robot: documentation for service
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..2eb0dd1 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. assign-account + init my_info + - my_email + - my_name + send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

1 0

[PATCH v5 compass-ci 08/12] mail-robot: build fail message
by Luan Shengde 11 Nov '20

11 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-fail-eamil.rb | 38 ++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 lib/assign-account-fail-eamil.rb diff --git a/lib/assign-account-fail-eamil.rb b/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..1c2133c --- /dev/null +++ b/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following error: + + #{message} + + In order to successfully apply an account, please pay attention to the following points: + + 1. mail subject + the subject should exactly: apply account + + 2. commit url + when you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + ensure you commit url exist and available to access. + + 3. ssh pubkey + you need to add a pubkey as an attachment with name 'id_rsa.pub' to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v5 compass-ci 06/12] mail-robot: apply account for user
by Luan Shengde 11 Nov '20

11 Nov '20

ApplyJumperAccount apply jumper account for user required data: my_info: - my_email - my_name - my_uuid my_ssh_pubkey usage: reference assign-account.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/apply-jumper-account.rb | 40 +++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 lib/apply-jumper-account.rb diff --git a/lib/apply-jumper-account.rb b/lib/apply-jumper-account.rb new file mode 100755 index 0000000..9d07abe --- /dev/null +++ b/lib/apply-jumper-account.rb @@ -0,0 +1,40 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +# apply jumper account for user +# apply_jumper_account +# apply jumper account with my_info +# check account exists +class ApplyJumperAccount + def initialize(my_info, my_ssh_pubkey) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + @my_ssh_pubkey = my_ssh_pubkey + end + + def apply_jumper_account + @my_info['my_ssh_pubkey'] = @my_ssh_pubkey + + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = 'No more available jumper account.' + error_message += 'You may try again later or consulting the manager for a solution.' + + raise error_message + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 03/12] mail-robot: check to assign account
by Luan Shengde 11 Nov '20

11 Nov '20

AssignAccount assign account for user required data: mail_content - send_account parse email_content parse commit url parse pubkey apply_my_account generate uuid apply account store account info to es send email send success email send fail email usage: reference mail-robot.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account.rb | 78 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100755 lib/assign-account.rb diff --git a/lib/assign-account.rb b/lib/assign-account.rb new file mode 100755 index 0000000..6efeee0 --- /dev/null +++ b/lib/assign-account.rb @@ -0,0 +1,78 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative 'es_client' + +# assign uuid/account for user +# send_account +# parse my_commit_url and my_ssh_pubkey +# apply account +# store account info +# send mail +class AssignAccount + def initialize(mail_content) + @send_mail_host = %x(/sbin/ip route | awk '/default/ {print $3}').chomp + @send_mail_port = SEND_MAIL_PORT + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def send_account + my_commit_url, my_ssh_pubkey = parse_commit_url_pub_key + acct_info = apply_my_account(my_ssh_pubkey) + + @my_info['my_commit_url'] = my_commit_url + @my_info['my_login_name'] = acct_info['my_login_name'] + + store_account_info + send_mail('') + rescue StandardError => e + puts e.message + puts e.backtrace + + send_mail(e.message) + end + + def parse_commit_url_pub_key + parse_apply_account_email = ParseApplyAccountEmail.new(@mail_content) + + my_commit_url = parse_apply_account_email.parse_commit_url + my_ssh_pubkey = parse_apply_account_email.parse_pub_key + + return my_commit_url, my_ssh_pubkey + end + + def apply_my_account(my_ssh_pubkey) + my_uuid = %x(uuidgen).chomp + + @my_info['my_uuid'] = my_uuid + + apply_account = ApplyJumperAccount.new(@my_info, my_ssh_pubkey) + acct_info = apply_account.apply_jumper_account + + return acct_info + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def send_mail(error_message) + if error_message.empty? + email_message = build_apply_account_email(@my_info) + else + email_message = build_apply_account_fail_email(@my_info, error_message) + end + + %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 02/12] mail-robot: listen and handle new email
by Luan Shengde 11 Nov '20

11 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..0ae80f3 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.send_account +end -- 2.23.0

1 0

[PATCH v4 compass-ci 08/12] mail-robot: build fail message
by Luan Shengde 11 Nov '20

11 Nov '20

build fail email message with raised error message - lack commit url - lack pubkey - repo url for the commit url not in upstream - commit not exist or not available - no more available account on jumper server Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-fail-eamil.rb | 38 ++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 lib/assign-account-fail-eamil.rb diff --git a/lib/assign-account-fail-eamil.rb b/lib/assign-account-fail-eamil.rb new file mode 100755 index 0000000..d15cd69 --- /dev/null +++ b/lib/assign-account-fail-eamil.rb @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_fail_email(my_info, message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: + + Your application for account failed with following errors + + #{message} + + In order to successfully apply a account, pay attention to the following points: + + 1. mail subject + the subject should exactly: apply account + + 2. commit url + when you writing the url, add prefix: my oss commit + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + + attention: + ensure you commit url exist and available to access. + + 3. ssh pubkey + you need to add a pubkey as an attachment with name 'id_rsa.pub' to the email. + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

2 2

[PATCH v4 compass-ci 05/12] mail-robot: check commit url for gitee
by Luan Shengde 11 Nov '20

11 Nov '20

GiteeCommitUrlCheck check commit url availability for hub gitee - clone the repo - check commit log for the commit usage: reference parse-apply-account-email.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/gitee-commit-url-check.rb | 44 +++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100755 lib/gitee-commit-url-check.rb diff --git a/lib/gitee-commit-url-check.rb b/lib/gitee-commit-url-check.rb new file mode 100755 index 0000000..11fbb6d --- /dev/null +++ b/lib/gitee-commit-url-check.rb @@ -0,0 +1,44 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# check commit url availability for hub gitee.com +# gitee_commit_index +# clone the repo +# check commit available +class GiteeCommitUrlCheck + def initialize(my_info, url, base_url) + @my_info = my_info + @url = url + @base_url = base_url + end + + def gitee_commit_index + repo_dir = @url.split('/')[-3] + repo_url = [@base_url, 'git'].join('.') + commit_id = @url.split('/')[-1] + + Dir.chdir '/tmp' + %x(/usr/bin/git clone #{repo_url} #{repo_dir}) + + email_index = %x(/usr/bin/git -C #{repo_dir} show #{commit_id}).index @my_info['my_email'] + + FileUtils.rm_rf repo_dir + + gitee_commit_exist(email_index) + end + + def gitee_commit_exist(email_index) + return unless email_index.nil? + + error_message = "We can not conform the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH v3 compass-ci] container: fix failed to build kibana images
by Liu Yinsi 11 Nov '20

11 Nov '20

[why] when build kibana images in x86_machine, error message: [root@localhost kibana]# ./build Sending build context to Docker daemon 5.12kB Step 1/3 : FROM gagara/kibana-oss-arm64:7.6.2 7.6.2: Pulling from gagara/kibana-oss-arm64 38163f410fa0: Pull complete 69a4d016f221: Pull complete 95e6c6e7c9ca: Pull complete d13f429dd982: Pull complete 508bb3330fb2: Pull complete 9634e726f1b6: Pull complete 9c26c37850c8: Pull complete 0d0ad8467060: Pull complete 940f92726f8b: Pull complete Digest: sha256:541632b7e9780a007f8a8be82ac8853ddcebcb04a596c00500b73f77eacfbd16 Status: Downloaded newer image for gagara/kibana-oss-arm64:7.6.2 ---> f482a0472f78 Step 2/3 : MAINTAINER Wu Zhende <wuzhende666(a)163.com> ---> Running in cfa86d8ce976 Removing intermediate container cfa86d8ce976 ---> 3be6c5f24d4b Step 3/3 : RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml ---> Running in ff455f66df8b standard_init_linux.go:220: exec user process caused "exec format error" libcontainer: container start initialization failed: standard_init_linux.go:220: exec user process caused "exec format error" The command '/bin/sh -c sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml' returned a non-zero code: 1 because arm base image not support to build in x86 machine. [how] 1. use images dict to store arm and x86 base images 2. use $(arch) to choose base image according to different system architecture Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/kibana/Dockerfile | 4 +++- container/kibana/build | 8 +++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 35802fe..6e0dba0 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -1,7 +1,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM gagara/kibana-oss-arm64:7.6.2 +ARG BASE_IMAGE + +FROM BASE_IMAGE # docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 diff --git a/container/kibana/build b/container/kibana/build index a7e4717..52d5a2a 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -3,4 +3,10 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -system 'docker build -t kibana:7.6.2 .' +BASE_IMAGE_DICT = { + 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', + 'x86_64' => 'kibana:7.6.2' }.freeze + +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] + +system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." -- 2.23.0

2 2

[PATCH v4 compass-ci 04/12] mail-robot: parse apply account email
by Luan Shengde 11 Nov '20

11 Nov '20

ParseApplyAccountEmail parse commit url and pubkey parse commit url check commit url exists check base url in upstream-repos check commit available hub: gitee.com call gitee-commit-url-check to check commit available hub: non-gitee.com check the feedback of curl command to check the commit url available parse pubkey attachment file name: id_rsa.pub usage: reference assign-account.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/parse-apply-account-email.rb | 115 +++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) create mode 100755 lib/parse-apply-account-email.rb diff --git a/lib/parse-apply-account-email.rb b/lib/parse-apply-account-email.rb new file mode 100755 index 0000000..98d4b9b --- /dev/null +++ b/lib/parse-apply-account-email.rb @@ -0,0 +1,115 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' + +# parse mail_content +# parse_commit_url +# extract_commit_url +# check commit url exists +# base_url_in_upstream_repos +# check base url in upstream-repos +# commit_url_availability +# check commit available +# parse pub_key +# check pub_key exists +class ParseApplyAccountEmail + def initialize(mail_content) + @mail_content = mail_content + + @my_info = { + 'my_email' => mail_content.from[0], + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + } + end + + def extract_commit_url + mail_content_body = @mail_content.part[0].part[0].body.decoded || @mail_content.part[0].body.decoded + mail_content_line = mail_content_body.gsub(/\n/, '') + + # the commit url should be headed with a prefix: my oss commit + # the commit url must be in a standart format, example: + # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + no_commit_url unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + + mail_content_body.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] + end + + def no_commit_url + error_message = "No matched commit url found.\n" + error_message += "Ensure that you have add a right commit url with prefix 'my oss commit:'." + + raise error_message + end + + def parse_commit_url + url = extract_commit_url + base_url = url.gsub(%r{/commit/[\w\d]{40}$}, '') + + base_url_in_upstream_repos('/c/upstream-repos', base_url) + commit_url_availability(url, base_url) + + return url + end + + def base_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + return unless match_out.empty? + + error_message = "The repo url for your commit is not in our upstream-repo list.\n" + error_message += 'Use a new one, or consulting the manager for available repo list.' + + raise error_message + end + + def commit_url_availability(url, base_url) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + if hub_name.eql? 'gitee.com' + gitee_commit(url, base_url) + else + non_gitee_commit(url) + end + end + + def gitee_commit(url, base_url) + my_gitee_commit = GiteeCommitUrlCheck.new(@my_info, url, base_url) + my_gitee_commit.gitee_commit_index + end + + def non_gitee_commit(url) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index @my_info['my_email'] + + return unless email_index.nil? + + error_message = "We can not conform the commit url matches your email.\n" + error_message += 'Make sure that the commit url is right,' + error_message += ' or it is truely submitted with you email.' + + raise error_message + end + + def parse_pub_key + pub_key = @mail_content.part[1].body.decoded if @mail_content.part[1].filename == 'id_rsa.pub' + pub_key_exist(pub_key) + + return pub_key + end + + def pub_key_exist(pub_key) + return unless pub_key.nil? + + error_message = "No pub_key found.\n" + error_message += 'Please add the pub_key as an attachment with filename: id_rsa.pub.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH v4 compass-ci 06/12] mail-robot: apply account for user
by Luan Shengde 11 Nov '20

11 Nov '20

ApplyJumperAccount apply jumper account for user required data: my_info: - my_email - my_name - my_uuid my_ssh_pubkey usage: reference assign-account.rb Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/apply-jumper-account.rb | 40 +++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 lib/apply-jumper-account.rb diff --git a/lib/apply-jumper-account.rb b/lib/apply-jumper-account.rb new file mode 100755 index 0000000..a34e147 --- /dev/null +++ b/lib/apply-jumper-account.rb @@ -0,0 +1,40 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' + +# apply jumper account for user +# apply_jumper_account +# apply jumper account with my_info +# check account exists +class ApplyJumperAccount + def initialize(my_info, my_ssh_pubkey) + @jumper_host = JUMPER_HOST + @jumper_port = JUMPER_PORT + @my_info = my_info.clone + @my_ssh_pubkey = my_ssh_pubkey + end + + def apply_jumper_account + @my_info['my_ssh_pubkey'] = @my_ssh_pubkey + + account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ + -d '#{(a)my_info.to_json}') + account_info = JSON.parse account_info_str + + account_info_exist(account_info) + + return account_info + end + + def account_info_exist(account_info) + return unless account_info['my_login_name'].nil? + + error_message = ' No more available jumper account.' + error_message += ' You may try again later or consulting the managet for a solution.' + + raise error_message + end +end -- 2.23.0

2 2

[PATCH compass-ci] sbin/compare optimize help message
by Lu Weitao 11 Nov '20

11 Nov '20

[why] compare tool support output result with json | html format, but haven't corresponding direction in help message. [how] add the corresponding direction in --theme option [example] compare id=crystal.109482 id=crystal.109483 --theme json Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/compare | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sbin/compare b/sbin/compare index b13682e..baa90d4 100755 --- a/sbin/compare +++ b/sbin/compare @@ -12,7 +12,7 @@ # compare "commit=a12d232e" "commit=b3bacc31" # compare "os=debian" "os=centos" -c "suite=iperf" # compare "os=centos" -d "os_version os_arch" -# compare "os=centos" -d "os_version os_arch" --color "classic" +# compare "os=centos" -d "os_version os_arch" --theme "classic" require 'optparse' require_relative '../lib/compare.rb' @@ -42,9 +42,9 @@ opt_parser = OptionParser.new do |opts| is_group = true end - opts.on('--color color', 'turn on colorful display with theme: classic|focus_good|focus_bad', - '|striking|light|none') do |color| - colorful = color + opts.on('--theme theme', 'turn on colorful display with theme: classic|focus_good|focus_bad', + '|striking|light|json|html') do |theme| + colorful = theme end opts.on('-t', '--template template', 'compare with user-defined template') do |t| -- 2.23.0

2 2

[PATCH v2 lkp-tests] jobs: add multi-device test parameters
by Zhang Yu 11 Nov '20

11 Nov '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- jobs/sysbench-mysql.yaml | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml index c060fc2a..83795a18 100644 --- a/jobs/sysbench-mysql.yaml +++ b/jobs/sysbench-mysql.yaml @@ -1,10 +1,22 @@ suite: sysbench-mysql category: benchmark -sysbench-mysql: - oltp_test_mode: complex - oltp_tables_count: 3 - oltp_table_size: 1000 - nr_threads: 64 - runtime: 120 - report_interval: 10 +cluster: cs-lkp-hsw-ep5 + +if role server: + mysql: + +if role client: + sysbench-mysql: + oltp_test_mode: complex + max_requests: 0 + mysql_table_engine: innodb + oltp_tables_count: 1000 + oltp_table_size: 100000 + nr_threads: 256 + runtime: 600 + rand_type: special + rand_spec_pct: 100 + oltp_read_only: off + report_interval: 1 + events: 2000000000 -- 2.23.0

1 0

[PATCH v7 lkp-tests 1/2] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 11 Nov '20

11 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=1.1 BUILDFILE -> PKGBUILD BUILDFILE-$PKGBUILD_TAG -> PKGBUILD-1.1 PKGBUILD-1.1 contains only the changed part. % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..fde82212 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -889,6 +889,7 @@ update_pkgver() { fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" source_safe "$BUILDFILE" + [[ -n $PKGBUILD_TAG ]] && source_safe "$BUILDFILE-$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -3692,6 +3693,7 @@ else BUILDFILE="$startdir/$BUILDFILE" fi source_buildfile "$BUILDFILE" + [[ -n $PKGBUILD_TAG ]] && source_buildfile "$BUILDFILE-$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

1 0

[PATCH v7 lkp-tests 2/2] tests/cci-makepkg: add download PKGBUILD patch function
by Lin Jiaxin 11 Nov '20

11 Nov '20

[Why] Support more version's PKGBUILD. [How] Download the required file through remote-git from /srv/git/pkg/ usage: job yaml: PKGBUILD_TAG: 1.0-20 About $PKGBUILD_TAG, it is from PKGBUILD file's $pkgver-$pkgrel. You should create a git repo that name same as $benchmark in /srv/git/pkg/ before you add PKGBUILD_TAG to yaml. If not, cci-makepkg will report an error. And everything same as before if no PKGBUILD_TAG. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index a105f189..67d055a4 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -39,12 +39,29 @@ cd $LKP_SRC/pkg/$benchmark || die "pkg is empty" mount -t cifs -o guest,vers=1.0,noacl,nouser_xattr //$LKP_SERVER$PKG_MNT $PKG_MNT || die "Failed to run mount" } +download_pkgfile() +{ + curl -sS -H 'Content-Type: Application/json' -XPOST "$LKP_SERVER"':8100/git_command' \ + -d '{"git_repo": "'$1'", "git_command": ["git-show", "HEAD:'$2'"]}' -o "$2" +} + +get_pkgfile() +{ + PKGBUILD_TAG_FILE="PKGBUILD-$PKGBUILD_TAG" + + download_pkgfile "pkg/$benchmark" "$PKGBUILD_TAG_FILE" + + [ -f "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE not exist" + [ -s "$PKGBUILD_TAG_FILE" ] || die "$PKGBUILD_TAG_FILE is empty" +} + get_pkg_info() { var=$1 pkg_dir=${2:-.} ( . $pkg_dir/PKGBUILD + [ -n "$PKGBUILD_TAG" ] && . $pkg_dir/PKGBUILD-$PKGBUILD_TAG eval echo '$'$var ) } @@ -71,6 +88,8 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +[ -n "$PKGBUILD_TAG" ] && get_pkgfile + distro_install_depends lkp-dev [ $os = "centos" ] && -- 2.23.0

1 1

[PATCH v7 lkp-tests 1/2] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 11 Nov '20

11 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 PKGBUILD-1.1 contains only the changed part. % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..fde82212 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -889,6 +889,7 @@ update_pkgver() { fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" source_safe "$BUILDFILE" + [[ -n $PKGBUILD_TAG ]] && source_safe "$BUILDFILE-$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -3692,6 +3693,7 @@ else BUILDFILE="$startdir/$BUILDFILE" fi source_buildfile "$BUILDFILE" + [[ -n $PKGBUILD_TAG ]] && source_buildfile "$BUILDFILE-$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

1 1

[PATCH compass-ci 3/4] assign-account: generate rsa keys
by Luan Shengde 11 Nov '20

11 Nov '20

generate rsa private/public key and return public key on jumper server [why] register the public key on testbox enable user one click to login in the testbox [how] generate rsa private/public key for user Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 32 ++++++++++++-------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 137f5c6..b5a2388 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -94,30 +94,36 @@ class AccountStorage end def setup_jumper_account_info - account_info = read_account_info - jumper_info = read_jumper_info + login_name, password = read_account_info + jumper_host, jumper_port = read_jumper_info pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? - login_name = account_info[0] - password = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + password = 'Use pub_key to login' unless pub_key.nil? + + config_authorized_key(login_name, pub_key) unless pub_key.nil? + config_default_yaml(login_name) + my_jumper_pubkey = generate_ssh_key(login_name) if @data['my_jumper_pubkey'].eql? true jumper_account_info = { 'my_login_name' => login_name, 'my_password' => password, - 'jumper_host' => jumper_info[0].chomp, - 'jumper_port' => jumper_info[1].chomp + 'jumper_host' => jumper_host, + 'jumper_port' => jumper_port, + 'my_jumper_pubkey' => my_jumper_pubkey } - config_authorized_key(login_name, pub_key) unless pub_key.nil? - config_default_yaml(login_name) - return jumper_account_info end + def generate_ssh_key(login_name) + ssh_dir = File.join('/home/', login_name, '.ssh') + Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + hostname = File.read('/etc/hostname') + %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@#{hostname}") + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) + File.read("/home/#{login_name}/.ssh/id_rsa.pub") + end + def config_default_yaml(login_name) default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') FileUtils.mkdir_p default_yaml_dir -- 2.23.0

1 0

[PATCH lab-z9] hosts: fix testbox included yaml error [Why] submit job with some physical machine's textbox will error: Included yaml file not found: 'hosts/taishan200-2280-2s48p-256g' (RuntimeError)
by Wei Jihui 11 Nov '20

11 Nov '20

[How] lkp-tests just included files that find $LKP_SRC and testbox's directory, so delete hosts is ok. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a103 | 2 +- hosts/taishan200-2280-2s48p-256g--a104 | 2 +- hosts/taishan200-2280-2s48p-256g--a106 | 2 +- hosts/taishan200-2280-2s48p-256g--a107 | 2 +- hosts/taishan200-2280-2s48p-512g--a105 | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/hosts/taishan200-2280-2s48p-256g--a103 b/hosts/taishan200-2280-2s48p-256g--a103 index 304c6fc..9fcc718 100644 --- a/hosts/taishan200-2280-2s48p-256g--a103 +++ b/hosts/taishan200-2280-2s48p-256g--a103 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bdc69 diff --git a/hosts/taishan200-2280-2s48p-256g--a104 b/hosts/taishan200-2280-2s48p-256g--a104 index a8a2789..41cdbb0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a104 +++ b/hosts/taishan200-2280-2s48p-256g--a104 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8892431 diff --git a/hosts/taishan200-2280-2s48p-256g--a106 b/hosts/taishan200-2280-2s48p-256g--a106 index ad587b3..95b2f05 100644 --- a/hosts/taishan200-2280-2s48p-256g--a106 +++ b/hosts/taishan200-2280-2s48p-256g--a106 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b883bf99 diff --git a/hosts/taishan200-2280-2s48p-256g--a107 b/hosts/taishan200-2280-2s48p-256g--a107 index fba1937..31b8dc6 100644 --- a/hosts/taishan200-2280-2s48p-256g--a107 +++ b/hosts/taishan200-2280-2s48p-256g--a107 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8917c39 diff --git a/hosts/taishan200-2280-2s48p-512g--a105 b/hosts/taishan200-2280-2s48p-512g--a105 index 096b5b0..66a0aca 100644 --- a/hosts/taishan200-2280-2s48p-512g--a105 +++ b/hosts/taishan200-2280-2s48p-512g--a105 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-512g +<<: taishan200-2280-2s48p-512g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b891945d -- 2.23.0

2 1

[PATCH v4 lkp-tests] job2sh.rb: define on_fail
by Hu Xuejiao 11 Nov '20

11 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh export_top_envexport_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7e6a5d7c..370c7899 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -358,6 +360,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

3 2

[PATCH v4 compass-ci 09/12] mail-robot: documentation for service
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..f829199 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. assign-account + init my_info + - my_email + - my_name + send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

2 2

[PATCH v6 lkp-tests 1/2] sbin/makepkg: support applying patch for PKGBUILD
by Lin Jiaxin 11 Nov '20

11 Nov '20

[Why] Support more version's PKGBUILD. [How] Replace the previous content with two source command. example: PKGBUILD_TAG=-1.1 BUILDFILE -> PKGBUILD BUILDFILE$PKGBUILD_TAG -> PKGBUILD-1.1 PKGBUILD-1.1 contains only the changed part. % cat PKGBUILD-1.1 pkgver=1.1 pkgrel=2 build() { make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" } Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..fbe9c432 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -888,7 +888,7 @@ update_pkgver() { exit 1 fi sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" - source_safe "$BUILDFILE" + source_safe "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -1604,10 +1604,14 @@ cd_safe() { source_safe() { shopt -u extglob - if ! source "$@"; then - error "$(gettext "Failed to source %s")" "$1" - exit 1 - fi + local file + for file in "$@" + do + if ! source "$file"; then + error "$(gettext "Failed to source %s")" "$file" + exit 1 + fi + done shopt -s extglob } @@ -1626,7 +1630,6 @@ merge_arch_attrs() { source_buildfile() { source_safe "$@" - if (( !SOURCEONLY )); then merge_arch_attrs fi @@ -3661,6 +3664,7 @@ SRCEXT=${_SRCEXT:-$SRCEXT} GPGKEY=${_GPGKEY:-$GPGKEY} PACKAGER=${_PACKAGER:-$PACKAGER} CARCH=${_CARCH:-$CARCH} +PKGBUILD_TAG=$PKGBUILD_TAG if (( INFAKEROOT )); then if [[ -z $FAKEROOTKEY ]]; then @@ -3691,7 +3695,7 @@ else if [[ ${BUILDFILE:0:1} != "/" ]]; then BUILDFILE="$startdir/$BUILDFILE" fi - source_buildfile "$BUILDFILE" + source_buildfile "$BUILDFILE" "$BUILDFILE$PKGBUILD_TAG" fi # set defaults if they weren't specified in buildfile -- 2.23.0

3 4

[PATCH lkp-tests] depends: add depends to sysbench-mysql
by Zhang Yu 11 Nov '20

11 Nov '20

Use source code install mysql, so no need mysql-server here. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/sysbench-mysql | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/distro/depends/sysbench-mysql b/distro/depends/sysbench-mysql index e9585216..cfbc4518 100644 --- a/distro/depends/sysbench-mysql +++ b/distro/depends/sysbench-mysql @@ -1,2 +1,2 @@ -mysql-server -libmysqlclient-dev +libnuma1 +xfsprogs -- 2.23.0

1 0

[PATCH v1 compass-ci] splitting the cifs services
by Xiao Shenwei 11 Nov '20

11 Nov '20

[why] our os-cifs container will mount three dir, /srv/os /srv/initrd /srv/result it's may cause: 1. our cifs service becomes slow 2. if the container exit, none of the three folders can be accessed. in order to reduce the coupling, need split them [how] one folder mount by one container usage: if you want mount these floder, you can use these command server: /srv/os client: mount -t cifs -o guest //ip/os /tmp/os server: /srv/initrd client: mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd server: /srv/result client: mount -t cifs -o guest,port=447 //ip/result /tmp/result --- container/os-cifs/start | 30 ++++++++++++++++++++++++++---- 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/container/os-cifs/start b/container/os-cifs/start index 6cc08c6..e20f887 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -8,18 +8,40 @@ lsmod | grep -q "^cifs\s" || { sudo modprobe cifs } -docker_rm samba +docker_rm os-cifs +docker_rm initrd-cifs +docker_rm result-cifs -cmd=( +os_cifs_cmd=( docker run -dt -p 445:445 + -v /etc/localtime:/etc/localtime:ro -v /srv/os:/srv/os + --name os-cifs + --restart=always + alpine/samba +) + +initrd_cifs_cmd=( + docker run -dt + -p 446:445 -v /etc/localtime:/etc/localtime:ro -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + alpine/samba +) + +result_cifs_cmd=( + docker run -dt + -p 447:445 + -v /etc/localtime:/etc/localtime:ro -v /srv/result:/srv/result - --name os-cifs + -name result-cifs --restart=always alpine/samba ) -"${cmd[@]}" +"${os_cifs_cmd[@]}" +"${initrd_cifs_cmd[@]}" +"${result_cifs_cmd[@]}" -- 2.23.0

2 2

[PATCH lkp-tests] job2sh: remove atomic job test
by Wu Fengguang 11 Nov '20

11 Nov '20

Make it easier to do simple demo. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/job2sh | 6 ------ sbin/job2sh.cr | 6 ------ 2 files changed, 12 deletions(-) diff --git a/sbin/job2sh b/sbin/job2sh index 4fb57453..9ada3cfa 100755 --- a/sbin/job2sh +++ b/sbin/job2sh @@ -38,12 +38,6 @@ begin job = Job2sh.new job.load(ARGV[0]) job.expand_params - - unless job.atomic_job? - log_error "Looks #{ARGV[0]} isn't a atomic jobfile, only atomic jobfile is supported" - log_error "Please run 'lkp split-job #{ARGV[0]}' first" - exit 1 - end rescue Job::ParamError => e log_error "Abandon job: #{e.message}" exit 1 diff --git a/sbin/job2sh.cr b/sbin/job2sh.cr index ffcd3359..3a921bd6 100644 --- a/sbin/job2sh.cr +++ b/sbin/job2sh.cr @@ -28,12 +28,6 @@ begin job = Job2sh.new job.load(ARGV[0]) job.expand_params - - unless job.atomic_job? - log_error "Looks #{ARGV[0]} isn't a atomic jobfile, only atomic jobfile is supported" - log_error "Please run 'lkp split-job #{ARGV[0]}' first" - exit(1) - end rescue e : Job::ParamError log_error "Abandon job: #{e.message}" exit(1) -- 2.23.0

1 0

[PATCH lkp-tests] tests: fix lmbench3 tool adapt to aarch64 framework
by Bai Jing 11 Nov '20

11 Nov '20

error: cat: results/x84_64-linux-gnu/taishan200-2280-2s48p-256g--a28.0 No such file or directory Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/lmbench3 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/lmbench3 b/tests/lmbench3 index 8570a35a..dc4d157d 100755 --- a/tests/lmbench3 +++ b/tests/lmbench3 @@ -138,7 +138,7 @@ run_lmbench3_development() eval "$test=yes" - sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/x86_64-linux-gnu/lmbench + sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/"${os_arch}"-linux-gnu/lmbench log_eval " ( echo $nr_threads @@ -177,7 +177,7 @@ run_lmbench3_development() show_result() { - cat results/x86_64-linux-gnu/$testbox.0 || { + cat results/"${os_arch}"-linux-gnu/$testbox.0 || { echo "no test result found" exit -1 } -- 2.23.0

1 0

[PATCH v4 compass-ci] container: fix failed to build archlinux images in x86_64 machine
by Liu Yinsi 11 Nov '20

11 Nov '20

[why] when build archlinux images in x86_64 machine error: error: failed retrieving file 'core.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'core.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'core.db' from mirror.archlinuxarm.org : The requested URL returned error: 404 error: failed to update core (failed to retrieve some files) error: failed retrieving file 'extra.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error:404 error: failed retrieving file 'extra.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'extra.db' from mirror.archlinuxarm.org : Resolving timed out after 10000 milliseconds error: failed to update extra (download library error) error: failed retrieving file 'community.db' from mirrors.tuna.tsinghua.edu.cn : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirrors.163.com : The requested URL returned error: 404 error: failed retrieving file 'community.db' from mirror.archlinuxarm.org : The requested URL returned error:404 error: failed to update community (failed to retrieve some files) error: failed to synchronize all databases The command '/bin/sh -c pacman --needed --noprogressbar --noconfirm -Syu && pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk' returned a non-zero code: 1 because archlinux mirror not support x86_64 machine. [how] root/etc/pacman.d/mirrorlist => root/aarch64/etc/pacman.d/mirrorlist root/x86_64/etc/pacman.d/mirrorlist use ARCH=$(arch) to choose mirrorlist according to different system architecture. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/archlinux/Dockerfile | 6 +++++- container/archlinux/build | 2 +- .../archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist | 0 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist | 1 + 4 files changed, 7 insertions(+), 2 deletions(-) rename container/archlinux/root/{ => aarch64}/etc/pacman.d/mirrorlist (100%) create mode 100644 container/archlinux/root/x86_64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/Dockerfile b/container/archlinux/Dockerfile index c0f05d3..1f80ae0 100644 --- a/container/archlinux/Dockerfile +++ b/container/archlinux/Dockerfile @@ -5,7 +5,11 @@ FROM lopsided/archlinux MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> -COPY root / +ARG ARCH + +COPY root/$ARCH / + RUN chmod 755 /etc + RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim gawk diff --git a/container/archlinux/build b/container/archlinux/build index 81feda2..9749489 100755 --- a/container/archlinux/build +++ b/container/archlinux/build @@ -2,4 +2,4 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t archlinux:testbed . +docker build --build-arg ARCH=$(arch) -t archlinux:testbed . diff --git a/container/archlinux/root/etc/pacman.d/mirrorlist b/container/archlinux/root/aarch64/etc/pacman.d/mirrorlist similarity index 100% rename from container/archlinux/root/etc/pacman.d/mirrorlist rename to container/archlinux/root/aarch64/etc/pacman.d/mirrorlist diff --git a/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist new file mode 100644 index 0000000..556fac8 --- /dev/null +++ b/container/archlinux/root/x86_64/etc/pacman.d/mirrorlist @@ -0,0 +1 @@ +Server = http://mirrors.tuna.tsinghua.edu.cn/archlinux/$repo/os/$arch -- 2.23.0

1 0

[PATCH compass-ci] container: keep docker_rm '$containername' consistent with name
by Liu Yinsi 11 Nov '20

11 Nov '20

because --name=os-cifs, docker_rm os-cifs should be the same. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/os-cifs/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/os-cifs/start b/container/os-cifs/start index 6cc08c6..1b2cd23 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -8,7 +8,7 @@ lsmod | grep -q "^cifs\s" || { sudo modprobe cifs } -docker_rm samba +docker_rm os-cifs cmd=( docker run -dt -- 2.23.0

1 0

[PATCH v4 lkp-tests] submit on_fail to the scheduler
by Hu Xuejiao 11 Nov '20

11 Nov '20

[why] When job2sh.rb defined sh_on_fail, it needs to be invoked by the scheduler. Finally, the task is in the waiting state when the task is failed. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index bc90e2e0..0bb41f2d 100755 --- a/sbin/submit +++ b/sbin/submit @@ -115,12 +115,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_fail sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

2 1

[PATCH v4 lkp-tests] execute on_fail when exit-code isn't 0
by Hu Xuejiao 11 Nov '20

11 Nov '20

[why] when a job-task fails(exit_code != 0), the on_fail function needs to be called to keep the program in the waiting state to user adjustment Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- bin/run-lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/run-lkp b/bin/run-lkp index 26669331..c4d43db4 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -97,6 +97,7 @@ elif [ "$run_job_exit_code" != 0 ]; then else set_job_state 'failed' fi + $LKP_DEBUG_PREFIX $job_script on_fail elif [ -f '/var/log/wtmp' ] && has_cmd last && last -n1 | grep -q -E '(pts/|tty)[0-9].*(still logged in)'; then set_job_state 'disturbed' disturbed=1 -- 2.23.0

1 0

[PATCH compass-ci] git_mirror: fix that worker thread die after system(git clone)
by Li Yuanchao 11 Nov '20

11 Nov '20

[why] As Signal.trap(:SIGCHLD, 'SIG_IGN') has been set, the init process will handle the return status of child process, so ret = system(git clone xxx) can't get return value, which cause worker thread die. Use %x() instead of system(), and use stderr to check if 'git clone' succeed. So return value is not needed. [errmsg] in `system': No child process - Another thread waited the process started by system(). (Errno::ECHILD) Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 6e6664a..99534f6 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -38,7 +38,8 @@ class GitMirror url = "/srv/git/#{url.delete_prefix('https://')}" end 10.times do - ret = system("git clone --mirror #{url} #{mirror_dir}") + stderr = %x(git clone --mirror #{url} #{mirror_dir} 2>&1) + ret = !stderr.include?('fatal') break if ret end FileUtils.rm_r(mirror_dir) unless ret -- 2.23.0

1 0

[PATCH v4 lkp-tests] job2sh.rb: define on_fail
by Hu Xuejiao 11 Nov '20

11 Nov '20

[why] If job.yaml is defined: on_fail: sleep(tests): 300(runtime) [how] By sh_on_fail, it will appear in job.sh: on_fail() { . $LKP_SRC/lib/http.sh . $LKP_SRC/lib/job.sh export_top_envexport_top_env run_test $LKP_SRC/tests/wrapper sleep 500 } Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- lib/job2sh.rb | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 7e6a5d7c..370c7899 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,11 +266,13 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines sh_define_files() + local_script_lines += @script_lines + + sh_on_fail(job) out_line '"$@"' @@ -358,6 +360,28 @@ class Job2sh < Job out_line "\tchmod +x $LKP_SRC/#{file}" end end - out_line '}' + out_line "}\n\n" + end + + def sh_on_fail(job = nil) + @script_lines = [] + + @programs = available_programs(:workload_elements) + + job ||= (@jobx || @job).clone + + @cur_func = :on_fail + + out_line 'on_fail()' + out_line '{' + out_line + out_line "\t. $LKP_SRC/lib/http.sh" + out_line "\t. $LKP_SRC/lib/job.sh" + out_line "\t. $LKP_SRC/lib/env.sh" + out_line + out_line "\texport_top_env" + out_line + parse_hash [], job['on_fail'] + out_line "}" end end -- 2.23.0

2 1

[PATCH v2 compass-ci] job.cr: obtain the cgz package of a specified version
by Zhang Yu 11 Nov '20

11 Nov '20

[why] We need to use specified version of performance tool to test. But the tool package uses latest.cgz by default. [how] Job yaml: ${program}_version: 1.0-19 About ${program}_version's value, it is from PKGBUILD file's $pkgver-$pkgrel. Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- src/lib/job.cr | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 07873ab..d10a1e9 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -424,8 +424,14 @@ class Job program = $1 end + if @hash["#{program}_version"]? + program_version = @hash["#{program}_version"] + else + program_version = "latest" + end + deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/latest.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}/#{program_version}.cgz" if File.exists?(deps_dest_file) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) -- 2.23.0

1 0

[PATCH lkp-tests] sbin/makepkg: download submodules code from LKP server
by Liu Shaofei 10 Nov '20

10 Nov '20

[why] Some repository have submodules. So when build packages, we need download submodules from LKP server, avoid the waste of time in downloading files from the remote end. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 25773621..afa665aa 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -623,6 +623,16 @@ extract_git() { cd_safe "${dir##*/}" + # modify submodules address to local server address + # url = https://github.com/nico/demumble.git + # => + # url = git://172.168.131.113/submodules/b/bloaty/demumble.git + if [[ -f ".gitmodules" ]]; then + local project=${upstream_repo%/*} + local url="git://${LKP_SERVER}/submodules/${project}/" + sed -i "s|url = .*\/|${url}|g" ".gitmodules" + fi + local ref=origin/HEAD if [[ -n $fragment ]]; then case ${fragment%%=*} in -- 2.23.0

2 2

[PATCH v4 compass-ci 12/12] mail-robot: start file
by Luan Shengde 10 Nov '20

10 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..035d4bf --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +defaults['CCI_SRC'] = '/c/compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v /srv/cci/Maildir/.compass-ci:/srv/cci/Maildir/.compass-ci:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'run-mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v4 compass-ci 11/12] mail-robot: build file
by Luan Shengde 10 Nov '20

10 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v4 compass-ci 10/12] mail-robot: Dockerfile
by Luan Shengde 10 Nov '20

10 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v4 compass-ci 07/12] mail-robot: build success email
by Luan Shengde 10 Nov '20

10 Nov '20

build email message for successfully assigning account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/assign-account-email.rb | 53 +++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100755 lib/assign-account-email.rb diff --git a/lib/assign-account-email.rb b/lib/assign-account-email.rb new file mode 100755 index 0000000..2bed2c1 --- /dev/null +++ b/lib/assign-account-email.rb @@ -0,0 +1,53 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def build_apply_account_email(my_info) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] Account Ready + + Dear #{my_info['my_name']}, + + Thank you for joining us. + + You can use the following info to submit jobs: + + notice: + the followings are ONE-TIME setup + + 1) setup default config + run the following command to add the below setup to default config file + + mkdir -p ~/.config/compass-ci/defaults/ + cat >> ~/.config/compass-ci/defaults/account.yaml <<-EOF + my_email: #{my_info['my_email']} + my_name: #{my_info['my_name']} + my_uuid: #{my_info['my_uuid']} + EOF + + 2) download lkp-tests and dependencies + run the following command to install the lkp-test and effect the configuration + + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source ${HOME}/.bashrc && source ${HOME}/.bash_profile + + 3) submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + + reference to 'how to write job yaml' section to write the job yaml + you can alse reference to files in compass-ci/jobs as example. + + submit jobs, for example: + + submit jobs/iperf.yaml textbox=vm-2p8g + + regards + compass-ci + EMAIL_MESSAGE + + return email_msg +end -- 2.23.0

1 0

[PATCH v4 compass-ci 02/12] mail-robot: listen and handle new email
by Luan Shengde 10 Nov '20

10 Nov '20

listen the mailbox for new email handle_new_email handle the email with defined opetions - apply_account email subject: apply account Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-robot.rb | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100755 lib/mail-robot.rb diff --git a/lib/mail-robot.rb b/lib/mail-robot.rb new file mode 100755 index 0000000..0ae80f3 --- /dev/null +++ b/lib/mail-robot.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' + +def monitor_new_email(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + next if added.empty? + + added.each do |mail_file| + handle_new_email(mail_file, mail_drafts) + end + end + listener.start + sleep +end + +def handle_new_email(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + apply_account(mail_content) + + FileUtils.mv(mail_file, mail_drafts) +end + +def apply_account(mail_content) + return unless mail_content.subject == 'apply account' + + assign_uuid = AssignAccount.new(mail_content) + assign_uuid.send_account +end -- 2.23.0

1 0