November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] support queues parameter
by Xiao Shenwei 12 Nov '20

12 Nov '20

register host2queues and requested specified queues Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 47 ++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 4e6bc2f..8e489c6 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -11,15 +11,26 @@ require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' +names = Set.new %w[ + SCHED_HOST + SCHED_PORT +] +defaults = relevant_defaults(names) +SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = defaults['SCHED_PORT'] || 3000 + def get_url(hostname) - names = Set.new %w[ - SCHED_HOST - SCHED_PORT - ] - defaults = relevant_defaults(names) - host = defaults['SCHED_HOST'] || '172.17.0.1' - port = defaults['SCHED_PORT'] || 3000 - "http://#{host}:#{port}/boot.container/hostname/#{hostname}" + "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" +end + +def set_host2queues(hostname, queues) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/set_host2queues?host=#{hostname}&queues=#{queues}'" + system cmd +end + +def del_host2queues(hostname) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/del_host2queues?host=#{hostname}'" + system cmd end def parse_response(url) @@ -67,7 +78,7 @@ def load_initrds(load_path, hash) wget_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") end -def run(hostname, load_path, hash) +def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( @@ -77,24 +88,26 @@ def run(hostname, load_path, hash) clean_dir(load_path) end -def main(hostname) +def main(hostname, queues) + set_host2queues(hostname, queues) url = get_url hostname puts url hash = parse_response url - return if hash.nil? + return del_host2queues(hostname) if hash.nil? load_path = build_load_path(hostname) load_initrds(load_path, hash) - run(hostname, load_path, hash) + start_container(hostname, load_path, hash) + del_host2queues(hostname) end -def loop_main(hostname) +def loop_main(hostname, queues) loop do begin - main(hostname) + main(hostname, queues) rescue StandardError => e puts e.backtrace - # if an exception happend, request the next time after 30 seconds + # if an exception occurs, request the next time after 30 seconds sleep 25 ensure sleep 5 @@ -109,11 +122,11 @@ def save_pid(pids) f.close end -def multi_docker(hostname, nr_container) +def multi_docker(hostname, nr_container, queues) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop_main("#{hostname}-#{i}") + loop_main("#{hostname}-#{i}", queues) end pids << pid end -- 2.23.0

2 2

[PATCH compass-ci 2/4] assign-account: add -r|--rsa-key option
by Luan Shengde 12 Nov '20

12 Nov '20

add -r|--rsa-key option for answerback-email [why] enable generate rsa private/public key on jumper server enable one click login whit the public key [how] add -r|--rsa-key option to enable generate rsa private/public key Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 686a327..6d8e3de 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -32,7 +32,8 @@ my_info = { 'my_email' => nil, 'my_name' => nil, 'my_uuid' => %x(uuidgen).chomp, - 'my_ssh_pubkey' => nil + 'my_ssh_pubkey' => nil, + 'my_jumper_pubkey' => false } def init_info(email_file, my_info) @@ -68,6 +69,10 @@ options = OptionParser.new do |opts| init_info(email_file, my_info) end + opts.on('-r', '--rsa-key', 'generate jumper ras public/private key and return pubkey') do + my_info['my_jumper_pubkey'] = true + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -114,10 +119,12 @@ def send_account(my_info) raise message if my_info['my_email'].nil? account_info = apply_account(my_info) + # for manually assign account, there will be no my_commit_url # but the key my_commit_url is required for es my_info['my_commit_url'] = '' my_info['my_login_name'] = account_info['my_login_name'] + my_info['my_jumper_pubkey'] = account_info['my_jumper_pubkey'] my_info.delete 'my_ssh_pubkey' store_account_info(my_info) message = build_message(my_info['my_email'], account_info) -- 2.23.0

2 2

[PATCH compass-ci 1/4] assign-account: Dockerfile add install ssh
by Luan Shengde 12 Nov '20

12 Nov '20

add install ssh for assign-account [why] it's need to generate rsa private/public key for user need install ssh for ssh-keygen command [how] add install ssh Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/Dockerfile | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/container/assign-account/Dockerfile b/container/assign-account/Dockerfile index f84b5e6..c1d360e 100644 --- a/container/assign-account/Dockerfile +++ b/container/assign-account/Dockerfile @@ -2,15 +2,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER luanshd0525(a)163.com +MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ruby-sinatra ruby-json curl && \ - ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ - echo 'Asia/Shanghai' > /etc/timezone + apt-get install -y ssh ruby-sinatra ruby-json curl + COPY assign-account.rb /usr/local/bin/ COPY get_account_info.rb /usr/local/bin/ -- 2.23.0

2 2

[PATCH compass-ci] support queues parameter
by Xiao Shenwei 12 Nov '20

12 Nov '20

register host2queues and requested specified queues Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 47 ++++++++++++++++++++++++-------------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 4e6bc2f..8e489c6 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -11,15 +11,26 @@ require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' +names = Set.new %w[ + SCHED_HOST + SCHED_PORT +] +defaults = relevant_defaults(names) +SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' +SCHED_PORT = defaults['SCHED_PORT'] || 3000 + def get_url(hostname) - names = Set.new %w[ - SCHED_HOST - SCHED_PORT - ] - defaults = relevant_defaults(names) - host = defaults['SCHED_HOST'] || '172.17.0.1' - port = defaults['SCHED_PORT'] || 3000 - "http://#{host}:#{port}/boot.container/hostname/#{hostname}" + "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" +end + +def set_host2queues(hostname, queues) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/set_host2queues?host=#{hostname}&queues=#{queues}'" + system cmd +end + +def del_host2queues(hostname) + cmd = "curl -X PUT 'http://#{SCHED_HOST}:#{SCHED_PORT}/del_host2queues?host=#{hostname}'" + system cmd end def parse_response(url) @@ -67,7 +78,7 @@ def load_initrds(load_path, hash) wget_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") end -def run(hostname, load_path, hash) +def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( @@ -77,24 +88,26 @@ def run(hostname, load_path, hash) clean_dir(load_path) end -def main(hostname) +def main(hostname, queues) + set_host2queues(hostname, queues) url = get_url hostname puts url hash = parse_response url - return if hash.nil? + return del_host2queues(hostname) if hash.nil? load_path = build_load_path(hostname) load_initrds(load_path, hash) - run(hostname, load_path, hash) + start_container(hostname, load_path, hash) + del_host2queues(hostname) end -def loop_main(hostname) +def loop_main(hostname, queues) loop do begin - main(hostname) + main(hostname, queues) rescue StandardError => e puts e.backtrace - # if an exception happend, request the next time after 30 seconds + # if an exception occurs, request the next time after 30 seconds sleep 25 ensure sleep 5 @@ -109,11 +122,11 @@ def save_pid(pids) f.close end -def multi_docker(hostname, nr_container) +def multi_docker(hostname, nr_container, queues) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop_main("#{hostname}-#{i}") + loop_main("#{hostname}-#{i}", queues) end pids << pid end -- 2.23.0

2 2

[PATCH v2 lab-z9] hosts: fix testbox included yaml error
by Wei Jihui 11 Nov '20

11 Nov '20

[Why] submit job with some physical machine's textbox will error: Included yaml file not found: 'hosts/taishan200-2280-2s48p-256g' (RuntimeError) [How] lkp-tests just included files that find $LKP_SRC and testbox's directory, so delete hosts is ok. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a103 | 2 +- hosts/taishan200-2280-2s48p-256g--a104 | 2 +- hosts/taishan200-2280-2s48p-256g--a106 | 2 +- hosts/taishan200-2280-2s48p-256g--a107 | 2 +- hosts/taishan200-2280-2s48p-512g--a105 | 2 +- 5 files changed, 5 insertions(+), 5 deletions(-) diff --git a/hosts/taishan200-2280-2s48p-256g--a103 b/hosts/taishan200-2280-2s48p-256g--a103 index 304c6fc..9fcc718 100644 --- a/hosts/taishan200-2280-2s48p-256g--a103 +++ b/hosts/taishan200-2280-2s48p-256g--a103 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bdc69 diff --git a/hosts/taishan200-2280-2s48p-256g--a104 b/hosts/taishan200-2280-2s48p-256g--a104 index a8a2789..41cdbb0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a104 +++ b/hosts/taishan200-2280-2s48p-256g--a104 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8892431 diff --git a/hosts/taishan200-2280-2s48p-256g--a106 b/hosts/taishan200-2280-2s48p-256g--a106 index ad587b3..95b2f05 100644 --- a/hosts/taishan200-2280-2s48p-256g--a106 +++ b/hosts/taishan200-2280-2s48p-256g--a106 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b883bf99 diff --git a/hosts/taishan200-2280-2s48p-256g--a107 b/hosts/taishan200-2280-2s48p-256g--a107 index fba1937..31b8dc6 100644 --- a/hosts/taishan200-2280-2s48p-256g--a107 +++ b/hosts/taishan200-2280-2s48p-256g--a107 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-256g +<<: taishan200-2280-2s48p-256g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8917c39 diff --git a/hosts/taishan200-2280-2s48p-512g--a105 b/hosts/taishan200-2280-2s48p-512g--a105 index 096b5b0..66a0aca 100644 --- a/hosts/taishan200-2280-2s48p-512g--a105 +++ b/hosts/taishan200-2280-2s48p-512g--a105 @@ -1,4 +1,4 @@ -<<: hosts/taishan200-2280-2s48p-512g +<<: taishan200-2280-2s48p-512g nr_hdd_partitions: 4 hdd_partitions: - /dev/disk/by-id/scsi-350000399b891945d -- 2.23.0

2 1

[PATCH v2 compass-ci] /lib: keep suite result in top for compare output
by Lu Kaiyi 11 Nov '20

11 Nov '20

[why] Currently, the output result of compare default order by alphabet. If there have too many lines in result, it will be difficult for user to find the lines that starts with $suite in metric column. [how] Divide the lines into two parts before print according to whether the line start with $suite or not, then combine output result into a whole to highlight the important info. [example] command: compare "os=openeuler" -d "tbox_group" in z9 one of group compare output result. suites_list:["iso2rootfs", "build-pkg", "deploy-cci"] Before: taishan200-2280-2s64p-256g--a101 vm-hi1620-2p8g--roota15 vm-hi1620-2p8g--roota14 taishan200-2280-2s64p-256g--a6 metric -------------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------ %stddev change %stddev change %stddev change %stddev \ | \ | \ | \ 493.41 ± 108% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.boot 47114.17 ± 109% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.idle 11.57 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.kernel_boot 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time.max 0.00 0 54815.06 ± 124% 0 52113.21 ± 173% 0 0.00 build-pkg.time.file_system_inputs 0.00 0 3730.76 ± 362% 0 9382.32 ± 552% 0 0.00 build-pkg.time.involuntary_context_switches 0.00 0 7.12 ± 365% 0 6.78 ± 379% 0 0.00 build-pkg.time.major_page_faults 0.00 0 35358.13 ± 194% 0 42519.61 ± 208% 0 0.00 build-pkg.time.maximum_resident_set_size 0.00 0 56001.11 ± 360% 0 160421.51 ± 553% 0 0.00 build-pkg.time.minor_page_faults 0.00 0 65536.00 0 65536.00 0 0.00 build-pkg.time.page_size 0.00 0 7.58 ± 755% 0 40.12 ± 903% 0 0.00 build-pkg.time.user_time 0.00 0 16269.71 ± 318% 0 29779.86 ± 394% 0 0.00 build-pkg.time.voluntary_context_switches 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time.max ... 5579.67 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU95.IPI1:Function_call_interrupts 3152478.83 ± 215% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.IPI0:Rescheduling_interrupts 570272.33 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.IPI1:Function_call_interrupts 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time.max 73066.67 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.file_system_inputs 124.50 ± 217% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.involuntary_context_switches 4.33 ± 172% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.major_page_faults 34816.00 ± 140% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.maximum_resident_set_size 23354.67 ± 202% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.minor_page_faults 65536.00 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.page_size 9.50 ± 68% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.percent_of_cpu_this_job_got 4.27 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.system_time 5.20 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.user_time 172904.17 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.voluntary_context_switches 2651512.00 ± 157% -61.9% 1011304.90 ± 25% -60.9% 1037595.96 ± 25% +582.1% 18086008.00 meminfo.Active 2253381.67 ± 176% -69.2% 694564.75 ± 35% -68.1% 718592.59 ± 35% +668.1% 17308404.00 meminfo.Active(anon) 466201.00 ± 223% -100.0% 0.00 -100.0% 0.00 +2509.3% 12164514.00 meminfo.AnonHugePages 1328043.33 ± 148% -50.7% 655286.32 ± 36% -49.3% 673334.38 ± 34% +966.0% 14157259.00 meminfo.AnonPages 0.00 0 524032.00 0 524032.00 0 520576.00 meminfo.CmaFree ... After: taishan200-2280-2s64p-256g--a101 vm-hi1620-2p8g--roota15 vm-hi1620-2p8g--roota14 taishan200-2280-2s64p-256g--a6 metric -------------------- ------------------------------ ------------------------------ ------------------------------ ------------------------------ %stddev change %stddev change %stddev change %stddev \ | \ | \ | \ 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time 0.00 0 401.60 ± 319% 0 553.75 ± 358% 0 0.00 build-pkg.time.elapsed_time.max 0.00 0 54815.06 ± 124% 0 52113.21 ± 173% 0 0.00 build-pkg.time.file_system_inputs 0.00 0 3730.76 ± 362% 0 9382.32 ± 552% 0 0.00 build-pkg.time.involuntary_context_switches 0.00 0 7.12 ± 365% 0 6.78 ± 379% 0 0.00 build-pkg.time.major_page_faults 0.00 0 35358.13 ± 194% 0 42519.61 ± 208% 0 0.00 build-pkg.time.maximum_resident_set_size 0.00 0 56001.11 ± 360% 0 160421.51 ± 553% 0 0.00 build-pkg.time.minor_page_faults 0.00 0 65536.00 0 65536.00 0 0.00 build-pkg.time.page_size 0.00 0 7.58 ± 755% 0 40.12 ± 903% 0 0.00 build-pkg.time.user_time 0.00 0 16269.71 ± 318% 0 29779.86 ± 394% 0 0.00 build-pkg.time.voluntary_context_switches 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time 0.00 0 0.00 0 0.00 0 5762.00 deploy-cci.time.elapsed_time.max 0.00 0 0.00 0 0.00 0 1.00 deploy-cci.time.exit_status 0.00 0 0.00 0 0.00 0 503552.00 deploy-cci.time.file_system_inputs 0.00 0 0.00 0 0.00 0 3925.00 deploy-cci.time.involuntary_context_switches 0.00 0 0.00 0 0.00 0 28.00 deploy-cci.time.major_page_faults 0.00 0 0.00 0 0.00 0 271168.00 deploy-cci.time.maximum_resident_set_size 0.00 0 0.00 0 0.00 0 2280195.00 deploy-cci.time.minor_page_faults 0.00 0 0.00 0 0.00 0 65536.00 deploy-cci.time.page_size 0.00 0 0.00 0 0.00 0 3.00 deploy-cci.time.percent_of_cpu_this_job_got 0.00 0 0.00 0 0.00 0 17.61 deploy-cci.time.system_time 0.00 0 0.00 0 0.00 0 176.09 deploy-cci.time.user_time 0.00 0 0.00 0 0.00 0 384342.00 deploy-cci.time.voluntary_context_switches 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time 551.49 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.elapsed_time.max 73066.67 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.file_system_inputs 124.50 ± 217% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.involuntary_context_switches 4.33 ± 172% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.major_page_faults 34816.00 ± 140% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.maximum_resident_set_size 23354.67 ± 202% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.minor_page_faults 65536.00 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.page_size 9.50 ± 68% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.percent_of_cpu_this_job_got 4.27 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.system_time 5.20 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.user_time 172904.17 ± 223% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 iso2rootfs.time.voluntary_context_switches 493.41 ± 108% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.boot 47114.17 ± 109% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.idle 11.57 -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 boot-time.kernel_boot 1073.50 ± 217% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.17:GICv3.23.Level.arm-pmu 1003392.50 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.226:ITS-MSI.65536001.Edge.hns3-0000:7d:00.0-TxRx-0 612284.17 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.4:GICv3.26.Level.arch_timer 8151.83 ± 213% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU0.4:GICv3.26.Level.arch_timer 165405.33 ± 222% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU0.IPI0:Rescheduling_interrupts 5886.83 ± 221% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU0.IPI1:Function_call_interrupts 12477.50 ± 214% -100.0% 0.00 -100.0% 0.00 -100.0% 0.00 interrupts.CPU1.4:GICv3.26.Level.arch_timer ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 15 +++++++++------ lib/compare_matrixes.rb | 33 ++++++++++++++++++++++----------- lib/matrix2.rb | 15 +++++++++++---- 3 files changed, 42 insertions(+), 21 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 645f682..3db5169 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -20,8 +20,8 @@ require 'yaml' def compare_matrices_list(argv, common_conditions, options) condition_list = parse_argv(argv, common_conditions) - matrices_list = create_matrices_list(condition_list) - compare_matrixes(matrices_list, options: options) + matrices_list, suites_list = create_matrices_list(condition_list) + compare_matrixes(matrices_list, suites_list, options: options) end def parse_argv(argv, common_conditions) @@ -37,12 +37,15 @@ end def create_matrices_list(conditions) matrices_list = [] + suites_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| query_results = es.multi_field_query(condition) - matrices_list << combine_query_data(query_results) + matrix, suites = combine_query_data(query_results) + matrices_list << matrix + suites_list.concat(suites) end - matrices_list + return matrices_list, suites_list end # ------------------------------------------------------------------------------------------- @@ -58,8 +61,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, options) + groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_list, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 119d42d..c1cee10 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -193,7 +193,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suites_list, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -208,6 +208,7 @@ def compare_matrixes(matrixes_list, matrixes_titles = nil, group_key = nil, opti result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( matrixes_values, + suites_list, matrixes_titles, matrixes_list.size, options[:theme] @@ -382,7 +383,7 @@ end # compare each matrices_list within pre dimension of group matrices # input: group matrices # output: pre compare result of each group -def compare_group_matrices(group_matrices, options) +def compare_group_matrices(group_matrices, suites_list, options) result_str = '' group_matrices.each do |k, v| matrices_list = [] @@ -392,9 +393,9 @@ def compare_group_matrices(group_matrices, options) matrices_list << matrix end if options[:no_print] - result_str += compare_matrixes(matrices_list, matrices_titles, k, options: options) + result_str += compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) else - print compare_matrixes(matrices_list, matrices_titles, k, options: options) + print compare_matrixes(matrices_list, suites_list.shift, matrices_titles, k, options: options) end end result_str @@ -807,26 +808,36 @@ def get_theme(matrixes_values, matrixes_titles, theme) return THEMES[:none] end -def get_all_result_str(matrixes_values, matrixes_titles, matrixes_number, theme) +def get_all_result_str(matrixes_values, suites_list, matrixes_titles, matrixes_number, theme) matrixes_titles ||= matrixes_number.times.to_a.map(&:to_s) theme = get_theme(matrixes_values, matrixes_titles, theme) return '' unless theme - failure_str = get_result_str(matrixes_values[false].sort, matrixes_titles, false, theme) - success_str = get_result_str(matrixes_values[true].sort, matrixes_titles, true, theme) + failure_str = get_result_str(matrixes_values[false].sort, suites_list, matrixes_titles, false, theme) + success_str = get_result_str(matrixes_values[true].sort, suites_list, matrixes_titles, true, theme) failure_str + success_str end -def get_result_str(values, matrixes_titles, success, theme) +def get_result_str(values, suites_list, matrixes_titles, success, theme) return '' if values.empty? + suites_list.uniq! result_str = "\n\n\n" common_title, compare_title = get_title_name(success) result_str += get_header(matrixes_titles, success, common_title, compare_title) + suite_str = '' + common_str = '' values.each do |field, matrixes| - result_str += get_values_str(matrixes, success, theme) - result_str += get_field_str(field) - result_str += "\n" + row = get_values_str(matrixes, success, theme) + row += get_field_str(field) + "\n" + field_start_with_suite = suites_list.any? { |suite| field.start_with?(suite) } + if field_start_with_suite + suite_str += row + else + common_str += row + end end + + result_str += suite_str + common_str result_str end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 26f07fa..479d280 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -76,7 +76,9 @@ end # } def create_matrix(job_list) matrix = {} + suites = [] job_list.each do |job| + suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats @@ -89,7 +91,7 @@ def create_matrix(job_list) matrix.each_value do |value| samples_fill_missing_zeros(value, col_size) end - matrix + return matrix, suites end # input: query results from es_query @@ -110,15 +112,20 @@ end # ... # } def combine_group_query_data(query_data, dims) + suites_list = [] job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| + suite_list = [] value.each do |dimension_key, jobs| - groups[group_key][dimension_key] = create_matrix(jobs) + groups[group_key][dimension_key], suites = create_matrix(jobs) + suite_list.concat(suites) end groups.delete(group_key) if value.size < 2 + suites_list << suite_list unless value.size < 2 end - groups + + return groups, suites_list end # input: @@ -139,7 +146,7 @@ def combine_group_jobs_list(query_data, groups_params, dimensions, metrics) groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) groups.each do |group_key, dims| dims.each do |dim_key, jobs| - groups[group_key][dim_key] = create_matrix(jobs) + groups[group_key][dim_key], = create_matrix(jobs) end end -- 2.23.0

3 2

[PATCH v5 compass-ci 12/12] mail-robot: start file
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/start | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..eab1f12 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,38 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative "#{ENV['CCI_SRC']}/container/defconfig" + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) +defaults['CCI_SRC'] = '/c/compass-ci' +defaults['MAILDIR'] = '/srv/cci/Maildir/.compass-ci' + +env = docker_env(defaults) + +cmd = %w[ + docker run + --restart=always + --name=mail-robot + -d +] + env + %W[ + -v /c/upstream-repos:/c/upstream-repos:ro + -v #{defaults['MAILDIR']}:#{defaults['MAILDIR']}:rw + -v #{ENV['CCI_SRC']}:/c/compass-ci:rw + -w /c/compass-ci/container/mail-robot + mail-robot +] + +cmd += ['ruby', 'run-mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v5 compass-ci 11/12] mail-robot: build file
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v5 compass-ci 10/12] mail-robot: Dockerfile
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..76dc94d --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER Luan Shengde <shdluan(a)163.com> + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH v5 compass-ci 09/12] mail-robot: documentation for service
by Luan Shengde 11 Nov '20

11 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..2eb0dd1 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. assign-account + init my_info + - my_email + - my_name + send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

1 0