October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 4/4] container/send-mail: update start
by Luan Shengde 19 Oct '20

19 Oct '20

add setup default variables update heredoc Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/start | 89 +++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 27 deletions(-) diff --git a/container/send-mail/start b/container/send-mail/start index 80be86a..068fc1e 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -1,29 +1,68 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -. $CCI_SRC/container/defconfig.sh +require 'set' +require_relative '../defconfig' -docker_rm send-mail +docker_rm 'send-mail' -cmd=( - docker run - --restart=always - --name=send-mail - -u nobody - -d - -p 11311:11311 - -v /etc/localtime:/etc/localtime:ro - send-mail -) +names = Set.new %w[ + EMAIL_PSWD + SEND_MAIL_PORT + SMTP_SERVER_ADDR + SMTP_SERVER_PORT + EMAIL_DOMAIN + EMAIL_ADDR + DEFAULT_EMAIL +] -"${cmd[@]}" +defaults = relevant_defaults(names) -cat<<EOF +EMAIL_PSWD = defaults['EMAIL_PSWD'] +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] +SMTP_SERVER_ADDR = defaults['SMTP_SERVER_ADDR'] +SMTP_SERVER_PORT = defaults['SMTP_SERVER_PORT'] +EMAIL_DOMAIN = defaults['EMAIL_DOMAIN'] +EMAIL_ADDR = defaults['EMAIL_ADDR'] +DEFAULT_EMAIL = defaults['DEFAULT_EMAIL'] + +cmd = %W[ + docker run + --restart=always + --name=send-mail + -u nobody + -d + -e EMAIL_PSWD=#{EMAIL_PSWD} + -e SEND_MAIL_PORT=#{SEND_MAIL_PORT} + -e SMTP_SERVER_ADDR=#{SMTP_SERVER_ADDR} + -e SMTP_SERVER_PORT=#{SMTP_SERVER_PORT} + -e EMAIL_DOMAIN=#{EMAIL_DOMAIN} + -e EMAIL_ADDR=#{EMAIL_ADDR} + -e DEFAULT_EMAIL=#{DEFAULT_EMAIL} + -p #{SEND_MAIL_PORT}:#{SEND_MAIL_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-driver json-file + send-mail +] + +cmd += ['bash', '-c', '/usr/local/bin/mail-post.rb'] + +system(*cmd) + +puts <<HEREDOC port: 11311 + +default from email: + when sending crystal.ci email. + team(a)crystal.ci if no from email appiontted + when sending to WAN email + compass-ci(a)qq.com in all cases + --- data='{ -"from": "email_from_addr" # default: team(a)crystal.ci +"from": "email_from_addr" "subject": "email subject", "to": "email_to_addr", "body": "email message" @@ -32,19 +71,17 @@ data='{ or data=" -from: email_from_addr # default: team(a)crystal.ci +from: email_from_addr subject: email subject to: email_to_addr body: email message" -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_yaml' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_yaml' -d "$data" +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_yaml' -d "$data" --- data=" -From: email_from_addr # default: team(a)crystal.ci +From: email_from_addr To: email_to_addr Subject: email_subject @@ -53,8 +90,6 @@ mail_msg_line2 ... " -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_text' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_text' -d "$data" -EOF +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_text' -d "$data" +HEREDOC -- 2.23.0

2 1

[PATCH v2 compass-ci 2/4] container/send-mail: send-mail.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add setup smtp for qq mail server enable send mail via smtp.qq.com Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/send-mail.rb | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index f459e78..844cad5 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -5,15 +5,29 @@ require 'mail' -mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp +# mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp # setup smtp config -smtp = { - address: mail_server, - enable_starttls_auto: false -} +def intranet_smtp(mail_server) + smtp = { + address: mail_server, + enable_starttls_auto: false + } + Mail.defaults { delivery_method :smtp, smtp } +end -Mail.defaults { delivery_method :smtp, smtp } +def internet_smtp + smtp = { + address: (ENV['SMTP_SERVER_ADDR']).to_s, + port: (ENV['SMPT_SERVER_PORT']), + domain: (ENV['EMAIL_DOMAIN']).to_s, + user_name: (ENV['EMAIL_ADDR']).to_s, + password: (ENV['EMAIL_PSWD']).to_s, + openssl_verify_mode: 'none', + enable_starttls_auto: true + } + Mail.defaults { delivery_method :smtp, smtp } +end # send mail def send_mail(mail_info) -- 2.23.0

2 1

[PATCH compass-ci 1/2] user-client/maintain: test os rootfs in z9 tbox
by Liu Yinsi 19 Oct '20

19 Oct '20

1. no needed my-qemu.sh after submit job in z9 tbox, remove. 2. monitor job_state boot/extract_finished to get job runtime. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 124 +++++++++--------- 1 file changed, 65 insertions(+), 59 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 21e1428..2c3fc9f 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -1,62 +1,68 @@ -#!/bin/bash -e +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ +# # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -# os test rounds -test_times=1 - -test_logs=walk-test.report -test_yaml=iperf-walk-os.yaml -test_os=( - 'openeuler aarch64 20' - 'openeuler aarch64 20.03' - 'centos aarch64 7.6' - 'centos aarch64 7.8' - 'centos aarch64 8.1.1911' +require "#{ENV['LKP_SRC']}/lib/monitor" + +TEST_OS = [ + 'openeuler aarch64 20.03', + 'centos aarch64 7.6.1810', + 'centos aarch64 7.8.2003', + 'centos aarch64 8.1.1911', 'debian aarch64 sid' -) - -trap 'echo User Cancelled! && exit' SIGINT - -sec2date() { - date -d@$1 +"%Y%m%d_%H:%M:%S" -} - -logging() { - if [ 'new' == "$1" ] ; then - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" ${@:2} > $test_logs - else - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" $@ >> $test_logs - fi -} - -create_test_yaml() { - local hwarch os os_arch os_ver cur_os - hwarch=$(arch) - cur_os="$@" - os=$(echo $cur_os |cut -d' ' -f1) - os_arch=$(echo $cur_os |cut -d' ' -f2) - os_ver=$(echo $cur_os |cut -d' ' -f3) - sed "s/USER/$USER/" $test_yaml > test.yaml - sed -i "s/OS_ARCH/$os_arch/" test.yaml - sed -i "s/OS_VER/$os_ver/" test.yaml - sed -i "s/OS/$os/" test.yaml - echo "$hwarch $os/$os_arch/$os_ver" -} - -job_test() { - local job_id bef aft - job_id=$(submit test.yaml |awk '{print $NF}') - bef=$(date +%s) - ./my-qemu.sh - aft=$(date +%s) - logging "$1" "$2" "$3" "$(sec2date $bef)" "$(sec2date $aft)" "$((aft - bef))" "$job_id" -} - -logging new "Index" "hwArch" "OS" "Begin" "End" "Cost/sec" "Job_ID" -for round in $(seq $test_times) -do - for idx in "${!test_os[@]}" - do - job_test $round $(create_test_yaml "${test_os[$idx]}") - done -done +] + +MOUNT_TYPE = ['cifs', 'initramfs'] + +def logging(job_id, os_params, start_time, end_time, cost_time) + line = format("%-12s %-36s %-28s %-28s %-8s\n", job_id, os_params, start_time, end_time, cost_time) + f = File.new('walk-test.report', 'a') + f.write(line) + f.close +end + +def monitor_job(job_id, job_state) + res = monitor({'job_id' => job_id, 'job_state' => job_state}, {'stop' => true}, '600s') + if res == 0 + Time.now + end +end + +def monitor(query, actions, timeout) + monitor = Monitor.new + monitor.overrides = query + monitor.action = actions + monitor.run(timeout) +end + +def submit_job(os_array) + os = os_array[0] + os_arch = os_array[1] + os_version = os_array[2] + os_mount = os_array[3] + message = %x(submit -s 'os: #{os}' -s 'os_arch: #{os_arch}' -s 'os_version: #{os_version}' -s 'os_mount: #{os_mount}' iperf-walk-os.yaml) + puts message + + job_id = message.split('=')[1].chomp + os_params = os_array.join('/') + start_time = monitor_job(job_id, 'boot') + end_time = monitor_job(job_id, 'extract_finished') + cost_time = (end_time - start_time).floor + + logging(job_id, os_params, start_time, end_time, cost_time) +end + +def get_os_array(os_mount) + TEST_OS.each do | item | + os_array = item.split.push(os_mount) + + submit_job(os_array) + end +end + +system "walk-test.report" + +logging("Job_ID", "OS", "Begin", "End", "Cost/sec") + +MOUNT_TYPE.each {| os_mount | get_os_array(os_mount)} -- 2.23.0

3 4

[PATCH compass-ci] fix: taskqueue: simplify logs
by Tong Qunfeng 19 Oct '20

19 Oct '20

[why] current taskqueue show 4 lines for 1 debug message - empty line for split: - end user request ip:port, request body: 172.17.0.1:48956 {"domain":"..."} - response body: <-- - kemal frame log: 2020-10... 200 POST /add?queue=... 710.15µs cut it two 2 line, and duplicate information - kemal frame log has time in {2020-10...} and span {710.15us} Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/taskqueue/taskqueue.cr | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/src/taskqueue/taskqueue.cr b/src/taskqueue/taskqueue.cr index c5c202d..98c5623 100644 --- a/src/taskqueue/taskqueue.cr +++ b/src/taskqueue/taskqueue.cr @@ -18,16 +18,22 @@ class TaskQueue @@rate_limiter.bucket(:l1pms, 1_u32, 0.001.seconds) @@rate_limiter.bucket(:l100pms, 1_u32, 0.00001.seconds) + # logs example: + # from: {172.17.0.1:6952} body {"domain":"compass-ci"} <-- ack: {"id":"z9.134780"} + # | from: {172.17.0.1:6952} <-- ack: {} + # 2020-10-19 03:25:01 UTC 201 PUT /consume?queue=sched%2Fdc-2g%2Fidle 356.51µs + # + # "from" is puts by this debug_message + # "2020-10..." auto puts by kemal frame, time in {2020-10-19 03:25:01 UTC} and span {356.51us} def debug_message(env, response, time_in) - puts("\n") + logs = "from: {#{env.request.remote_address}}" - from_message = "#{time_in} --> #{env.request.remote_address}" - if env.request.body != nil - from_message += " #{env.request.body}" - end - puts(from_message) + logs += " body: #{env.request.body}" if env.request.body + + logs += " <-- ack: " + logs += response ? "#{response}" : "{}" - puts("#{Time.utc} <-- #{response}") + puts(logs) end def run -- 2.23.0

1 0

[PATCH compass-ci] add exception handing for docker.rb
by Xiao Shenwei 19 Oct '20

19 Oct '20

[why] when run multi-docker job on physical machine, run 50 client to request job, if server down or error like Connection refused. these client will exit. when the server recovers, need to run these client again. [how] add exception handing to ensure all client does not exit --- providers/docker/docker.rb | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index c241d3c..2239930 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -88,6 +88,19 @@ def main(hostname) run(hostname, load_path, hash) end +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.message + sleep 10 + ensure + sleep 5 + end + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -99,10 +112,7 @@ def multi_docker(hostname, nr_container) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop do - main("#{hostname}-#{i}") - sleep 5 - end + loop_main("#{hostname}-#{i}") end pids << pid end -- 2.23.0

3 4

[PATCH v2 compass-ci 1/5] container/send-internet-mail: mail-post.rb
by Luan Shengde 18 Oct '20

18 Oct '20

send mail interface: get send mail request build email info from the request data call send-internet-mail to send email with email info Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/mail-post.rb | 46 +++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100755 container/send-internet-mail/mail-post.rb diff --git a/container/send-internet-mail/mail-post.rb b/container/send-internet-mail/mail-post.rb new file mode 100755 index 0000000..3b33c7f --- /dev/null +++ b/container/send-internet-mail/mail-post.rb @@ -0,0 +1,46 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'yaml' +require 'open3' +require_relative 'send-internet-mail.rb' + +set :bind, '0.0.0.0' +set :port, 11312 + +post '/send_mail_yaml' do + data = YAML.safe_load request.body.read + raise TypeError, data, 'request data type error' unless data.class.eql? Hash + + mail_info = { + 'references' => data['references'] || '', + 'subject' => data['subject'] || '', + 'to' => data['to'] || '', + 'body' => data['body'] || '' + } + check_send_mail(mail_info) if data +end + +post '/send_mail_text' do + data = Mail.read_from_string(request.body.read) + + mail_info = { + 'references' => data.references || '', + 'subject' => data.subject || '', + 'to' => data.to || '', + 'body' => data.body.decoded || '' + } + check_send_mail(mail_info) if data +end + +def check_send_mail(mail_info) + raise 'No subject.' if mail_info['subject'].empty? + raise 'No email address.' if mail_info['to'].empty? + raise 'No email body.' if mail_info['body'].empty? + + send_mail(mail_info) +end -- 2.23.0

3 5

[PATCH lkp-tests] jobs/fio-basic-openeuler.yaml: add fio test args
by Wei Jihui 17 Oct '20

17 Oct '20

1. add disk args, it should be tested on physical machine 2. raise nr_task to 24, for numjobs is 24. 3. add direct args Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/fio-basic-openeuler.yaml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/jobs/fio-basic-openeuler.yaml b/jobs/fio-basic-openeuler.yaml index 3fcdaac8..67d4e998 100644 --- a/jobs/fio-basic-openeuler.yaml +++ b/jobs/fio-basic-openeuler.yaml @@ -3,12 +3,18 @@ suite: fio-basic category: benchmark runtime: 60 +disk: +- 1HDD +fs: +- ext4 +nr_task: 24 + fio-setup-basic: ioengine: libaio iodepth: 1024 - nr_task: 2 test_size: 1G rwmixread: 70 + direct: 1 rw: - write - read -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: load yaml config in advance
by Wei Jihui 17 Oct '20

17 Oct '20

[why] for commit c725e0f022b7: def each_jobs(&block) each_job_init - load_hosts_config job = deepcopy(@job) @job2 = {} load_defaults + load_hosts_config each_job_init each_job(&block) @jobs.each do |hash| 1. it exec load_defaults first, so it not load hosts file when start exec load_defaults. load_defaults call expand_params, expand_params maybe expand host params, but do not have load hosts file, load_one_defaults can load host file, so exec load_one_defaults in advance. 2. load_one_defaults(file, @job) use @job is because load_hosts_file is add host config to @job, so it should be executed before "job=deepcopy(@job)" Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..27c4bd93 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -400,6 +400,11 @@ class Job @file_loaded ||= {} end + Dir.glob(["/etc/compass-ci/defaults/*.yaml", + "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| + load_one_defaults(file, @job) + end + i = include_files job = deepcopy(@job) revise_hash(job, deepcopy(@job2), true) @@ -418,11 +423,6 @@ class Job end @jobx = nil - Dir.glob(["/etc/compass-ci/defaults/*.yaml", - "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| - load_one_defaults(file, job) - end - for_each_in(job, i.keys) do |_pk, _h, k, v| job['___'] = v -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: add define files when program is cci-depends
by Wei Jihui 17 Oct '20

17 Oct '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..9a08dc30 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -617,9 +617,9 @@ class Job program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| - # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg + # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files - if %w(makepkg makepkg-deps pack-deps cci-makepkg).include?(program) + if %w(makepkg makepkg-deps pack-deps cci-makepkg cci-depends).include?(program) program = programs_params_hash[program]["benchmark"] end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: merge overrides to defaults
by Wei Jihui 17 Oct '20

17 Oct '20

[why] submit job.yaml testbox=vm-2p8g it should use vm-2p8g to load host file, not use defaults's testbox [slove] merge @overrides to @defaults, then use @defaults to load host file Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..4e398a5a 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -382,7 +382,8 @@ class Job if defaults.is_a?(Hash) && !defaults.empty? @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) - load_hosts_config if check_load_hosts_config(defaults) + @defaults.merge!(@overrides) + load_hosts_config if check_load_hosts_config(@defaults) end @file_loaded[file] = true true -- 2.23.0

1 0