- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests 1/2] feat(iso2rootfs): modify the rootfs dir layout and rootfs dir name
by Yu Chuan 19 Oct '20

19 Oct '20

[Why] 1. Modify the dir structure of dailybuild rootfs dir. 2. Modify the dir name of dailybuild rootfs dir. [How] before: "{os}/{os_arch}/debug-versions/dailybuild/{iso_version}-%Y%m%d%H%M%S" after: "{os}/{os_arch}/{iso_version}-%Y-%m-%d-%H-%M-%S" Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 524ab6695f93..4782378424b5 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,6 @@ get_daily_iso_checksum() mount_rootfs() { - DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +102,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-newest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +254,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +309,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -366,7 +365,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ + ln -s "$(basename ${ROOTFS_DES_DIR})" \ "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 2

[PATCH lkp-tests] tests: add test-tools
by Li Ping 19 Oct '20

19 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- tests/test-tools | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 tests/test-tools diff --git a/tests/test-tools b/tests/test-tools new file mode 100755 index 00000000..cee882aa --- /dev/null +++ b/tests/test-tools @@ -0,0 +1,37 @@ +#!/bin/bash +# - testsuite +# - case + +cd $BENCHMARK_ROOT/test-tools/mugen || exit +: "${testsuite:=smoke-testing}" +: "${case:=oe_test_dd_001}" + +set_env() +{ + bash runoet.sh -c + source ~/.bash_profile +} + +test_one_case() +{ + + bash runoet.sh -f $testsuite -r $case -x +} + +test_all_cases() +{ + bash runoet.sh -a +} + +run() +{ + set_env + if [ "${testsuite}" == "all" ] + then + test_all_cases + else + test_one_case + fi +} + +run -- 2.23.0

4 3

[PATCH v7 compass-ci 6/6] container/mail-robot: mail-robot.md
by Luan Shengde 19 Oct '20

19 Oct '20

guidance file for mail-robot Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..d5eeaa9 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,45 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +monitor received mails + check if the mail is for applying uuid/account + +## steps overview + +1. monitor mailbox dir + - check for new email file + +2. check email + - check email subject + - return if unmatched subject + - call answerback-email + - check email available + - check base_url for the oss_url in upstream-repos list + if not, it will raise "url not in upstream-repos list" error + - check if the email has commit for the url + - if gitee, git clone the repo and check + if non-gitee, curl url to check + - if not, raise "no commit found" error + - check pub_key + - check if there is a pub_key in the email + if not, it will raise 'no pub_key' error + - apply ssh account + - apply account uuid + - write account info to es + - send uuid to user + - mv mail file to drafts + +3. continue the monitor + - after 1 and 2, continue to monitor the mailbox + +4. service log + - use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot + +5. usage: + - mail subject: apply account + - mail body: commit url + - mail attachment: id_rsa.pub -- 2.23.0

1 0

[PATCH v7 compass-ci 3/6] container/mail-robot: build file
by Luan Shengde 19 Oct '20

19 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] container/send-mail: update start
by Luan Shengde 19 Oct '20

19 Oct '20

add setup default variables update heredoc Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/start | 89 +++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 27 deletions(-) diff --git a/container/send-mail/start b/container/send-mail/start index 80be86a..068fc1e 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -1,29 +1,68 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -. $CCI_SRC/container/defconfig.sh +require 'set' +require_relative '../defconfig' -docker_rm send-mail +docker_rm 'send-mail' -cmd=( - docker run - --restart=always - --name=send-mail - -u nobody - -d - -p 11311:11311 - -v /etc/localtime:/etc/localtime:ro - send-mail -) +names = Set.new %w[ + EMAIL_PSWD + SEND_MAIL_PORT + SMTP_SERVER_ADDR + SMTP_SERVER_PORT + EMAIL_DOMAIN + EMAIL_ADDR + DEFAULT_EMAIL +] -"${cmd[@]}" +defaults = relevant_defaults(names) -cat<<EOF +EMAIL_PSWD = defaults['EMAIL_PSWD'] +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] +SMTP_SERVER_ADDR = defaults['SMTP_SERVER_ADDR'] +SMTP_SERVER_PORT = defaults['SMTP_SERVER_PORT'] +EMAIL_DOMAIN = defaults['EMAIL_DOMAIN'] +EMAIL_ADDR = defaults['EMAIL_ADDR'] +DEFAULT_EMAIL = defaults['DEFAULT_EMAIL'] + +cmd = %W[ + docker run + --restart=always + --name=send-mail + -u nobody + -d + -e EMAIL_PSWD=#{EMAIL_PSWD} + -e SEND_MAIL_PORT=#{SEND_MAIL_PORT} + -e SMTP_SERVER_ADDR=#{SMTP_SERVER_ADDR} + -e SMTP_SERVER_PORT=#{SMTP_SERVER_PORT} + -e EMAIL_DOMAIN=#{EMAIL_DOMAIN} + -e EMAIL_ADDR=#{EMAIL_ADDR} + -e DEFAULT_EMAIL=#{DEFAULT_EMAIL} + -p #{SEND_MAIL_PORT}:#{SEND_MAIL_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-driver json-file + send-mail +] + +cmd += ['bash', '-c', '/usr/local/bin/mail-post.rb'] + +system(*cmd) + +puts <<HEREDOC port: 11311 + +default from email: + when sending crystal.ci email. + team(a)crystal.ci if no from email appiontted + when sending to WAN email + compass-ci(a)qq.com in all cases + --- data='{ -"from": "email_from_addr" # default: team(a)crystal.ci +"from": "email_from_addr" "subject": "email subject", "to": "email_to_addr", "body": "email message" @@ -32,19 +71,17 @@ data='{ or data=" -from: email_from_addr # default: team(a)crystal.ci +from: email_from_addr subject: email subject to: email_to_addr body: email message" -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_yaml' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_yaml' -d "$data" +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_yaml' -d "$data" --- data=" -From: email_from_addr # default: team(a)crystal.ci +From: email_from_addr To: email_to_addr Subject: email_subject @@ -53,8 +90,6 @@ mail_msg_line2 ... " -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_text' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_text' -d "$data" -EOF +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_text' -d "$data" +HEREDOC -- 2.23.0

2 1

[PATCH v2 compass-ci 2/4] container/send-mail: send-mail.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add setup smtp for qq mail server enable send mail via smtp.qq.com Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/send-mail.rb | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index f459e78..844cad5 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -5,15 +5,29 @@ require 'mail' -mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp +# mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp # setup smtp config -smtp = { - address: mail_server, - enable_starttls_auto: false -} +def intranet_smtp(mail_server) + smtp = { + address: mail_server, + enable_starttls_auto: false + } + Mail.defaults { delivery_method :smtp, smtp } +end -Mail.defaults { delivery_method :smtp, smtp } +def internet_smtp + smtp = { + address: (ENV['SMTP_SERVER_ADDR']).to_s, + port: (ENV['SMPT_SERVER_PORT']), + domain: (ENV['EMAIL_DOMAIN']).to_s, + user_name: (ENV['EMAIL_ADDR']).to_s, + password: (ENV['EMAIL_PSWD']).to_s, + openssl_verify_mode: 'none', + enable_starttls_auto: true + } + Mail.defaults { delivery_method :smtp, smtp } +end # send mail def send_mail(mail_info) -- 2.23.0

2 1

[PATCH compass-ci 1/2] user-client/maintain: test os rootfs in z9 tbox
by Liu Yinsi 19 Oct '20

19 Oct '20

1. no needed my-qemu.sh after submit job in z9 tbox, remove. 2. monitor job_state boot/extract_finished to get job runtime. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 124 +++++++++--------- 1 file changed, 65 insertions(+), 59 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 21e1428..2c3fc9f 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -1,62 +1,68 @@ -#!/bin/bash -e +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ +# # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -# os test rounds -test_times=1 - -test_logs=walk-test.report -test_yaml=iperf-walk-os.yaml -test_os=( - 'openeuler aarch64 20' - 'openeuler aarch64 20.03' - 'centos aarch64 7.6' - 'centos aarch64 7.8' - 'centos aarch64 8.1.1911' +require "#{ENV['LKP_SRC']}/lib/monitor" + +TEST_OS = [ + 'openeuler aarch64 20.03', + 'centos aarch64 7.6.1810', + 'centos aarch64 7.8.2003', + 'centos aarch64 8.1.1911', 'debian aarch64 sid' -) - -trap 'echo User Cancelled! && exit' SIGINT - -sec2date() { - date -d@$1 +"%Y%m%d_%H:%M:%S" -} - -logging() { - if [ 'new' == "$1" ] ; then - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" ${@:2} > $test_logs - else - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" $@ >> $test_logs - fi -} - -create_test_yaml() { - local hwarch os os_arch os_ver cur_os - hwarch=$(arch) - cur_os="$@" - os=$(echo $cur_os |cut -d' ' -f1) - os_arch=$(echo $cur_os |cut -d' ' -f2) - os_ver=$(echo $cur_os |cut -d' ' -f3) - sed "s/USER/$USER/" $test_yaml > test.yaml - sed -i "s/OS_ARCH/$os_arch/" test.yaml - sed -i "s/OS_VER/$os_ver/" test.yaml - sed -i "s/OS/$os/" test.yaml - echo "$hwarch $os/$os_arch/$os_ver" -} - -job_test() { - local job_id bef aft - job_id=$(submit test.yaml |awk '{print $NF}') - bef=$(date +%s) - ./my-qemu.sh - aft=$(date +%s) - logging "$1" "$2" "$3" "$(sec2date $bef)" "$(sec2date $aft)" "$((aft - bef))" "$job_id" -} - -logging new "Index" "hwArch" "OS" "Begin" "End" "Cost/sec" "Job_ID" -for round in $(seq $test_times) -do - for idx in "${!test_os[@]}" - do - job_test $round $(create_test_yaml "${test_os[$idx]}") - done -done +] + +MOUNT_TYPE = ['cifs', 'initramfs'] + +def logging(job_id, os_params, start_time, end_time, cost_time) + line = format("%-12s %-36s %-28s %-28s %-8s\n", job_id, os_params, start_time, end_time, cost_time) + f = File.new('walk-test.report', 'a') + f.write(line) + f.close +end + +def monitor_job(job_id, job_state) + res = monitor({'job_id' => job_id, 'job_state' => job_state}, {'stop' => true}, '600s') + if res == 0 + Time.now + end +end + +def monitor(query, actions, timeout) + monitor = Monitor.new + monitor.overrides = query + monitor.action = actions + monitor.run(timeout) +end + +def submit_job(os_array) + os = os_array[0] + os_arch = os_array[1] + os_version = os_array[2] + os_mount = os_array[3] + message = %x(submit -s 'os: #{os}' -s 'os_arch: #{os_arch}' -s 'os_version: #{os_version}' -s 'os_mount: #{os_mount}' iperf-walk-os.yaml) + puts message + + job_id = message.split('=')[1].chomp + os_params = os_array.join('/') + start_time = monitor_job(job_id, 'boot') + end_time = monitor_job(job_id, 'extract_finished') + cost_time = (end_time - start_time).floor + + logging(job_id, os_params, start_time, end_time, cost_time) +end + +def get_os_array(os_mount) + TEST_OS.each do | item | + os_array = item.split.push(os_mount) + + submit_job(os_array) + end +end + +system "walk-test.report" + +logging("Job_ID", "OS", "Begin", "End", "Cost/sec") + +MOUNT_TYPE.each {| os_mount | get_os_array(os_mount)} -- 2.23.0

3 4

[PATCH compass-ci] fix: taskqueue: simplify logs
by Tong Qunfeng 19 Oct '20

19 Oct '20

[why] current taskqueue show 4 lines for 1 debug message - empty line for split: - end user request ip:port, request body: 172.17.0.1:48956 {"domain":"..."} - response body: <-- - kemal frame log: 2020-10... 200 POST /add?queue=... 710.15µs cut it two 2 line, and duplicate information - kemal frame log has time in {2020-10...} and span {710.15us} Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/taskqueue/taskqueue.cr | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/src/taskqueue/taskqueue.cr b/src/taskqueue/taskqueue.cr index c5c202d..98c5623 100644 --- a/src/taskqueue/taskqueue.cr +++ b/src/taskqueue/taskqueue.cr @@ -18,16 +18,22 @@ class TaskQueue @@rate_limiter.bucket(:l1pms, 1_u32, 0.001.seconds) @@rate_limiter.bucket(:l100pms, 1_u32, 0.00001.seconds) + # logs example: + # from: {172.17.0.1:6952} body {"domain":"compass-ci"} <-- ack: {"id":"z9.134780"} + # | from: {172.17.0.1:6952} <-- ack: {} + # 2020-10-19 03:25:01 UTC 201 PUT /consume?queue=sched%2Fdc-2g%2Fidle 356.51µs + # + # "from" is puts by this debug_message + # "2020-10..." auto puts by kemal frame, time in {2020-10-19 03:25:01 UTC} and span {356.51us} def debug_message(env, response, time_in) - puts("\n") + logs = "from: {#{env.request.remote_address}}" - from_message = "#{time_in} --> #{env.request.remote_address}" - if env.request.body != nil - from_message += " #{env.request.body}" - end - puts(from_message) + logs += " body: #{env.request.body}" if env.request.body + + logs += " <-- ack: " + logs += response ? "#{response}" : "{}" - puts("#{Time.utc} <-- #{response}") + puts(logs) end def run -- 2.23.0

1 0

[PATCH compass-ci] add exception handing for docker.rb
by Xiao Shenwei 19 Oct '20

19 Oct '20

[why] when run multi-docker job on physical machine, run 50 client to request job, if server down or error like Connection refused. these client will exit. when the server recovers, need to run these client again. [how] add exception handing to ensure all client does not exit --- providers/docker/docker.rb | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index c241d3c..2239930 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -88,6 +88,19 @@ def main(hostname) run(hostname, load_path, hash) end +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.message + sleep 10 + ensure + sleep 5 + end + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -99,10 +112,7 @@ def multi_docker(hostname, nr_container) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop do - main("#{hostname}-#{i}") - sleep 5 - end + loop_main("#{hostname}-#{i}") end pids << pid end -- 2.23.0

3 4

[PATCH v2 compass-ci 1/5] container/send-internet-mail: mail-post.rb
by Luan Shengde 18 Oct '20

18 Oct '20

send mail interface: get send mail request build email info from the request data call send-internet-mail to send email with email info Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/mail-post.rb | 46 +++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100755 container/send-internet-mail/mail-post.rb diff --git a/container/send-internet-mail/mail-post.rb b/container/send-internet-mail/mail-post.rb new file mode 100755 index 0000000..3b33c7f --- /dev/null +++ b/container/send-internet-mail/mail-post.rb @@ -0,0 +1,46 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'yaml' +require 'open3' +require_relative 'send-internet-mail.rb' + +set :bind, '0.0.0.0' +set :port, 11312 + +post '/send_mail_yaml' do + data = YAML.safe_load request.body.read + raise TypeError, data, 'request data type error' unless data.class.eql? Hash + + mail_info = { + 'references' => data['references'] || '', + 'subject' => data['subject'] || '', + 'to' => data['to'] || '', + 'body' => data['body'] || '' + } + check_send_mail(mail_info) if data +end + +post '/send_mail_text' do + data = Mail.read_from_string(request.body.read) + + mail_info = { + 'references' => data.references || '', + 'subject' => data.subject || '', + 'to' => data.to || '', + 'body' => data.body.decoded || '' + } + check_send_mail(mail_info) if data +end + +def check_send_mail(mail_info) + raise 'No subject.' if mail_info['subject'].empty? + raise 'No email address.' if mail_info['to'].empty? + raise 'No email body.' if mail_info['body'].empty? + + send_mail(mail_info) +end -- 2.23.0

3 5