- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] service/scheduler: add ssh port to job content
by Wu Zhende 19 Oct '20

19 Oct '20

[Why] Network users need to use sshd to login testbox. So testbox should set sshd. And testbox get sshd info from job content. Scheduler needs to add ssh info to job content Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/start | 4 ++++ src/lib/job.cr | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 81e7139..1183cde 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -24,10 +24,14 @@ names = Set.new %w[ GIT_SERVER FLUENTD_SERVER_HOST FLUENTD_SERVER_PORT + SSH_PORT_BASE + SSH_PORT_LEN lab ] defaults = relevant_defaults(names) +defaults["SSH_PORT_BASE"] = 50000 unless defaults["SSH_PORT_BASE"] +defaults["SSH_PORT_LEN"]= 2000 unless defaults["SSH_PORT_LEN"] env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/lib/job.cr b/src/lib/job.cr index 20afa65..fcc1711 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -139,6 +139,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() + set_ssh_port() end private def append_init_field @@ -168,6 +169,11 @@ class Job end end + private def set_ssh_port + self["ssh_port_base"] = ENV["SSH_PORT_BASE"] + self["ssh_port_len"] = ENV["SSH_PORT_LEN"] + end + private def set_os_dir self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end -- 2.23.0

2 1

[PATCH v7 compass-ci 4/6] container/mail-robot: Dockerfile
by Luan Shengde 19 Oct '20

19 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..57307bf --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER shdluan(a)163.com + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport && \ + ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ + echo 'Asia/Shanghai' > /etc/timezone + +COPY mail-robot.rb /usr/local/bin/ + +COPY answerback-email.rb /usr/local/bin/ + +WORKDIR /usr/local/bin/ -- 2.23.0

2 1

[PATCH compass-ci] container/assign-account: answerback-email.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add parameters for answerback-email -e,--email email_addr: add email address -s,--ssh-pubkey pub_key: add ssh pub_key -f,--raw-email mail_file: add email_adress[,pub_key] via email file Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 162 ++++++++----------- 1 file changed, 70 insertions(+), 92 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index b5283ba..4a86a87 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -3,46 +3,18 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -=begin - -repo_list: - all repos url list from upstream-repos.git - -API: -call graph: -read_mail_content -send_account_request - email_addr - get email address - email_message_id - get email message_id - check_email_available - check email available - pub_key_value - get pub key - account - send apply account request and return account info - build_message - build email message - send_mail - call send_mail to send mail whit build message - -the returned data for account_info like: -{ - "account" => "guest", - "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", - "jumper_port" => "10000" -} -=end +# xx --email xxx --login --ssh-pubkey xxx --raw-email email-file +# samba mount +# ssh logshn (huawei, ) (install pubkey / send password) require 'json' require 'mail' require 'set' -require_relative '../defconfig.rb' +require 'optparse' +require_relative '../defconfig' names = Set.new %w[ - JUMPER_IP + JUMPER_HOST JUMPER_PORT CRYSTAL_INTRANET SEND_MAIL_PORT @@ -50,18 +22,64 @@ names = Set.new %w[ defaults = relevant_defaults(names) -JUMPER_IP = defaults['JUMPER_IP'] +JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] CRYSTAL_INTRANET = defaults['CRYSTAL_INTRANET'] SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] -def build_message(email, message_id, infos) +email_info = { + 'email_addr' => '', + 'pub_key' => '', +} + +def format_info(email_file) + mail_content = Mail.read(email_file) + + email_addr = mail_content.from[0] + pub_key = mail_content.part[1].body.decoded.split(/\r|\n/).join if mail_content.part[1].filename == 'id_rsa.pub' + + infos = { + 'email_addr' => email_addr, + 'pub_key' => pub_key, + } + + infos +end + +options = OptionParser.new do |opts| + opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key] [--raw-email email_file]\n" + opts.banner += ' -e or -f is required' + opts.banner += ' -s is optional when use -e' + + opts.separator '' + opts.separator 'options:' + + opts.on('-e email_addr', '--email email_addr', 'appoint email address') do |email_addr| + email_info['email_addr'] = email_addr + end + + opts.on('-s pub_key', '--ssh-pubkey pub_key', 'ssh pub_key to enable keyless login') do |pub_key| + email_info['pub_key'] = pub_key + end + + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + email_info = format_info(email_file) + end + + opts.on_tail('', '-h', '--help', 'show this message') do + puts opts + exit + end +end + +options.parse!(ARGV) + +def build_message(email, infos) message = <<~EMAIL_MESSAGE To: #{email} - Message-ID: #{message_id} Subject: jumper account is ready - Dear #{email} + Dear #{email.split('@')[0]} Thank you for joining us. You can use the following command to login the jumper server: @@ -79,67 +97,27 @@ def build_message(email, message_id, infos) return message end -def email_addr(mail_content) - msg = 'not an applying account email' - - raise msg unless mail_content.subject =~ /apply ssh account/i - - email = mail_content.from.join(',') - - return email -end - -# def check_email_available(mail_content, email) -# oos_list = File.read('/c/upstream-repos/repo_list').split(/\n/) -# url = mail_content.body.decoded.split(/\n/).find { |line| line =~ /https?:\/\// } -# base_url = url.split('/')[0,5].join('/') -# message = 'The url is not in upstream repo_list' -# -# raise message unless oos_list.include? base_url -# -# url_fdback = %x(curl #{url}) -# email_index = url_fdback.index email -# -# message = 'No commit info found from the url for the email' -# raise message unless email_index -# end - -def email_message_id(mail_content) - message_id = mail_content.message_id - return message_id -end - -def pub_key_value(mail_content) - pub_key = mail_content.body.decoded.split(/\n/).find { |line| line =~ /ssh-rsa/ } - return pub_key -end - def account_info(pub_key) - account_info_str = %x(curl -XGET '#{JUMPER_IP}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + account_info_str = if pub_key.empty? + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') + else + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + end account_info = JSON.parse account_info_str - return account_info + account_info end -def send_account(mail_content) - email = email_addr(mail_content) - message_id = email_message_id(mail_content) - # check_email_available(mail_content, email) - - pub_key = pub_key_value(mail_content) - acct_info = account_info(pub_key) +def send_account(email_info) + message = "No email address specified\n" + message += "use -e email_addr add a email address\n" + message += 'or use -f to add a mail file' + raise message if email_info['email_addr'].empty? + acct_info = account_info(email_info['pub_key']) - message = build_message(email, message_id, acct_info) + message = build_message(email_info['email_addr'], acct_info) %x(curl -XPOST '#{CRYSTAL_INTRANET}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -def read_mail_content(mail_file) - mail_content = Mail.read(mail_file) - - return mail_content -end - -mail_file = ARGV[0] -mail_content = read_mail_content(mail_file) -send_account(mail_content) +send_account(email_info) -- 2.23.0

2 2

[PATCH compass-ci 2/2] /src/lib/web_backend.rb: remove common es methods
by Zhang Yuhang 19 Oct '20

19 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 59 +++++++++--------------------------------- 1 file changed, 12 insertions(+), 47 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 48cfadb..721ad77 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -30,16 +30,9 @@ NOT_SHOW_FIELDS = %w[result_root].freeze ALL_FIELDS = FIELDS + NOT_SHOW_FIELDS NOT_NEED_EXIST_FIELDS = %w[error_ids upstream_repo].freeze PREFIX_SEARCH_FIELDS = ['tbox_group'].freeze -ES_CLIENT = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") COMPARE_RECORDS_NUMBER = 50 - -def es_query(query) - ES_CLIENT.search index: 'jobs*', body: query -end - -def es_count(query) - ES_CLIENT.count(index: 'jobs*', body: query)['count'] -end +JOBS = ESQuery.new(index: 'jobs*') +REPO = ESQuery.new(index: 'repo') # "vm-2p8g--212" remove "--212" # "vm-2p8g--zzz" remove "--zzz" @@ -47,7 +40,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end @@ -56,42 +49,15 @@ def filter_tbox_group(es_result) result.to_a end -def all_suite - query = { - aggs: { - all_suite: { - terms: { field: 'suite', size: 1000 } - } - }, - size: 0 - } - es_result = es_query(query)['aggregations']['all_suite']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } - - es_result -end - def all_tbox_group - query = { - aggs: { - all_tbox_group: { - terms: { field: 'tbox_group', size: 1000 } - } - }, - size: 0 - } - es_result = es_query(query)['aggregations']['all_tbox_group']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } - + es_result = JOBS.all_values('tbox_group') filter_tbox_group(es_result) end def compare_candidates_body body = { query_conditions: { - suite: all_suite, + suite: JOBS.all_values('suite'), OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, @@ -148,7 +114,7 @@ end def get_groups_matrices(conditions, dimension, must, size, from) must += build_mutli_field_subquery_body(conditions) count_query = { query: { bool: { must: must } } } - total = es_count(count_query) + total = JOBS.es_count(count_query) return {} if total < 1 query = { @@ -164,14 +130,14 @@ def get_groups_matrices(conditions, dimension, must, size, from) }] } - result = es_query(query) + result = JOBS.es_query(query) matrices = combine_group_query_data(result, dimension) while matrices.empty? from += size break if from > total query[:from] = from - result = es_query(query) + result = JOBS.es_query(query) matrices = combine_group_query_data(result, dimension) end matrices @@ -229,7 +195,7 @@ def es_search(must, size, from) must << { exists: { field: f } } end count_query = { query: { bool: { must: must } } } - total = es_count(count_query) + total = JOBS.es_count(count_query) unless size size = total from = 0 @@ -245,7 +211,7 @@ def es_search(must, size, from) } return {}, total if wrong_size?(size, from) - return es_query(query)['hits']['hits'], total + return JOBS.es_query(query)['hits']['hits'], total end def get_job(result) @@ -327,7 +293,6 @@ def get_repo_url(urls) end def get_repo(git_repo) - repo = nil if git_repo must = [{ regexp: { git_repo: ".*#{git_repo}.*" } }] repo = query_repos(must, from: 0, size: 1)[0] @@ -344,7 +309,7 @@ def query_repos(must, from: 0, size: 1) git_repo: { order: 'asc' } }] } - result = ES_CLIENT.search index: 'repo', body: query + result = REPO.es_query(query) repos = [] result['hits']['hits'].each do |r| r = r['_source'] @@ -361,7 +326,7 @@ def search_repos(git_repo, page_size, page_num) from = size * page_num must = git_repo ? [{ regexp: { git_repo: ".*#{git_repo}.*" } }] : [] count_query = { query: { bool: { must: must } } } - total = ES_CLIENT.count(index: 'repo', body: count_query)['count'] + total = REPO.es_count(count_query) return [], total if wrong_size?(size, from) return query_repos(must, from: from, size: size), total -- 2.23.0

2 2

[PATCH lkp-tests 2/2] fix(iso2rootfs): support skipping iso that not used for dailybuild
by Yu Chuan 19 Oct '20

19 Oct '20

[Why] Openeuler release two kinds of iso type now, we need to skip the released iso type that not used for dailybuild. [How] 1. Confirm whether the purpose is dailybuild by iso name. 2. Exit 0 while the released iso is not used for dailybuild. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 4782378424b5..2ce4d9659f7e 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -22,6 +22,12 @@ log_info() echo "[INFO] $*" } +exit_info() +{ + log_info "$@" + exit 0 +} + die() { echo "[ERROR] $*" >&2 @@ -56,6 +62,28 @@ check_yaml_vars() done } +check_iso_name() +{ + local skipped_iso_prefixes + case ${iso_os} in + "openeuler") + skipped_iso_prefixes=( + "openEuler-2.0-SP8" + ) + ;; + *) + return + ;; + esac + + local prefix + for prefix in "${skipped_iso_prefixes[@]}" + do + [[ ${ISO_NAME} != ${prefix}* ]] || + exit_info "${iso_os} haven't release new iso for openEuler, no need to generate rootfs" + done +} + get_daily_iso_checksum() { ISO_URL="$(curl "${iso_url_file}")" @@ -63,8 +91,10 @@ get_daily_iso_checksum() local pub_ip=$(echo "${iso_url_file}" | grep -oEw "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}") ISO_URL="$(curl "${iso_url_file}" |sed -r "s/[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/${pub_ip}/g")" } + ISO_NAME=$(basename "$ISO_URL") ISO_CHECKSUM_URL="${ISO_URL}.sha256sum" + check_iso_name curl -o "${CHECKSUM_FILE_CACHE}.tmp" "${ISO_CHECKSUM_URL}" SHA256SUM_NET=$(awk '{print $1}' "${CHECKSUM_FILE_CACHE}.tmp") @@ -118,9 +148,8 @@ check_sha256sum_update() if [ "$SHA256SUM_CACHE" == "$SHA256SUM_NET" ] then - log_info "${iso_os} haven't release new iso, no need to generate rootfs" rm -f "${CHECKSUM_FILE_CACHE}.tmp" - exit 0 + exit_info "${iso_os} haven't release new iso, no need to generate rootfs" else log_info "${iso_os} release a new iso, start to generate rootfs ..." return -- 2.23.0

3 4

[PATCH lkp-tests 1/2] feat(iso2rootfs): modify the rootfs dir layout and rootfs dir name
by Yu Chuan 19 Oct '20

19 Oct '20

[Why] 1. Modify the dir structure of dailybuild rootfs dir. 2. Modify the dir name of dailybuild rootfs dir. [How] before: "{os}/{os_arch}/debug-versions/dailybuild/{iso_version}-%Y%m%d%H%M%S" after: "{os}/{os_arch}/{iso_version}-%Y-%m-%d-%H-%M-%S" Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 524ab6695f93..4782378424b5 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,6 @@ get_daily_iso_checksum() mount_rootfs() { - DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +102,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-newest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +254,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +309,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -366,7 +365,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ + ln -s "$(basename ${ROOTFS_DES_DIR})" \ "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 2

[PATCH lkp-tests] tests: add test-tools
by Li Ping 19 Oct '20

19 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- tests/test-tools | 37 +++++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 tests/test-tools diff --git a/tests/test-tools b/tests/test-tools new file mode 100755 index 00000000..cee882aa --- /dev/null +++ b/tests/test-tools @@ -0,0 +1,37 @@ +#!/bin/bash +# - testsuite +# - case + +cd $BENCHMARK_ROOT/test-tools/mugen || exit +: "${testsuite:=smoke-testing}" +: "${case:=oe_test_dd_001}" + +set_env() +{ + bash runoet.sh -c + source ~/.bash_profile +} + +test_one_case() +{ + + bash runoet.sh -f $testsuite -r $case -x +} + +test_all_cases() +{ + bash runoet.sh -a +} + +run() +{ + set_env + if [ "${testsuite}" == "all" ] + then + test_all_cases + else + test_one_case + fi +} + +run -- 2.23.0

4 3

[PATCH v7 compass-ci 6/6] container/mail-robot: mail-robot.md
by Luan Shengde 19 Oct '20

19 Oct '20

guidance file for mail-robot Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..d5eeaa9 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,45 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +monitor received mails + check if the mail is for applying uuid/account + +## steps overview + +1. monitor mailbox dir + - check for new email file + +2. check email + - check email subject + - return if unmatched subject + - call answerback-email + - check email available + - check base_url for the oss_url in upstream-repos list + if not, it will raise "url not in upstream-repos list" error + - check if the email has commit for the url + - if gitee, git clone the repo and check + if non-gitee, curl url to check + - if not, raise "no commit found" error + - check pub_key + - check if there is a pub_key in the email + if not, it will raise 'no pub_key' error + - apply ssh account + - apply account uuid + - write account info to es + - send uuid to user + - mv mail file to drafts + +3. continue the monitor + - after 1 and 2, continue to monitor the mailbox + +4. service log + - use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot + +5. usage: + - mail subject: apply account + - mail body: commit url + - mail attachment: id_rsa.pub -- 2.23.0

1 0

[PATCH v7 compass-ci 3/6] container/mail-robot: build file
by Luan Shengde 19 Oct '20

19 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] container/send-mail: update start
by Luan Shengde 19 Oct '20

19 Oct '20

add setup default variables update heredoc Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/start | 89 +++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 27 deletions(-) diff --git a/container/send-mail/start b/container/send-mail/start index 80be86a..068fc1e 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -1,29 +1,68 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -. $CCI_SRC/container/defconfig.sh +require 'set' +require_relative '../defconfig' -docker_rm send-mail +docker_rm 'send-mail' -cmd=( - docker run - --restart=always - --name=send-mail - -u nobody - -d - -p 11311:11311 - -v /etc/localtime:/etc/localtime:ro - send-mail -) +names = Set.new %w[ + EMAIL_PSWD + SEND_MAIL_PORT + SMTP_SERVER_ADDR + SMTP_SERVER_PORT + EMAIL_DOMAIN + EMAIL_ADDR + DEFAULT_EMAIL +] -"${cmd[@]}" +defaults = relevant_defaults(names) -cat<<EOF +EMAIL_PSWD = defaults['EMAIL_PSWD'] +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] +SMTP_SERVER_ADDR = defaults['SMTP_SERVER_ADDR'] +SMTP_SERVER_PORT = defaults['SMTP_SERVER_PORT'] +EMAIL_DOMAIN = defaults['EMAIL_DOMAIN'] +EMAIL_ADDR = defaults['EMAIL_ADDR'] +DEFAULT_EMAIL = defaults['DEFAULT_EMAIL'] + +cmd = %W[ + docker run + --restart=always + --name=send-mail + -u nobody + -d + -e EMAIL_PSWD=#{EMAIL_PSWD} + -e SEND_MAIL_PORT=#{SEND_MAIL_PORT} + -e SMTP_SERVER_ADDR=#{SMTP_SERVER_ADDR} + -e SMTP_SERVER_PORT=#{SMTP_SERVER_PORT} + -e EMAIL_DOMAIN=#{EMAIL_DOMAIN} + -e EMAIL_ADDR=#{EMAIL_ADDR} + -e DEFAULT_EMAIL=#{DEFAULT_EMAIL} + -p #{SEND_MAIL_PORT}:#{SEND_MAIL_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-driver json-file + send-mail +] + +cmd += ['bash', '-c', '/usr/local/bin/mail-post.rb'] + +system(*cmd) + +puts <<HEREDOC port: 11311 + +default from email: + when sending crystal.ci email. + team(a)crystal.ci if no from email appiontted + when sending to WAN email + compass-ci(a)qq.com in all cases + --- data='{ -"from": "email_from_addr" # default: team(a)crystal.ci +"from": "email_from_addr" "subject": "email subject", "to": "email_to_addr", "body": "email message" @@ -32,19 +71,17 @@ data='{ or data=" -from: email_from_addr # default: team(a)crystal.ci +from: email_from_addr subject: email subject to: email_to_addr body: email message" -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_yaml' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_yaml' -d "$data" +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_yaml' -d "$data" --- data=" -From: email_from_addr # default: team(a)crystal.ci +From: email_from_addr To: email_to_addr Subject: email_subject @@ -53,8 +90,6 @@ mail_msg_line2 ... " -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_text' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_text' -d "$data" -EOF +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_text' -d "$data" +HEREDOC -- 2.23.0

2 1

[PATCH v2 compass-ci 2/4] container/send-mail: send-mail.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add setup smtp for qq mail server enable send mail via smtp.qq.com Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/send-mail.rb | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index f459e78..844cad5 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -5,15 +5,29 @@ require 'mail' -mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp +# mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp # setup smtp config -smtp = { - address: mail_server, - enable_starttls_auto: false -} +def intranet_smtp(mail_server) + smtp = { + address: mail_server, + enable_starttls_auto: false + } + Mail.defaults { delivery_method :smtp, smtp } +end -Mail.defaults { delivery_method :smtp, smtp } +def internet_smtp + smtp = { + address: (ENV['SMTP_SERVER_ADDR']).to_s, + port: (ENV['SMPT_SERVER_PORT']), + domain: (ENV['EMAIL_DOMAIN']).to_s, + user_name: (ENV['EMAIL_ADDR']).to_s, + password: (ENV['EMAIL_PSWD']).to_s, + openssl_verify_mode: 'none', + enable_starttls_auto: true + } + Mail.defaults { delivery_method :smtp, smtp } +end # send mail def send_mail(mail_info) -- 2.23.0

2 1

[PATCH compass-ci 1/2] user-client/maintain: test os rootfs in z9 tbox
by Liu Yinsi 19 Oct '20

19 Oct '20

1. no needed my-qemu.sh after submit job in z9 tbox, remove. 2. monitor job_state boot/extract_finished to get job runtime. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 124 +++++++++--------- 1 file changed, 65 insertions(+), 59 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 21e1428..2c3fc9f 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -1,62 +1,68 @@ -#!/bin/bash -e +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ +# # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -# os test rounds -test_times=1 - -test_logs=walk-test.report -test_yaml=iperf-walk-os.yaml -test_os=( - 'openeuler aarch64 20' - 'openeuler aarch64 20.03' - 'centos aarch64 7.6' - 'centos aarch64 7.8' - 'centos aarch64 8.1.1911' +require "#{ENV['LKP_SRC']}/lib/monitor" + +TEST_OS = [ + 'openeuler aarch64 20.03', + 'centos aarch64 7.6.1810', + 'centos aarch64 7.8.2003', + 'centos aarch64 8.1.1911', 'debian aarch64 sid' -) - -trap 'echo User Cancelled! && exit' SIGINT - -sec2date() { - date -d@$1 +"%Y%m%d_%H:%M:%S" -} - -logging() { - if [ 'new' == "$1" ] ; then - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" ${@:2} > $test_logs - else - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" $@ >> $test_logs - fi -} - -create_test_yaml() { - local hwarch os os_arch os_ver cur_os - hwarch=$(arch) - cur_os="$@" - os=$(echo $cur_os |cut -d' ' -f1) - os_arch=$(echo $cur_os |cut -d' ' -f2) - os_ver=$(echo $cur_os |cut -d' ' -f3) - sed "s/USER/$USER/" $test_yaml > test.yaml - sed -i "s/OS_ARCH/$os_arch/" test.yaml - sed -i "s/OS_VER/$os_ver/" test.yaml - sed -i "s/OS/$os/" test.yaml - echo "$hwarch $os/$os_arch/$os_ver" -} - -job_test() { - local job_id bef aft - job_id=$(submit test.yaml |awk '{print $NF}') - bef=$(date +%s) - ./my-qemu.sh - aft=$(date +%s) - logging "$1" "$2" "$3" "$(sec2date $bef)" "$(sec2date $aft)" "$((aft - bef))" "$job_id" -} - -logging new "Index" "hwArch" "OS" "Begin" "End" "Cost/sec" "Job_ID" -for round in $(seq $test_times) -do - for idx in "${!test_os[@]}" - do - job_test $round $(create_test_yaml "${test_os[$idx]}") - done -done +] + +MOUNT_TYPE = ['cifs', 'initramfs'] + +def logging(job_id, os_params, start_time, end_time, cost_time) + line = format("%-12s %-36s %-28s %-28s %-8s\n", job_id, os_params, start_time, end_time, cost_time) + f = File.new('walk-test.report', 'a') + f.write(line) + f.close +end + +def monitor_job(job_id, job_state) + res = monitor({'job_id' => job_id, 'job_state' => job_state}, {'stop' => true}, '600s') + if res == 0 + Time.now + end +end + +def monitor(query, actions, timeout) + monitor = Monitor.new + monitor.overrides = query + monitor.action = actions + monitor.run(timeout) +end + +def submit_job(os_array) + os = os_array[0] + os_arch = os_array[1] + os_version = os_array[2] + os_mount = os_array[3] + message = %x(submit -s 'os: #{os}' -s 'os_arch: #{os_arch}' -s 'os_version: #{os_version}' -s 'os_mount: #{os_mount}' iperf-walk-os.yaml) + puts message + + job_id = message.split('=')[1].chomp + os_params = os_array.join('/') + start_time = monitor_job(job_id, 'boot') + end_time = monitor_job(job_id, 'extract_finished') + cost_time = (end_time - start_time).floor + + logging(job_id, os_params, start_time, end_time, cost_time) +end + +def get_os_array(os_mount) + TEST_OS.each do | item | + os_array = item.split.push(os_mount) + + submit_job(os_array) + end +end + +system "walk-test.report" + +logging("Job_ID", "OS", "Begin", "End", "Cost/sec") + +MOUNT_TYPE.each {| os_mount | get_os_array(os_mount)} -- 2.23.0

3 4

[PATCH compass-ci] fix: taskqueue: simplify logs
by Tong Qunfeng 19 Oct '20

19 Oct '20

[why] current taskqueue show 4 lines for 1 debug message - empty line for split: - end user request ip:port, request body: 172.17.0.1:48956 {"domain":"..."} - response body: <-- - kemal frame log: 2020-10... 200 POST /add?queue=... 710.15µs cut it two 2 line, and duplicate information - kemal frame log has time in {2020-10...} and span {710.15us} Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/taskqueue/taskqueue.cr | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/src/taskqueue/taskqueue.cr b/src/taskqueue/taskqueue.cr index c5c202d..98c5623 100644 --- a/src/taskqueue/taskqueue.cr +++ b/src/taskqueue/taskqueue.cr @@ -18,16 +18,22 @@ class TaskQueue @@rate_limiter.bucket(:l1pms, 1_u32, 0.001.seconds) @@rate_limiter.bucket(:l100pms, 1_u32, 0.00001.seconds) + # logs example: + # from: {172.17.0.1:6952} body {"domain":"compass-ci"} <-- ack: {"id":"z9.134780"} + # | from: {172.17.0.1:6952} <-- ack: {} + # 2020-10-19 03:25:01 UTC 201 PUT /consume?queue=sched%2Fdc-2g%2Fidle 356.51µs + # + # "from" is puts by this debug_message + # "2020-10..." auto puts by kemal frame, time in {2020-10-19 03:25:01 UTC} and span {356.51us} def debug_message(env, response, time_in) - puts("\n") + logs = "from: {#{env.request.remote_address}}" - from_message = "#{time_in} --> #{env.request.remote_address}" - if env.request.body != nil - from_message += " #{env.request.body}" - end - puts(from_message) + logs += " body: #{env.request.body}" if env.request.body + + logs += " <-- ack: " + logs += response ? "#{response}" : "{}" - puts("#{Time.utc} <-- #{response}") + puts(logs) end def run -- 2.23.0

1 0

[PATCH compass-ci] add exception handing for docker.rb
by Xiao Shenwei 19 Oct '20

19 Oct '20

[why] when run multi-docker job on physical machine, run 50 client to request job, if server down or error like Connection refused. these client will exit. when the server recovers, need to run these client again. [how] add exception handing to ensure all client does not exit --- providers/docker/docker.rb | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index c241d3c..2239930 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -88,6 +88,19 @@ def main(hostname) run(hostname, load_path, hash) end +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.message + sleep 10 + ensure + sleep 5 + end + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -99,10 +112,7 @@ def multi_docker(hostname, nr_container) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop do - main("#{hostname}-#{i}") - sleep 5 - end + loop_main("#{hostname}-#{i}") end pids << pid end -- 2.23.0

3 4

[PATCH v2 compass-ci 1/5] container/send-internet-mail: mail-post.rb
by Luan Shengde 18 Oct '20

18 Oct '20

send mail interface: get send mail request build email info from the request data call send-internet-mail to send email with email info Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/mail-post.rb | 46 +++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100755 container/send-internet-mail/mail-post.rb diff --git a/container/send-internet-mail/mail-post.rb b/container/send-internet-mail/mail-post.rb new file mode 100755 index 0000000..3b33c7f --- /dev/null +++ b/container/send-internet-mail/mail-post.rb @@ -0,0 +1,46 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'yaml' +require 'open3' +require_relative 'send-internet-mail.rb' + +set :bind, '0.0.0.0' +set :port, 11312 + +post '/send_mail_yaml' do + data = YAML.safe_load request.body.read + raise TypeError, data, 'request data type error' unless data.class.eql? Hash + + mail_info = { + 'references' => data['references'] || '', + 'subject' => data['subject'] || '', + 'to' => data['to'] || '', + 'body' => data['body'] || '' + } + check_send_mail(mail_info) if data +end + +post '/send_mail_text' do + data = Mail.read_from_string(request.body.read) + + mail_info = { + 'references' => data.references || '', + 'subject' => data.subject || '', + 'to' => data.to || '', + 'body' => data.body.decoded || '' + } + check_send_mail(mail_info) if data +end + +def check_send_mail(mail_info) + raise 'No subject.' if mail_info['subject'].empty? + raise 'No email address.' if mail_info['to'].empty? + raise 'No email body.' if mail_info['body'].empty? + + send_mail(mail_info) +end -- 2.23.0

3 5

[PATCH lkp-tests] jobs/fio-basic-openeuler.yaml: add fio test args
by Wei Jihui 17 Oct '20

17 Oct '20

1. add disk args, it should be tested on physical machine 2. raise nr_task to 24, for numjobs is 24. 3. add direct args Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/fio-basic-openeuler.yaml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/jobs/fio-basic-openeuler.yaml b/jobs/fio-basic-openeuler.yaml index 3fcdaac8..67d4e998 100644 --- a/jobs/fio-basic-openeuler.yaml +++ b/jobs/fio-basic-openeuler.yaml @@ -3,12 +3,18 @@ suite: fio-basic category: benchmark runtime: 60 +disk: +- 1HDD +fs: +- ext4 +nr_task: 24 + fio-setup-basic: ioengine: libaio iodepth: 1024 - nr_task: 2 test_size: 1G rwmixread: 70 + direct: 1 rw: - write - read -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: load yaml config in advance
by Wei Jihui 17 Oct '20

17 Oct '20

[why] for commit c725e0f022b7: def each_jobs(&block) each_job_init - load_hosts_config job = deepcopy(@job) @job2 = {} load_defaults + load_hosts_config each_job_init each_job(&block) @jobs.each do |hash| 1. it exec load_defaults first, so it not load hosts file when start exec load_defaults. load_defaults call expand_params, expand_params maybe expand host params, but do not have load hosts file, load_one_defaults can load host file, so exec load_one_defaults in advance. 2. load_one_defaults(file, @job) use @job is because load_hosts_file is add host config to @job, so it should be executed before "job=deepcopy(@job)" Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..27c4bd93 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -400,6 +400,11 @@ class Job @file_loaded ||= {} end + Dir.glob(["/etc/compass-ci/defaults/*.yaml", + "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| + load_one_defaults(file, @job) + end + i = include_files job = deepcopy(@job) revise_hash(job, deepcopy(@job2), true) @@ -418,11 +423,6 @@ class Job end @jobx = nil - Dir.glob(["/etc/compass-ci/defaults/*.yaml", - "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| - load_one_defaults(file, job) - end - for_each_in(job, i.keys) do |_pk, _h, k, v| job['___'] = v -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: add define files when program is cci-depends
by Wei Jihui 17 Oct '20

17 Oct '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..9a08dc30 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -617,9 +617,9 @@ class Job program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| - # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg + # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files - if %w(makepkg makepkg-deps pack-deps cci-makepkg).include?(program) + if %w(makepkg makepkg-deps pack-deps cci-makepkg cci-depends).include?(program) program = programs_params_hash[program]["benchmark"] end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: merge overrides to defaults
by Wei Jihui 17 Oct '20

17 Oct '20

[why] submit job.yaml testbox=vm-2p8g it should use vm-2p8g to load host file, not use defaults's testbox [slove] merge @overrides to @defaults, then use @defaults to load host file Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..4e398a5a 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -382,7 +382,8 @@ class Job if defaults.is_a?(Hash) && !defaults.empty? @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) - load_hosts_config if check_load_hosts_config(defaults) + @defaults.merge!(@overrides) + load_hosts_config if check_load_hosts_config(@defaults) end @file_loaded[file] = true true -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: set flag true only when find hosts file
by Wei Jihui 17 Oct '20

17 Oct '20

it should set load_hosts_done is true only when find and load hosts file Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..a5fe450f 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -307,10 +307,10 @@ class Job hwconfig = load_yaml(hosts_file, nil) @job[source_file_symkey(hosts_file)] = nil @job.merge!(hwconfig) { |_k, a, _b| a } # job's key/value has priority over hwconfig + @load_hosts_done = true end @job['arch'] ||= 'aarch64' - @load_hosts_done = true end def check_set_tbox_group -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor.rb: return the monitored logs
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] When other people use monitor to get logs, the logs only output, not return. It's hard to deal the logs. [How] Append result logs to a list. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 2a10b5c1..baf172d3 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -11,7 +11,7 @@ require 'eventmachine' require 'json' class Monitor - attr_accessor :monitor_url, :query, :overrides, :action, :job + attr_accessor :monitor_url, :query, :overrides, :action, :job, :result def initialize(monitor_url = '', query = {}, action = {}) @monitor_url = monitor_url @@ -25,6 +25,7 @@ class Monitor @exit_status_code = 0 @defaults = {} load_default + @result = [] end def load_default @@ -122,6 +123,7 @@ class Monitor ws.on :message do |event| data = JSON.parse(event.data) + @result << data output(data) connect(data, ws) -- 2.23.0

1 0

[PATCH compass-ci] monitoring/parse_serial_logs.cr: fix ameba warning
by Wu Zhende 17 Oct '20

17 Oct '20

parse_serial_logs.cr:46:5 [W] Lint/UselessAssign: Useless assignment to variable `host` > host = File.basename(msg["serial_path"].to_s) ^ parse_serial_logs.cr:95:5 [W] Lint/UselessAssign: Useless assignment to variable `job_id` > job_id = matched.named_captures["job_id"] ^ parse_serial_logs.cr:101:5 [W] Lint/UselessAssign: Useless assignment to variable `job` > job = Elasticsearch::Client.new.get_job_content(job_id) ^ Finished in 10.93 milliseconds 1 inspected, 3 failures. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/parse_serial_logs.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index a000ca3..6c8a220 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -43,7 +43,7 @@ class SerialParser def host_in_msg(msg) return unless msg["serial_path"]? - host = File.basename(msg["serial_path"].to_s) + File.basename(msg["serial_path"].to_s) end def detect_start_or_end(msg, host, pattern_list) @@ -92,13 +92,13 @@ class SerialParser matched = msg["message"].to_s.match(/.*\/job_initrd_tmpfs\/(?<job_id>.*?)\//) return unless matched - job_id = matched.named_captures["job_id"] + matched.named_captures["job_id"] end def find_job(job_id) return unless job_id - job = Elasticsearch::Client.new.get_job_content(job_id) + Elasticsearch::Client.new.get_job_content(job_id) end def dump_cache(job, msg, host) -- 2.23.0

1 0

[PATCH compass-ci] container/fluentd: forward docker logs to master-fluentd
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] I use a fluentd to collect docker logs, and use three output plugins:stdout/es/mq. But when I use them at the same time, will meet some problems. This fluentd will be very slow. Logs will be stacked. Another problem is that dockerd always sends logs to the same worker of fluentd.The multi-process function of fluentd can't work. [How] Change the architecture. A fluentd collects docker logs, do stdout and forward logs to master-fluentd. master-fluentd send logs to es and mq. Like this: |-----------| |--------|--> stdout |--------------|---> es |docker logs| --> |fluentd | |master-fluentd| |-----------| |--------|--> forword---> |--------------|---> mq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/fluentd/docker-fluentd.conf | 33 ++++++++++----------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/container/fluentd/docker-fluentd.conf b/container/fluentd/docker-fluentd.conf index 8e76441..ac73dfb 100644 --- a/container/fluentd/docker-fluentd.conf +++ b/container/fluentd/docker-fluentd.conf @@ -60,28 +60,19 @@ </store> <store> - @type elasticsearch - host 172.17.0.1 - port 9202 - suppress_type_name true - flush_interval 1s - index_name ${tag} - ssl_verify false - num_threads 2 - </store> - - <store> - @type rabbitmq - host 172.17.0.1 - exchange logging-test - exchange_type fanout - exchange_durable false - heartbeat 10 - <format> - @type json - </format> + @type forward + flush_interval 0 + send_timeout 60 + heartbeat_type udp + heartbeat_interval 1 + recover_wait 10 + hard_timeout 60 + <server> + master-fluentd + host 172.17.0.1 + port 24225 + </server> </store> - </match> <label @FLUENT_LOG> -- 2.23.0

1 0

[PATCH lkp-tests] skip build these container for multi-docker
by Xiao Shenwei 17 Oct '20

17 Oct '20

multi-docker do't need thses container, skip it Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- setup/simplify-ci | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index e3ee3f77..eb93c150 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -36,23 +36,22 @@ boot_ipxe() run_service() { + [[ $suite == "multi-docker" ]] && return + ( cd $CCI_SRC/container/dnsmasq || return - ./build - ./start + ./build && ./start boot_ipxe )& ( cd $CCI_SRC/container/qemu-efi || return - ./build - ./install + ./build && ./install )& ( cd $CCI_SRC/container/fluentd-base || return ./build cd $CCI_SRC/container/sub-fluentd || return - ./build - ./start + ./build && ./start )& } -- 2.23.0

1 1

[PATCH compass-ci] container/master-fluentd: add a new fluentd container
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] I use a fluentd to collect docker logs, and use three output plugins:stdout/es/mq. But when I use them at the same time, will meet some problems. This fluentd will be very slow. Logs will be stacked. Another problem is that dockerd always sends logs to the same worker of fluentd.The multi-process function of fluentd can't work. [How] Change the architecture. A fluentd collects docker logs, do stdout and forward logs to master-fluentd. master-fluentd send logs to es and mq. Like this: |-----------| |--------|--> stdout |--------------|---> es |docker logs| --> |fluentd | |master-fluentd| |-----------| |--------|--> forword---> |--------------|---> mq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/Dockerfile | 6 +++ container/master-fluentd/build | 6 +++ container/master-fluentd/build-depends | 1 + container/master-fluentd/docker-fluentd.conf | 48 ++++++++++++++++++++ container/master-fluentd/start | 26 +++++++++++ 5 files changed, 87 insertions(+) create mode 100644 container/master-fluentd/Dockerfile create mode 100755 container/master-fluentd/build create mode 100755 container/master-fluentd/build-depends create mode 100644 container/master-fluentd/docker-fluentd.conf create mode 100755 container/master-fluentd/start diff --git a/container/master-fluentd/Dockerfile b/container/master-fluentd/Dockerfile new file mode 100644 index 0000000..192ece3 --- /dev/null +++ b/container/master-fluentd/Dockerfile @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM fluentd-base:alpine + +COPY --chown=1090:1090 docker-fluentd.conf /fluentd/etc/docker-fluentd.conf diff --git a/container/master-fluentd/build b/container/master-fluentd/build new file mode 100755 index 0000000..7d0291d --- /dev/null +++ b/container/master-fluentd/build @@ -0,0 +1,6 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +system 'docker build -t master-fluentd:alpine .' diff --git a/container/master-fluentd/build-depends b/container/master-fluentd/build-depends new file mode 100755 index 0000000..29ee9d3 --- /dev/null +++ b/container/master-fluentd/build-depends @@ -0,0 +1 @@ +fluentd-base diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf new file mode 100644 index 0000000..ef97d5c --- /dev/null +++ b/container/master-fluentd/docker-fluentd.conf @@ -0,0 +1,48 @@ +<system> + workers 10 +</system> + +<source> + @type forward + bind 0.0.0.0 +</source> + +<match **> + @type copy + + <store> + @type elasticsearch + host 172.17.0.1 + port 9202 + suppress_type_name true + flush_interval 1s + index_name ${tag} + ssl_verify false + log_es_400_reason true + with_transporter_log true + </store> + + <store> + @type rabbitmq + host 172.17.0.1 + exchange logging-test + exchange_type fanout + exchange_durable false + heartbeat 10 + <format> + @type json + </format> + </store> +</match> + +<label @FLUENT_LOG> + <match fluent.*> + @type stdout + <format> + @type stdout + output_type single_value + message_key message + add_newline true + </format> + </match> +</label> diff --git a/container/master-fluentd/start b/container/master-fluentd/start new file mode 100755 index 0000000..9e02980 --- /dev/null +++ b/container/master-fluentd/start @@ -0,0 +1,26 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "master-fluentd" + +cmd = %w[ + docker run + --restart=always + --name master-fluentd + -v /etc/localtime:/etc/localtime:ro + -d + -u 1090:1090 + -p 24225:24224/tcp + -p 24225:24224/udp + -e FLUENTD_CONF=docker-fluentd.conf + --log-driver json-file + --log-opt max-size=1g + master-fluentd:alpine +] +cmd += ['sh', '-c', 'umask 002 & fluentd -c /fluentd/etc/docker-fluentd.conf'] + +system(*cmd) -- 2.23.0

1 0