- Compass-ci - mailweb.openeuler.org

[PATCH v7 compass-ci 1/6] container/mail-robot: mail-robot.rb
by Luan Shengde 19 Oct '20

19 Oct '20

mail robot for email application monitor mailbox check new added email file in new check new mail's subject return if unmatched subject call answerback-email check to apply uuid/account Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/mail-robot.rb | 43 ++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..70b8b25 --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'answerback-email.rb' + +mail_inbox = '/srv/cci/Maildir/.compass-ci/new/' +mail_drafts = '/srv/cci/Maildir/.compass-ci/cur/' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + unless added.empty? + added.each do |mail_file| + begin + check_to_send_account(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + puts e.backtrace + end + end + end + end + listener.start + sleep +end + +def check_to_send_account(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + subject = mail_content.subject + return unless subject =~ /apply account/i + + assign_uuid = AssignUuid.new(mail_content) + assign_uuid.send_account + FileUtils.mv(mail_file, mail_drafts) +end + +monitor_dir(mail_inbox, mail_drafts) -- 2.23.0

3 4

[PATCH lkp-tests] lib/job.rb: load hosts file for atomic job
by Wei Jihui 19 Oct '20

19 Oct '20

[why] for commit c725e0f022b7: def each_jobs(&block) each_job_init - load_hosts_config job = deepcopy(@job) @job2 = {} load_defaults + load_hosts_config each_job_init each_job(&block) @jobs.each do |hash| job = deepcopy(@job) this @job has no load hosts config, so it should do it under "@jobs.each do |hash|" for atomic job. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..6359254e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -696,9 +696,11 @@ class Job each_job_init each_job(&block) @jobs.each do |hash| + @load_hosts_done = false @job = deepcopy(job) @job2 = hash load_defaults + load_hosts_config each_job_init each_job(&block) end -- 2.23.0

2 2

[PATCH v7 compass-ci 5/6] container/mail-robot: start file
by Luan Shengde 19 Oct '20

19 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/start | 39 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..108ed83 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,39 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +docker_rm 'mail-robot' + +names = Set.new %w[ + JUMPER_HOST + JUMPER_PORT + SEND_INTERNET_MAIL_PORT +] + +defaults = relevant_defaults(names) + +JUMPER_HOST = defaults['JUMPER_HOST'] +JUMPER_PORT = defaults['JUMPER_PORT'] +SEND_MAIL_PORT = defaults['SEND_INTERNET_MAIL_PORT'] + +cmd = %W[ + docker run + --restart=always + --name=mail-robot + -d + -e J_HOST=#{JUMPER_HOST} + -e J_PORT=#{JUMPER_PORT} + -e S_M_PORT=#{SEND_MAIL_PORT} + -v /c/upstream-repos:/c/upstream-repos:ro + -v /srv/cci/Maildir:/srv/cci/Maildir:rw + -v /c/compass-ci:/c/compass-ci:ro + mail-robot +] + +cmd += ['bash', '-c', 'umask 002 && ruby ./mail-robot.rb'] + +system(*cmd) -- 2.23.0

2 2

[PATCH v2 compass-ci] sched: throw an exception when deps/pkg does not exist
by Cao Xueliang 19 Oct '20

19 Oct '20

When submitting a job, an exception is thrown when deps/pkg does not exist on the server. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..7785c2a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -389,10 +389,12 @@ class Job program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") && File.exists?(deps_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") && File.exists?(pkg_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) end end -- 2.23.0

2 5

[PATCH v2 lkp-tests] sbin/submit: submit with yaml file without directory
by Wei Jihui 19 Oct '20

19 Oct '20

if get string named with *.yaml from ARGV, treat it as job yaml. even it is without directory. and find_jobfile check whether the job yaml exists. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 831bfb5a..ce8accf0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -60,7 +60,7 @@ options.parse!(ARGV) find_job_yaml = false ARGV.delete_if do |arg| - find_job_yaml = true if File.file?(arg) + find_job_yaml = true if arg =~ /^.+\.yaml$/ if arg.index '=' if find_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') -- 2.23.0

2 1

[PATCH v2 lkp-tests 1/2] feat(iso2rootfs): modify the rootfs dir layout and rootfs dir name
by Yu Chuan 19 Oct '20

19 Oct '20

[Why] 1. Modify the dir structure of dailybuild rootfs dir. 2. Modify the dir name of dailybuild rootfs dir. [How] before: "{os}/{os_arch}/debug-versions/dailybuild/{iso_version}-%Y%m%d%H%M%S" after: "{os}/{os_arch}/{iso_version}-%Y-%m-%d-%H-%M-%S" Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 524ab6695f93..89328f2f4a9b 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,6 @@ get_daily_iso_checksum() mount_rootfs() { - DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +102,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${iso_version}-latest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +254,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +309,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -366,8 +365,7 @@ post_works() local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && rm -f "${soft_link}" && - ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ - "${soft_link}" + ln -s "$(basename ${ROOTFS_DES_DIR})" "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 1

[PATCH compass-ci 1/2] /lib/es_query.rb: add web-backend common es methods
by Zhang Yuhang 19 Oct '20

19 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- lib/es_query.rb | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index aaaadc2..9a4e45a 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -9,12 +9,20 @@ require_relative 'constants.rb' class ESQuery HOST = (ENV.key?('ES_HOST') ? ENV['ES_HOST'] : ES_HOST) PORT = (ENV.key?('ES_PORT') ? ENV['ES_PORT'] : ES_PORT).to_i - def initialize(host = HOST, port = PORT, index: 'jobs') + def initialize(host = HOST, port = PORT, index: 'jobs*') @index = index @client = Elasticsearch::Client.new url: "http://#{host}:#{port}" raise 'Connect Elasticsearch error!' unless @client.ping end + def es_query(query) + @client.search index: @index, body: query + end + + def es_count(query) + @client.count(index: @index, body: query)['count'] + end + # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) def multi_field_query(items, size: 10_000) @@ -26,7 +34,23 @@ class ESQuery } }, size: size } - @client.search index: 'jobs*', body: query + es_query(query) + end + + def all_values(field, size: 1000) + query = { + aggs: { + all_field: { + terms: { field: field, size: size } + } + }, + size: 0 + } + es_result = es_query(query)['aggregations']['all_field']['buckets'] + es_result.sort_by! { |h| h['doc_count'] } + es_result.reverse!.map! { |x| x['key'] } + + es_result end def query_by_id(id) -- 2.23.0

3 4

[PATCH lkp-tests] sbin/submit: submit with yaml file without directory
by Wei Jihui 19 Oct '20

19 Oct '20

if get string named with *.yaml from ARGV, task it as job yaml. even it is without directory. and find_jobfile check whether the job yaml exists. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 831bfb5a..ce8accf0 100755 --- a/sbin/submit +++ b/sbin/submit @@ -60,7 +60,7 @@ options.parse!(ARGV) find_job_yaml = false ARGV.delete_if do |arg| - find_job_yaml = true if File.file?(arg) + find_job_yaml = true if arg =~ /^.+\.yaml$/ if arg.index '=' if find_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') -- 2.23.0

2 1

[PATCH compass-ci] fix: sched: alter conditions when submitting cluster and normal jobs
by Ren Wen 19 Oct '20

19 Oct '20

1) when a job without 'cluster' (a job field), or has 'cluster' but starts with 'cs-localhost', the job is a normal job; otherwise, it's a cluster job. 2) alter the numbers of return values of 'get_cluster_config' function, to only return 'cluster config'. Because the conditions how to judge a job is whether a cluster job are changed. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 33 ++++++++++++--------------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a4ddcec..a18bdee 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -139,22 +139,12 @@ class Sched return node_state end - # EXAMPLE: - # cluster_file: "cs-lkp-hsw-ep5" - # return: Hash(YAML::Any, YAML::Any) | Nil, 0 | <hosts_size> - # {"lkp-hsw-ep5" => {"roles" => ["server"], "macs" => ["ec:f4:bb:cb:7b:92"]}, - # "lkp-hsw-ep2" => {"roles" => ["client"], "macs" => ["ec:f4:bb:cb:54:92"]}}, 2 + # get cluster config using own lkp_src cluster file, + # a hash type will be returned def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) lkp_src = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) cluster_file_path = Path.new(lkp_src, "cluster", cluster_file) - - if File.file?(cluster_file_path) - cluster_config = YAML.parse(File.read(cluster_file_path)).as_h - hosts_size = cluster_config.values.size - return cluster_config, hosts_size - end - - return nil, 0 + return YAML.parse(File.read(cluster_file_path)).as_h end def get_commit_date(job) @@ -177,16 +167,17 @@ class Sched job = Job.new(job_content, job_content["id"]?) job["commit_date"] = get_commit_date(job) + # it is not a cluster job if cluster field is empty or + # field's prefix is 'cs-localhost' cluster_file = job["cluster"] - if cluster_file != "" - cluster_config, hosts_size = get_cluster_config( - cluster_file, job.lkp_initrd_user, job.os_arch) - - return submit_cluster_job( - job, cluster_config.not_nil!) if hosts_size >= 2 + if cluster_file.empty? || cluster_file.starts_with?("cs-localhost") + return submit_single_job(job) + else + cluster_config = get_cluster_config(cluster_file, + job.lkp_initrd_user, + job.os_arch) + return submit_cluster_job(job, cluster_config) end - - return submit_single_job(job) rescue ex puts ex.inspect_with_backtrace return [{ -- 2.23.0

2 3

[PATCH v2 compass-ci] container/assign-account: answerback-email.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add parameters for answerback-email -e,--email email_addr: add email address -s,--ssh-pubkey pub_key: add ssh pub_key -f,--raw-email mail_file: add email_adress[,pub_key] via email file why: easier for administrator to manully assign jumper account for user with: - user's email - user's email with pub_key - user's email_file Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/assign-account/answerback-email.rb | 165 ++++++++----------- 1 file changed, 70 insertions(+), 95 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index b5283ba..0789c38 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -3,46 +3,18 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -=begin - -repo_list: - all repos url list from upstream-repos.git - -API: -call graph: -read_mail_content -send_account_request - email_addr - get email address - email_message_id - get email message_id - check_email_available - check email available - pub_key_value - get pub key - account - send apply account request and return account info - build_message - build email message - send_mail - call send_mail to send mail whit build message - -the returned data for account_info like: -{ - "account" => "guest", - "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", - "jumper_port" => "10000" -} -=end +# xx --email xxx --login --ssh-pubkey xxx --raw-email email-file +# samba mount +# ssh logshn (huawei, ) (install pubkey / send password) require 'json' require 'mail' require 'set' -require_relative '../defconfig.rb' +require 'optparse' +require_relative '../defconfig' names = Set.new %w[ - JUMPER_IP + JUMPER_HOST JUMPER_PORT CRYSTAL_INTRANET SEND_MAIL_PORT @@ -50,27 +22,71 @@ names = Set.new %w[ defaults = relevant_defaults(names) -JUMPER_IP = defaults['JUMPER_IP'] +JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] CRYSTAL_INTRANET = defaults['CRYSTAL_INTRANET'] SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] -def build_message(email, message_id, infos) +apply_info = { + 'email_addr' => nil, + 'pub_key' => nil +} + +def format_info(email_file, apply_info) + mail_content = Mail.read(email_file) + + email_addr = mail_content.from[0] + pub_key = mail_content.part[1].body.decoded.split(/\r|\n/).join if mail_content.part[1].filename == 'id_rsa.pub' + + apply_info['email_addr'] = email_addr + apply_info['pub_key'] = pub_key + + apply_info +end + +options = OptionParser.new do |opts| + opts.banner = "Usage: answerback-mail.rb [--email email] [--ssh-pubkey pub_key] [--raw-email email_file]\n" + opts.banner += " -e or -f is required\n" + opts.banner += ' -s is optional when use -e' + + opts.separator '' + opts.separator 'options:' + + opts.on('-e email_addr', '--email email_addr', 'appoint email address') do |email_addr| + apply_info['email_addr'] = email_addr + end + + opts.on('-s pub_key', '--ssh-pubkey pub_key', 'ssh pub_key to enable keyless login') do |pub_key| + apply_info['pub_key'] = pub_key + end + + opts.on('-f email_file', '--raw-email email_file', 'email file') do |email_file| + format_info(email_file, apply_info) + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +options.parse!(ARGV) + +def build_message(email, acct_infos) message = <<~EMAIL_MESSAGE To: #{email} - Message-ID: #{message_id} Subject: jumper account is ready - Dear #{email} + Dear user: Thank you for joining us. You can use the following command to login the jumper server: login command: - ssh -p #{infos['jumper_port']} #{infos['account']}@#{infos['jumper_ip']} + ssh -p #{acct_infos['jumper_port']} #{acct_infos['account']}@#{acct_infos['jumper_ip']} account passwd: - account_password: #{infos['passwd']} + account_password: #{acct_infos['passwd']} regards compass-ci @@ -79,67 +95,26 @@ def build_message(email, message_id, infos) return message end -def email_addr(mail_content) - msg = 'not an applying account email' - - raise msg unless mail_content.subject =~ /apply ssh account/i - - email = mail_content.from.join(',') - - return email -end - -# def check_email_available(mail_content, email) -# oos_list = File.read('/c/upstream-repos/repo_list').split(/\n/) -# url = mail_content.body.decoded.split(/\n/).find { |line| line =~ /https?:\/\// } -# base_url = url.split('/')[0,5].join('/') -# message = 'The url is not in upstream repo_list' -# -# raise message unless oos_list.include? base_url -# -# url_fdback = %x(curl #{url}) -# email_index = url_fdback.index email -# -# message = 'No commit info found from the url for the email' -# raise message unless email_index -# end - -def email_message_id(mail_content) - message_id = mail_content.message_id - return message_id -end - -def pub_key_value(mail_content) - pub_key = mail_content.body.decoded.split(/\n/).find { |line| line =~ /ssh-rsa/ } - return pub_key -end - def account_info(pub_key) - account_info_str = %x(curl -XGET '#{JUMPER_IP}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") - account_info = JSON.parse account_info_str - - return account_info + account_info_str = if pub_key.nil? + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account') + else + %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d "pub_key: #{pub_key}") + end + JSON.parse account_info_str end -def send_account(mail_content) - email = email_addr(mail_content) - message_id = email_message_id(mail_content) - # check_email_available(mail_content, email) +def send_account(apply_info) + message = "No email address specified\n" + message += "use -e email_addr add a email address\n" + message += 'or use -f to add a mail file' + raise message if apply_info['email_addr'].nil? - pub_key = pub_key_value(mail_content) - acct_info = account_info(pub_key) + acct_info = account_info(apply_info['pub_key']) - message = build_message(email, message_id, acct_info) + message = build_message(apply_info['email_addr'], acct_info) %x(curl -XPOST '#{CRYSTAL_INTRANET}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -def read_mail_content(mail_file) - mail_content = Mail.read(mail_file) - - return mail_content -end - -mail_file = ARGV[0] -mail_content = read_mail_content(mail_file) -send_account(mail_content) +send_account(apply_info) -- 2.23.0

2 1