September 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci 1/7] container/mail-robot: mail-robot.rb
by Luan Shengde 10 Oct '20

10 Oct '20

mail robot for application email monitor mailbox check new added email file in new check new mail's subject return if unmatched subject call answerback-email apply account apply uuid write account/uuid etc. to es send uuid to user Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/mail-robot.rb | 40 ++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..f91b89d --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,40 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'answerback-email.rb' + +mail_inbox = '/Maildir/.compass-ci/new/' +mail_drafts = '/Maildir/.compass-ci/cur/' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |modified, added, _removed| + unless added.empty? + added.each do |mail_file| + begin + check_to_send_account(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + end + end + end + end + listener.start + sleep +end + +def check_to_send_account(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + subject = mail_content.subject + return unless subject =~ /apply account/i + + send_account(mail_content) + FileUtils.mv(mail_file, mail_drafts) +end + +monitor_dir(mail_inbox, mail_drafts) -- 2.23.0

3 4

[PATCH compass-ci 2/2] feat(dnsmasq): add two dhcp client types
by Yu Chuan 10 Oct '20

10 Oct '20

Two client types: - pxeclient-arm64 - pxeclient-x64 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/dnsmasq/dnsmasq.d/dnsmasq.conf | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/container/dnsmasq/dnsmasq.d/dnsmasq.conf b/container/dnsmasq/dnsmasq.d/dnsmasq.conf index a51b43e9e588..0ea37b9c1c27 100644 --- a/container/dnsmasq/dnsmasq.d/dnsmasq.conf +++ b/container/dnsmasq/dnsmasq.d/dnsmasq.conf @@ -19,9 +19,12 @@ dhcp-range=172.18.10.10,172.18.255.250,168h # br0 dhcp-match=set:ipxe,175 dhcp-match=set:pxeclient,60,PXEClient* +dhcp-match=set:pxeclient-arm64,93,11 +dhcp-match=set:pxeclient-x64,93,7 -dhcp-boot=tag:!ipxe,/tftpboot/ipxe/bin-arm64-efi/snp.efi dhcp-boot=tag:ipxe,boot.ipxe +dhcp-boot=tag:pxeclient-arm64,/tftpboot/ipxe/bin-arm64-efi/snp.efi +dhcp-boot=tag:pxeclient-x64,ipxe/bin-x86_64-efi/snp.efi log-queries log-facility=/var/log/dnsmasq/dnsmasq.log -- 2.23.0

2 2

[PATCH v4 compass-ci 7/7] container/mail-robot: mail-robot.md
by Luan Shengde 09 Oct '20

09 Oct '20

guidance file for mail-robot Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..5c42866 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +monitor received mails + check if the mail is for applying uuid/account + +## steps overview + +1. monitor mailbox dir + - check for new email file + +2. check email + - check email subject + - return if unmatched subject + - call answerback-email + - check email available + - check base_url for the oss_url in upstream-repos list + if not, it will raise "url not in upstream-repos list" error + - check if the email has commit for the url + if not, it will raise "no commit found" error + - check pub_key + - check if there is a pub_key in the email + if not, it will raise 'no pub_key' error + - apply ssh account + - apply account uuid + - write account info to es + - send uuid to user + - mv mail file to drafts + +3. continue the monitor + - after 1 and 2, continue to monitor the mailbox + +4. see the log + - use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

2 1

[PATCH v4 compass-ci 4/7] container/mail-robot: Dockerfile
by Luan Shengde 09 Oct '20

09 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..87c04e7 --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +FROM debian +MAINTAINER luanshd0525(a)163.com +ENV DEBIAN_FRONTEND noninteractive + +COPY conf/sources.list* /etc/apt/ + +RUN apt-get update && \ + apt-get install -y uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport && \ + ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ + echo 'Asia/Shanghai' > /etc/timezone +COPY mail-robot.rb /usr/local/bin/ +COPY answerback-email.rb /usr/local/bin/ +WORKDIR /usr/local/bin/ -- 2.23.0

2 1

[PATCH v4 compass-ci 3/7] container/mail-robot: build file
by Luan Shengde 09 Oct '20

09 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..79af1de --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ + +docker build -t mail-robot:latest . + -- 2.23.0

2 1

[PATCH v4 compass-ci 2/7] container/mail-robot: answerback-email.rb
by Luan Shengde 09 Oct '20

09 Oct '20

get email content for mail-robot.rb check email available check base url in upstream-repos check commit url valid check pub_key return if no pub_key assign account generate uuid write account/uuid to es send uuid to user Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/answerback-email.rb | 201 +++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100755 container/mail-robot/answerback-email.rb diff --git a/container/mail-robot/answerback-email.rb b/container/mail-robot/answerback-email.rb new file mode 100755 index 0000000..6f21358 --- /dev/null +++ b/container/mail-robot/answerback-email.rb @@ -0,0 +1,201 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +=begin + +repo_list: + all repos url list from upstream-repos.git + +API: +call graph: +read_mail_content +send_account_request + email_addr + get email address + email_message_id + get email message_id + check_email_available + check email available + pub_key_value + get pub key + uuid + get jumper server account + generate uuid + mail_message + uuid + submit steps + store account_info + call es_client + send_mail + send mail with mail_message + +the returned data for apply ssh account: +{ + "account" => "guest", + "passwd" => "Use pub_key to login", + "jumper_ip" => "10.10.10.10", + "jumper_port" => "10000" +} + +=end + +require 'json' +require 'mail' +require_relative '/compass-ci/lib/es_client.rb' + +def build_message(to, message_id, account_uuid) + message = <<~EMAIL_MESSAGE + To: #{to} + Message-ID: #{message_id} + Subject: Account Ready + + Dear #{to.split('@')[0]} + + Thank you for joining us. + You can use the following info to submit jobs: + + First add the info to: ~/.config/compass-ci/defaults/{USER}.yaml + + uuid: #{account_uuid} + lab: z9 + + Second: download lkp-tests and dependencies + git clone https://gitee.com_/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source {HOME}/.bashrc && source {HOME}/.bash_profile + + Third: submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + how to write job yaml + submit jobs/netperf.yaml + + Repalce the fields {...} with your local environment variable. + + regards + compass-ci + EMAIL_MESSAGE + + return message +end + +def email_addr(mail_content) + email = mail_content.from[0] + + return email +end + +def url_addr(mail_content) + url_line = mail_content.part[0].body.decoded.split(/\n/).find { |line| line =~ %r{https?://} } + message = 'No commit url found' + + raise message unless url_line + + url = url_line.split[-1] + + return url +end + +def base_url_addr(url) + url_a = url.split('/') + url_a.pop 2 + base_url = url_a.join('/') + + return base_url +end + +def check_email_available(mail_content, email) + url = url_addr(mail_content) + + base_url = base_url_addr(url) + + upstream_dir = '/upstream-repos' + check_url_in_upstream_repos(upstream_dir, base_url) + check_url_commit(url, email) + + return url +end + +def check_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + message = 'The url is not in upstream-repo list' + + raise StandardError, message, base_url if match_out.empty? +end + +def check_url_commit(url, email) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index email + + message = 'No email info found from the url feedback.' + raise StandardError, message, url if email_index.nil? +end + +def email_message_id(mail_content) + message_id = mail_content.message_id + return message_id +end + +def pub_key_value(mail_content) + pub_key = mail_content.part[1].body.decoded.split(/\r|\n/).join if mail_content.part[1].filename == 'id_rsa.pub' + + message = 'No pub_key fount' + + raise message unless pub_key + + return pub_key +end + +def account_info(pub_key) + account_info_str = %x(curl -XGET "#{ENV['JUMPER_IP']}:#{ENV['JUMPER_PORT']}/assign_account" -d "pub_key: #{pub_key}") + account_info = JSON.parse account_info_str + + message = 'No available jumper account returned.' + raise StandardError, message, account_info if account_info['account'].empty? + + return account_info +end + +def setup_account_uuid + account_uuid = %x(uuidgen).chomp + + return account_uuid +end + +def send_account(mail_content) + email = email_addr(mail_content) + message_id = email_message_id(mail_content) + url = check_email_available(mail_content, email) + pub_key = pub_key_value(mail_content) + acct_infos = account_info(pub_key) + account_uuid = setup_account_uuid + + store_account_info(email, account_uuid, acct_infos, url) + + message = build_message(email, message_id, account_uuid) + send_mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp + + %x(curl -XPOST "#{send_mail_server}:#{ENV['SEND_MAIL_PORT']}/send_mail_text" -d "#{message}") +end + +def store_account_info(to, account_uuid, acct_infos, url) + account_uuid_info = { + 'email' => to, + 'name' => to.split('@')[0], + 'uuid' => account_uuid, + 'user' => acct_infos['account'], + 'my_commit_url' => url + } + + es = ESClient.new(index: 'accounts') + es.put_source_by_id(to, account_uuid_info) +end + +def read_mail_content(mail_file) + mail_content = Mail.read(mail_file) + + return mail_content +end -- 2.23.0

2 1

[PATCH compass-ci] sched: refactor get_boot_ipxe to job class
by Cao Xueliang 09 Oct '20

09 Oct '20

According os_mount to add initrds full path and add initrds file name in root append. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 89 +++++++++++++++++++++++++++++++++----- src/lib/sched.cr | 54 ++++------------------- src/scheduler/constants.cr | 4 ++ 3 files changed, 89 insertions(+), 58 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 53a560c..bedfec0 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -72,10 +72,12 @@ class Job lab initrd_pkg initrd_deps + initrds result_root access_key access_key_file lkp_initrd_user + kernel kernel_append_root docker_image ) @@ -123,8 +125,10 @@ class Job set_result_root() set_result_service() set_os_mount() - set_kernel_append_root() set_pp_initrd() + set_initrds() + set_kernel() + set_kernel_append_root() set_lkp_server() end @@ -228,23 +232,83 @@ class Job end end - private def set_kernel_append_root - os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") - lkp_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}/initrd.lkp") - lkp_basename = File.basename(lkp_real_path) + private def vmlinuz + if os_mount == "initramfs" + return "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/vmlinuz")}" + else + return "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" + end + end + + private def set_kernel + self["kernel"] = "kernel #{vmlinuz()}" + end + + private def common_initrds + temp_initrds = [] of String + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz")}" + temp_initrds << "#{SCHED_HTTP_PREFIX}/job_initrd_tmpfs/#{id}/job.cgz" + + return temp_initrds + end + + private def initramfs_initrds + temp_initrds = [] of String + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + + temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? + temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? + + return temp_initrds + end + + private def nfs_cifs_initrds + temp_initrds = [] of String + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/initrd.lkp")}" + + return temp_initrds + end + + private def set_initrds + temp_initrds = [] of String - current_basename = "" if "#{os_mount}" == "initramfs" - current_real_path = JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current") - current_basename = File.basename(current_real_path) + temp_initrds.concat(initramfs_initrds()) + else + temp_initrds.concat(nfs_cifs_initrds()) + end + + temp_initrds.concat(common_initrds()) + + self["initrds"] = temp_initrds.join(" ") + end + + private def initrd_basename + basenames = "" + + initrds.split(/ /).each do |initrd| + basenames += "initrd=#{File.basename(initrd)} " end + return basenames + end + + private def set_kernel_append_root + os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") + fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} initrd=#{lkp_basename}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrd_basename()}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr initrd=#{lkp_basename}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 initrd=#{current_basename}", - "container" => "" + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrd_basename()}", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrd_basename()}", + "container" => "", } self["kernel_append_root"] = fs2root[os_mount] end @@ -270,6 +334,7 @@ class Job end end end + self["initrd_deps"] = initrd_deps_arr.join(" ") self["initrd_pkg"] = initrd_pkg_arr.join(" ") end diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 309df04..990641c 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -488,61 +488,23 @@ class Sched return returned end - private def get_pp_initrd(job : Job) - initrd_deps = "" - initrd_pkg = "" - if job.os_mount == "initramfs" - initrd_deps += job.initrd_deps.split.join() { |item| "initrd #{item}\n" } - initrd_pkg += job.initrd_pkg.split.join() { |item| "initrd #{item}\n" } + private def get_initrds(job_initrds) + initrds = "" + job_initrds.each do |initrd| + initrds += "initrd #{initrd}\n" end - return initrd_deps, initrd_pkg + return initrds end private def get_boot_ipxe(job : Job) - initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" - - initrd_deps, initrd_pkg = get_pp_initrd(job) - - initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" - sched_http_prefix = "http://#{SCHED_HOST}:#{SCHED_PORT}" - os_http_prefix = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" - response = "#!ipxe\n\n" - if job.os_mount == "initramfs" - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/current")}\n" - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/run-ipconfig.cgz")}\n" - else - response += "initrd #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/initrd.lkp")}\n" - end - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/#{initrd_lkp_cgz}")}\n" - response += "initrd #{sched_http_prefix}/job_initrd_tmpfs/#{job.id}/job.cgz\n" - response += initrd_deps - response += initrd_pkg - if job.os_mount == "initramfs" - response += "kernel #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" - else - response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" - end + response += get_initrds(job.initrds.split(/ /)) + response += job.kernel response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" response += " #{job.kernel_append_root}" response += add_kernel_console_param(job.os_arch) - if job.os_mount == "initramfs" - response += " initrd=#{initrd_lkp_cgz} initrd=job.cgz" - job.initrd_deps.split.each do |initrd_dep| - response += " initrd=#{File.basename(initrd_dep)}" - end - response += " initrd=#{File.basename(JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/run-ipconfig.cgz"))}\n" - else - response += " initrd=#{initrd_lkp_cgz} initrd=job.cgz\n" - end - response += "boot\n" + response += "\nboot\n" puts %({"job_id": "#{job.id}", "job_state": "boot"}) return response diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 7ce49a1..839c241 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -24,4 +24,8 @@ OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i3 SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" +INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" +OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" +SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" + DEMO_JOB = %({"suite":"pixz","testcase":"pixz","category":"benchmark","nr_threads":1,"pixz":null,"job_origin":"jobs/pixz.yaml","testbox":"wfg-e595","arch":"x86_64","tbox_group":"wfg-e595","id":"100","kmsg":null,"boot-time":null,"uptime":null,"iostat":null,"heartbeat":null,"vmstat":null,"numa-numastat":null,"numa-vmstat":null,"numa-meminfo":null,"proc-vmstat":null,"proc-stat":null,"meminfo":null,"slabinfo":null,"interrupts":null,"kconfig":"x86_64-rhel-7.6","compiler":"gcc-7"}) -- 2.23.0

3 6

[PATCH v2 lkp-tests] lib: add transfer_port.sh
by Zhang Yale 09 Oct '20

09 Oct '20

Provide a login port for user, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- lib/transfer_port.sh | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 lib/transfer_port.sh diff --git a/lib/transfer_port.sh b/lib/transfer_port.sh new file mode 100755 index 00000000..61fb67b4 --- /dev/null +++ b/lib/transfer_port.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# uuid + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 + +get_port() +{ + [ -n "$uuid" ] || return + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%10+11333)) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] || continue + echo "$port" + break + done +} + -- 2.23.0

4 8

[PATCH v4 compass-ci 2/2] sparrow: open the masquerade at public zone
by Hu Xuejiao 09 Oct '20

09 Oct '20

Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/2-network/iptables | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 7183145..0fd82e5 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -28,3 +28,4 @@ DOCKER0_SUBNET=172.17.0.0/16 firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" +firewall-cmd --zone=public --add-masquerade -- 2.23.0

3 2

[PATCH v4 compass-ci 1/2] fix(sparrow): process will exit if firewalld.service was not installed
by Hu Xuejiao 09 Oct '20

09 Oct '20

When firewalld is inactive, return code of 'systemctl status firewalld' will be 3, and install-tiny has set -e, so this will cause this process exit. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/2-network/iptables | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..7183145 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -12,15 +12,19 @@ iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE -systemctl status firewalld | grep -q "running" && { +command -v firewall-cmd > /dev/null || { + echo "firewalld.service could not be found" + exit 0 +} - DOCKER0_IFACE=docker0 - DOCKER0_SUBNET=172.17.0.0/16 +systemctl start firewalld +[ "$(systemctl is-active firewalld)" == "active" ] || { + echo "start firewalld.service failed" + exit 0 +} - iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE - iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE +DOCKER0_SUBNET=172.17.0.0/16 - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -} +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -- 2.23.0

2 1

[PATCH lkp-tests 2/2] hosts: add vm-aarch64-*
by Wu Fengguang 30 Sep '20

30 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- hosts/vm-aarch64-1p1g | 6 ++++++ hosts/vm-aarch64-2p1g | 6 ++++++ hosts/vm-aarch64-2p4g | 6 ++++++ hosts/vm-aarch64-2p8g | 6 ++++++ hosts/vm-pxe-aarch64-1p1g | 6 ++++++ hosts/vm-pxe-aarch64-2p1g | 6 ++++++ hosts/vm-pxe-aarch64-2p4g | 6 ++++++ hosts/vm-pxe-aarch64-2p8g | 6 ++++++ 8 files changed, 48 insertions(+) create mode 100644 hosts/vm-aarch64-1p1g create mode 100644 hosts/vm-aarch64-2p1g create mode 100644 hosts/vm-aarch64-2p4g create mode 100644 hosts/vm-aarch64-2p8g create mode 100644 hosts/vm-pxe-aarch64-1p1g create mode 100644 hosts/vm-pxe-aarch64-2p1g create mode 100644 hosts/vm-pxe-aarch64-2p4g create mode 100644 hosts/vm-pxe-aarch64-2p8g diff --git a/hosts/vm-aarch64-1p1g b/hosts/vm-aarch64-1p1g new file mode 100644 index 00000000..b7a53601 --- /dev/null +++ b/hosts/vm-aarch64-1p1g @@ -0,0 +1,6 @@ +provider: qemu +template: kvm +arch: aarch64 +nr_node: 1 +nr_cpu: 1 +memory: 1G diff --git a/hosts/vm-aarch64-2p1g b/hosts/vm-aarch64-2p1g new file mode 100644 index 00000000..08d0c25c --- /dev/null +++ b/hosts/vm-aarch64-2p1g @@ -0,0 +1,6 @@ +provider: qemu +template: kvm +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 1G diff --git a/hosts/vm-aarch64-2p4g b/hosts/vm-aarch64-2p4g new file mode 100644 index 00000000..d57774b9 --- /dev/null +++ b/hosts/vm-aarch64-2p4g @@ -0,0 +1,6 @@ +provider: qemu +template: kvm +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 4G diff --git a/hosts/vm-aarch64-2p8g b/hosts/vm-aarch64-2p8g new file mode 100644 index 00000000..40fe2716 --- /dev/null +++ b/hosts/vm-aarch64-2p8g @@ -0,0 +1,6 @@ +provider: qemu +template: kvm +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 8G diff --git a/hosts/vm-pxe-aarch64-1p1g b/hosts/vm-pxe-aarch64-1p1g new file mode 100644 index 00000000..554e8fb1 --- /dev/null +++ b/hosts/vm-pxe-aarch64-1p1g @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +arch: aarch64 +nr_node: 1 +nr_cpu: 1 +memory: 1G diff --git a/hosts/vm-pxe-aarch64-2p1g b/hosts/vm-pxe-aarch64-2p1g new file mode 100644 index 00000000..27b8438b --- /dev/null +++ b/hosts/vm-pxe-aarch64-2p1g @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 1G diff --git a/hosts/vm-pxe-aarch64-2p4g b/hosts/vm-pxe-aarch64-2p4g new file mode 100644 index 00000000..c31dd953 --- /dev/null +++ b/hosts/vm-pxe-aarch64-2p4g @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 4G diff --git a/hosts/vm-pxe-aarch64-2p8g b/hosts/vm-pxe-aarch64-2p8g new file mode 100644 index 00000000..3687d460 --- /dev/null +++ b/hosts/vm-pxe-aarch64-2p8g @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 8G -- 2.23.0

1 1

[PATCH lkp-tests 1/2] hosts: rename dc-* to dc-aarch64-*
by Wu Fengguang 30 Sep '20

30 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- hosts/{dc-1g => dc-aarch64-1g} | 0 hosts/{dc-2g => dc-aarch64-2g} | 0 hosts/{dc-4g => dc-aarch64-4g} | 0 hosts/{dc-8g => dc-aarch64-8g} | 0 4 files changed, 0 insertions(+), 0 deletions(-) rename hosts/{dc-1g => dc-aarch64-1g} (100%) rename hosts/{dc-2g => dc-aarch64-2g} (100%) rename hosts/{dc-4g => dc-aarch64-4g} (100%) rename hosts/{dc-8g => dc-aarch64-8g} (100%) diff --git a/hosts/dc-1g b/hosts/dc-aarch64-1g similarity index 100% rename from hosts/dc-1g rename to hosts/dc-aarch64-1g diff --git a/hosts/dc-2g b/hosts/dc-aarch64-2g similarity index 100% rename from hosts/dc-2g rename to hosts/dc-aarch64-2g diff --git a/hosts/dc-4g b/hosts/dc-aarch64-4g similarity index 100% rename from hosts/dc-4g rename to hosts/dc-aarch64-4g diff --git a/hosts/dc-8g b/hosts/dc-aarch64-8g similarity index 100% rename from hosts/dc-8g rename to hosts/dc-aarch64-8g -- 2.23.0

1 0

[PATCH v4 compass-ci] container: consider the situation that GITCACHE_HOST is undefined
by Hu Xuejiao 30 Sep '20

30 Sep '20

- "[ -n $GITCACHE_HOST ] && ... " if $GITCACHE_HOST is none, process will exit 1 - "[ -z $GITCACHE_HOST ] || ... " if $GITCACHE_HOST is none, process will keep executing. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- container/crystal-shards/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/crystal-shards/Dockerfile b/container/crystal-shards/Dockerfile index 0c5713c..bdc5b66 100644 --- a/container/crystal-shards/Dockerfile +++ b/container/crystal-shards/Dockerfile @@ -7,7 +7,7 @@ MAINTAINER chief <taxcom(a)tom.com> ARG GITCACHE_HOST ARG GITCACHE_PORT -RUN [ -n "$GITCACHE_HOST" ] && echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig +RUN [ -z "$GITCACHE_HOST" ] || echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig copy shard.yml /usr/share/crystal/app/shard.yml -- 2.23.0

1 0

[PATCH v2 lkp-tests] sshd: send emails to provide login services for users
by Zhang Yale 30 Sep '20

30 Sep '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 61 +++++++++++++++++++++++++++++++++++------------------ 1 file changed, 41 insertions(+), 20 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..f921a7e5 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,10 @@ #!/bin/sh # pub_key # email +# uuid + +. $LKP_SRC/lib/transfer_port.sh +. $LKP_SRC/lib/http.sh run_ssh() { @@ -11,40 +15,57 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh +Subject: [NOTIFY Compass-ci] $testbox ready to use -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: +Dear $user_name: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: - Login: - ssh root@$PUB_IP - Due time: - $deadline + login: ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: $deadline - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch +Regards +Compass-ci +" +} + +data_failure() +{ + data="To: $email +Subject: [NOTIFY Compass-ci] Applying $testbox failed + +Dear $user_name: + Sorry to inform you that your application failed. + You may need to wait for a while and then thy again. Regards -compass-ci +Compass-ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$email" ] && [ -n "$uuid" ] || return 0 + user_name=$(echo "$email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +port=$(get_port) +report_ssh_port $port run_email -- 2.23.0

1 0

[PATCH v2 compass-ci] container/srv-http forbid access parent directory of job result_root
by Lu Weitao 30 Sep '20

30 Sep '20

by forbid access level 1, 2, 4 directory of /srv/result Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 53833eb..2bf55b6 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -19,4 +19,8 @@ server { add_header Content-Type text/plain; } } + + location ~* ^/result/(([^/]+)?/?|[^/]+/[^/]+/[^/]+/?)$ { + deny all; + } } -- 2.23.0

1 0

[PATCH v2 compass-ci] fix: sched: submit idle jobs
by Ren Wen 30 Sep '20

30 Sep '20

1) submit all idles jobs once use '*' -> submit each idle job once until all done. 2) add two fields into job if job's filename is 'multi-qemu.yaml'. 3) add field 'idle_job' for idle jobs, value is always 'true'. Two extra fields are used for: monitoring logs of virtual machines started by physical machine. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 15f5c9e..984c7db 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -467,10 +467,20 @@ class Sched end def auto_submit_idle_job(tbox_group) - full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" - Jobfile::Operate.auto_submit_job( - full_path_patterns, - "testbox: #{tbox_group}") if Dir.glob(full_path_patterns).size > 0 + idle_job_path = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/" + + if Dir.exists?(idle_job_path) + jobs = Dir[idle_job_path + "*.yaml"] + + jobs.each do |job| + extra_job_fields = ["idle_job=true"] + if Path[job].basename == "multi-qemu.yaml" + extra_job_fields += ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", + "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}"] + end + Jobfile::Operate.auto_submit_job(job, "testbox: #{tbox_group}", extra_job_fields) + end + end end private def add_kernel_console_param(arch_tmp) @@ -520,7 +530,7 @@ class Sched "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" else response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" end response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" -- 2.23.0

3 7

[PATCH v3 compass-ci] src/extract-stats deal with potential exceptions
by Lu Weitao 30 Sep '20

30 Sep '20

background: extract-stats process is at risk of be interrupted, due to other dependent services(task-queue, es_client) may not working properly solution: 1. use begin ... rescue to handle potential exceptions - when ES and task-queue not working properly, sleep 10s and visit again - when "result_root" does not exist, skip and remove task from task-queue 2. refactor for consume_sched_queue() of stats_worker.cr Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 60 ++++++++++++++++++------------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index fcba6af..2af07d4 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -18,40 +18,50 @@ class StatsWorker def consume_sched_queue(queue_path : String) loop do begin - response = @tq.consume_task(queue_path) + job_id = get_job_id(queue_path) + if job_id # will consume the job by post-processing + job = @es.get_job_content(job_id) + result_root = job["result_root"]? + result_post_processing(job_id, result_root.to_s, queue_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + end rescue e STDERR.puts e.message - # incase of many error message when task-queue is busy + # incase of many error message when task-queue, ES does not work sleep(10) - next end + end + end - if response[0] == 200 - job_id = JSON.parse(response[1].to_json)["id"] - job = @es.get_job_content(job_id.to_s) - result_root = job["result_root"] - begin - # extract stats.json - system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" - # storage stats to job in es - store_stats_es(result_root.to_s, job_id.to_s) if result_root - # send mail to submitter for job results - system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" - rescue e - STDERR.puts e.message - next - end - - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - else - sleep(60) - end + # get job_id from task-queue + def get_job_id(queue_path : String) + response = @tq.consume_task(queue_path) + if response[0] == 200 + JSON.parse(response[1].to_json)["id"].to_s + else + # will sleep 60s if no task in task-queue + sleep(60) + nil end end - def store_stats_es(result_root : String, job_id : String) + def result_post_processing(job_id : String, result_root : String, queue_path : String) + return nil unless result_root && File.exists?(result_root) + + # extract stats.json + system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" + # storage stats to job in es + store_stats_es(result_root, job_id, queue_path) + # send mail to submitter for job results + system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" + end + + def store_stats_es(result_root : String, job_id : String, queue_path : String) stats_path = "#{result_root}/stats.json" - raise "#{stats_path} file not exists." unless File.exists?(stats_path) + unless File.exists?(stats_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + raise "#{stats_path} file not exists." + end stats = File.open(stats_path) do |file| JSON.parse(file) -- 2.23.0

1 0

[PATCH compass-ci] container: crystal-shards: fix build error
by Liu Yinsi 30 Sep '20

30 Sep '20

build crystal-shards depends gitcache, When build crystal-shards, error message as follows: Step 8/12 : RUN shards ---> Running in be90213caa48 Resolving dependencies Fetching https://github.com/kemalcr/kemal.git Failed to clone https://github.com/kemalcr/kemal.git The command '/bin/sh -c shards' returned a non-zero code: 1 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/crystal-shards/build-depends | 1 + 1 file changed, 1 insertion(+) diff --git a/container/crystal-shards/build-depends b/container/crystal-shards/build-depends index 6627b6f..6eed7ec 100755 --- a/container/crystal-shards/build-depends +++ b/container/crystal-shards/build-depends @@ -1 +1,2 @@ crystal-base +gitcache -- 2.23.0

1 0

[PATCH compass-ci] sparrow/install-tiny: fix process exit
by Liu Yinsi 30 Sep '20

30 Sep '20

-e will cause process exit when return code is not 0, only use when debugging. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-tiny | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index 1c0b5d2..a291549 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -1,4 +1,4 @@ -#!/bin/bash -e +#!/bin/bash # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # For your reference only. -- 2.23.0

1 0

[PATCH compass-ci] fix: sched: add idle job to idle queue
by Ren Wen 30 Sep '20

30 Sep '20

Add idle job to idle queue, then idle machine find job in idle queue. For example: when idle job under "allot/idle/vm-hi1620-2p8g", queue should be: "queues/sched/vm-hi1620-2p8g/idle" Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 309df04..47a2e54 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -266,7 +266,7 @@ class Sched }] unless tbox_group # only single job will has "idle job" and "execute rate limiter" - if %<job["job_origin"]>.includes?("allot/idle/") + unless job["idle_job"].empty? tbox_group = "#{tbox_group}/idle" else tbox_group += "#{job.get_uuid_tag}" -- 2.23.0

1 0

[PATCH lkp-tests 1/3] sshd: add build channel
by Zhang Yale 30 Sep '20

30 Sep '20

Auto send email to remind user to login with port/domain_name and send port to sched Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 68 ++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 47 insertions(+), 21 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..d1fcae52 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,12 @@ #!/bin/sh # pub_key # email +# uuid + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 run_ssh() { @@ -14,37 +20,57 @@ run_ssh() compose_email() { deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") - data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh - -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: - - Login: - ssh root@$PUB_IP - Due time: - $deadline - - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch + if [ -n $port ]; then + data="To: $email +Subject: [NOTIFY compass-ci] $testbox ready to use + +Dear $user_name: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + Please use the following command to login: + + ssh root(a)api.compass-ci.openeuler.org -p $port + +Regards +compassi-ci +" + else + data="To: $email +Subject: [NOTIFY compass-ci] Applying $testbox failed + +Dear $user_name: + Sorry to inform you than your application failed. + + You may need to wait sometime and then have a second try. Regards compass-ci " + fi +} + +get_port() +{ + [ -n "$uuid" ] || return + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%10+11333)) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -Nf -R $port:$PUB_IP:22 sshr@"$sshr_ip" -p "$sshr_port" > out 2>&1 + [ -z "`grep -i "error" out`" ] || continue + return "$port" + break + done } run_email() { - [ -n "$email" ] || return 0 + [ -n "$email" ] && [ -n "$uuid" ] || return 0 + user_name=$(echo "$email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +get_port +report_ssh_port $port run_email -- 2.23.0

1 1

[PATCH lkp-tests 3/3] lib: add transfer_port.sh
by Zhang Yale 30 Sep '20

30 Sep '20

Provide a login port for user, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- lib/transfer_port.sh | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 lib/transfer_port.sh diff --git a/lib/transfer_port.sh b/lib/transfer_port.sh new file mode 100755 index 00000000..61fb67b4 --- /dev/null +++ b/lib/transfer_port.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# uuid + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 + +get_port() +{ + [ -n "$uuid" ] || return + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%10+11333)) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] || continue + echo "$port" + break + done +} + -- 2.23.0

1 0

[PATCH lkp-tests 2/3] sshd: send emails to provide login services for users
by Zhang Yale 30 Sep '20

30 Sep '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- daemon/sshd | 49 ++++++++++++++++++++++--------------------------- 1 file changed, 22 insertions(+), 27 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index d1fcae52..35809ffe 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -3,11 +3,9 @@ # email # uuid +. $LKP_SRC/lib/transfer_port.sh . $LKP_SRC/lib/http.sh -sshr_ip=$LKP_SERVER -sshr_port=5050 - run_ssh() { [ -n "$pub_key" ] || return @@ -17,49 +15,46 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") - if [ -n $port ]; then - data="To: $email + data="To: $email Subject: [NOTIFY compass-ci] $testbox ready to use Dear $user_name: Thanks for your participation in software ecosystem! According to your application, $testbox has been provisioned. - Please use the following command to login: + The datails are as follows: - ssh root(a)api.compass-ci.openeuler.org -p $port + login: ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: $deadline Regards -compassi-ci +compass-ci " - else - data="To: $email +} + +data_failure() +{ + data="To: $email Subject: [NOTIFY compass-ci] Applying $testbox failed Dear $user_name: - Sorry to inform you than your application failed. - - You may need to wait sometime and then have a second try. + Sorry to inform you that your application failed. + You may need to wait for a while and then have a second try. Regards compass-ci " - fi } -get_port() +compose_email() { - [ -n "$uuid" ] || return - for i in $(seq 1 10) - do - port=$(($(date +%s%N)%10+11333)) - ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -Nf -R $port:$PUB_IP:22 sshr@"$sshr_ip" -p "$sshr_port" > out 2>&1 - [ -z "`grep -i "error" out`" ] || continue - return "$port" - break - done + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi } run_email() @@ -71,6 +66,6 @@ run_email() } run_ssh -get_port +port=$(get_port) report_ssh_port $port run_email -- 2.23.0

1 0

[PATCH lkp-tests] fix: run-ipconfug: delete the ntpdate
by Wang Chenglong 30 Sep '20

30 Sep '20

the run_ntpdate function in bootstrap.sh here run ntpdate command is unnecessary Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- bin/run-ipconfig | 1 - 1 file changed, 1 deletion(-) diff --git a/bin/run-ipconfig b/bin/run-ipconfig index ad2a9808..0f80feaf 100755 --- a/bin/run-ipconfig +++ b/bin/run-ipconfig @@ -20,4 +20,3 @@ read_ip() [ -n "$IP" ] || read_ip configure_networking -ntpdate -u ntp.api.bz -- 2.23.0

1 0

[PATCH lkp-tests] fix: lib/bootstrap.sh: ntpdate from $LKP_SERVER
by Wang Chenglong 30 Sep '20

30 Sep '20

befor: [ "$LKP_SERVER"=inn ] || return ntpdate -b $LKP_SERVER the ntpdate command can't run after: [ -n "$LKP_SERVER" ] || return ntpdate -b $LKP_SERVER now, if $LKP_SERVER is ok, it can synchronised the time Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/bootstrap.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index f8a98de6..0be7ed3b 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -188,7 +188,7 @@ clearlinux_timesync() run_ntpdate() { [ -z "$NO_NETWORK" ] || return - [ "$LKP_SERVER" = inn ] || return + [ -n "$LKP_SERVER" ] || return is_clearlinux && clearlinux_timesync && return [ -x '/usr/sbin/ntpdate' ] || return -- 2.23.0

1 0