September 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci 1/7] container/mail-robot: mail-robot.rb
by Luan Shengde 10 Oct '20

10 Oct '20

mail robot for application email monitor mailbox check new added email file in new check new mail's subject return if unmatched subject call answerback-email apply account apply uuid write account/uuid etc. to es send uuid to user Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/mail-robot.rb | 40 ++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..f91b89d --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,40 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'answerback-email.rb' + +mail_inbox = '/Maildir/.compass-ci/new/' +mail_drafts = '/Maildir/.compass-ci/cur/' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |modified, added, _removed| + unless added.empty? + added.each do |mail_file| + begin + check_to_send_account(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + end + end + end + end + listener.start + sleep +end + +def check_to_send_account(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + subject = mail_content.subject + return unless subject =~ /apply account/i + + send_account(mail_content) + FileUtils.mv(mail_file, mail_drafts) +end + +monitor_dir(mail_inbox, mail_drafts) -- 2.23.0

3 4

[PATCH compass-ci 2/2] feat(dnsmasq): add two dhcp client types
by Yu Chuan 10 Oct '20

10 Oct '20

Two client types: - pxeclient-arm64 - pxeclient-x64 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/dnsmasq/dnsmasq.d/dnsmasq.conf | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/container/dnsmasq/dnsmasq.d/dnsmasq.conf b/container/dnsmasq/dnsmasq.d/dnsmasq.conf index a51b43e9e588..0ea37b9c1c27 100644 --- a/container/dnsmasq/dnsmasq.d/dnsmasq.conf +++ b/container/dnsmasq/dnsmasq.d/dnsmasq.conf @@ -19,9 +19,12 @@ dhcp-range=172.18.10.10,172.18.255.250,168h # br0 dhcp-match=set:ipxe,175 dhcp-match=set:pxeclient,60,PXEClient* +dhcp-match=set:pxeclient-arm64,93,11 +dhcp-match=set:pxeclient-x64,93,7 -dhcp-boot=tag:!ipxe,/tftpboot/ipxe/bin-arm64-efi/snp.efi dhcp-boot=tag:ipxe,boot.ipxe +dhcp-boot=tag:pxeclient-arm64,/tftpboot/ipxe/bin-arm64-efi/snp.efi +dhcp-boot=tag:pxeclient-x64,ipxe/bin-x86_64-efi/snp.efi log-queries log-facility=/var/log/dnsmasq/dnsmasq.log -- 2.23.0

2 2

[PATCH v4 compass-ci 7/7] container/mail-robot: mail-robot.md
by Luan Shengde 09 Oct '20

09 Oct '20

guidance file for mail-robot Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..5c42866 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +monitor received mails + check if the mail is for applying uuid/account + +## steps overview + +1. monitor mailbox dir + - check for new email file + +2. check email + - check email subject + - return if unmatched subject + - call answerback-email + - check email available + - check base_url for the oss_url in upstream-repos list + if not, it will raise "url not in upstream-repos list" error + - check if the email has commit for the url + if not, it will raise "no commit found" error + - check pub_key + - check if there is a pub_key in the email + if not, it will raise 'no pub_key' error + - apply ssh account + - apply account uuid + - write account info to es + - send uuid to user + - mv mail file to drafts + +3. continue the monitor + - after 1 and 2, continue to monitor the mailbox + +4. see the log + - use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

2 1

[PATCH v4 compass-ci 4/7] container/mail-robot: Dockerfile
by Luan Shengde 09 Oct '20

09 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..87c04e7 --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +FROM debian +MAINTAINER luanshd0525(a)163.com +ENV DEBIAN_FRONTEND noninteractive + +COPY conf/sources.list* /etc/apt/ + +RUN apt-get update && \ + apt-get install -y uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport && \ + ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ + echo 'Asia/Shanghai' > /etc/timezone +COPY mail-robot.rb /usr/local/bin/ +COPY answerback-email.rb /usr/local/bin/ +WORKDIR /usr/local/bin/ -- 2.23.0

2 1

[PATCH v4 compass-ci 3/7] container/mail-robot: build file
by Luan Shengde 09 Oct '20

09 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..79af1de --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ + +docker build -t mail-robot:latest . + -- 2.23.0

2 1

[PATCH v4 compass-ci 2/7] container/mail-robot: answerback-email.rb
by Luan Shengde 09 Oct '20

09 Oct '20

get email content for mail-robot.rb check email available check base url in upstream-repos check commit url valid check pub_key return if no pub_key assign account generate uuid write account/uuid to es send uuid to user Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/answerback-email.rb | 201 +++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100755 container/mail-robot/answerback-email.rb diff --git a/container/mail-robot/answerback-email.rb b/container/mail-robot/answerback-email.rb new file mode 100755 index 0000000..6f21358 --- /dev/null +++ b/container/mail-robot/answerback-email.rb @@ -0,0 +1,201 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +=begin + +repo_list: + all repos url list from upstream-repos.git + +API: +call graph: +read_mail_content +send_account_request + email_addr + get email address + email_message_id + get email message_id + check_email_available + check email available + pub_key_value + get pub key + uuid + get jumper server account + generate uuid + mail_message + uuid + submit steps + store account_info + call es_client + send_mail + send mail with mail_message + +the returned data for apply ssh account: +{ + "account" => "guest", + "passwd" => "Use pub_key to login", + "jumper_ip" => "10.10.10.10", + "jumper_port" => "10000" +} + +=end + +require 'json' +require 'mail' +require_relative '/compass-ci/lib/es_client.rb' + +def build_message(to, message_id, account_uuid) + message = <<~EMAIL_MESSAGE + To: #{to} + Message-ID: #{message_id} + Subject: Account Ready + + Dear #{to.split('@')[0]} + + Thank you for joining us. + You can use the following info to submit jobs: + + First add the info to: ~/.config/compass-ci/defaults/{USER}.yaml + + uuid: #{account_uuid} + lab: z9 + + Second: download lkp-tests and dependencies + git clone https://gitee.com_/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source {HOME}/.bashrc && source {HOME}/.bash_profile + + Third: submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + how to write job yaml + submit jobs/netperf.yaml + + Repalce the fields {...} with your local environment variable. + + regards + compass-ci + EMAIL_MESSAGE + + return message +end + +def email_addr(mail_content) + email = mail_content.from[0] + + return email +end + +def url_addr(mail_content) + url_line = mail_content.part[0].body.decoded.split(/\n/).find { |line| line =~ %r{https?://} } + message = 'No commit url found' + + raise message unless url_line + + url = url_line.split[-1] + + return url +end + +def base_url_addr(url) + url_a = url.split('/') + url_a.pop 2 + base_url = url_a.join('/') + + return base_url +end + +def check_email_available(mail_content, email) + url = url_addr(mail_content) + + base_url = base_url_addr(url) + + upstream_dir = '/upstream-repos' + check_url_in_upstream_repos(upstream_dir, base_url) + check_url_commit(url, email) + + return url +end + +def check_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + message = 'The url is not in upstream-repo list' + + raise StandardError, message, base_url if match_out.empty? +end + +def check_url_commit(url, email) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index email + + message = 'No email info found from the url feedback.' + raise StandardError, message, url if email_index.nil? +end + +def email_message_id(mail_content) + message_id = mail_content.message_id + return message_id +end + +def pub_key_value(mail_content) + pub_key = mail_content.part[1].body.decoded.split(/\r|\n/).join if mail_content.part[1].filename == 'id_rsa.pub' + + message = 'No pub_key fount' + + raise message unless pub_key + + return pub_key +end + +def account_info(pub_key) + account_info_str = %x(curl -XGET "#{ENV['JUMPER_IP']}:#{ENV['JUMPER_PORT']}/assign_account" -d "pub_key: #{pub_key}") + account_info = JSON.parse account_info_str + + message = 'No available jumper account returned.' + raise StandardError, message, account_info if account_info['account'].empty? + + return account_info +end + +def setup_account_uuid + account_uuid = %x(uuidgen).chomp + + return account_uuid +end + +def send_account(mail_content) + email = email_addr(mail_content) + message_id = email_message_id(mail_content) + url = check_email_available(mail_content, email) + pub_key = pub_key_value(mail_content) + acct_infos = account_info(pub_key) + account_uuid = setup_account_uuid + + store_account_info(email, account_uuid, acct_infos, url) + + message = build_message(email, message_id, account_uuid) + send_mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp + + %x(curl -XPOST "#{send_mail_server}:#{ENV['SEND_MAIL_PORT']}/send_mail_text" -d "#{message}") +end + +def store_account_info(to, account_uuid, acct_infos, url) + account_uuid_info = { + 'email' => to, + 'name' => to.split('@')[0], + 'uuid' => account_uuid, + 'user' => acct_infos['account'], + 'my_commit_url' => url + } + + es = ESClient.new(index: 'accounts') + es.put_source_by_id(to, account_uuid_info) +end + +def read_mail_content(mail_file) + mail_content = Mail.read(mail_file) + + return mail_content +end -- 2.23.0

2 1

[PATCH compass-ci] sched: refactor get_boot_ipxe to job class
by Cao Xueliang 09 Oct '20

09 Oct '20

According os_mount to add initrds full path and add initrds file name in root append. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 89 +++++++++++++++++++++++++++++++++----- src/lib/sched.cr | 54 ++++------------------- src/scheduler/constants.cr | 4 ++ 3 files changed, 89 insertions(+), 58 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 53a560c..bedfec0 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -72,10 +72,12 @@ class Job lab initrd_pkg initrd_deps + initrds result_root access_key access_key_file lkp_initrd_user + kernel kernel_append_root docker_image ) @@ -123,8 +125,10 @@ class Job set_result_root() set_result_service() set_os_mount() - set_kernel_append_root() set_pp_initrd() + set_initrds() + set_kernel() + set_kernel_append_root() set_lkp_server() end @@ -228,23 +232,83 @@ class Job end end - private def set_kernel_append_root - os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") - lkp_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}/initrd.lkp") - lkp_basename = File.basename(lkp_real_path) + private def vmlinuz + if os_mount == "initramfs" + return "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/vmlinuz")}" + else + return "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" + end + end + + private def set_kernel + self["kernel"] = "kernel #{vmlinuz()}" + end + + private def common_initrds + temp_initrds = [] of String + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz")}" + temp_initrds << "#{SCHED_HTTP_PREFIX}/job_initrd_tmpfs/#{id}/job.cgz" + + return temp_initrds + end + + private def initramfs_initrds + temp_initrds = [] of String + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + + temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? + temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? + + return temp_initrds + end + + private def nfs_cifs_initrds + temp_initrds = [] of String + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/initrd.lkp")}" + + return temp_initrds + end + + private def set_initrds + temp_initrds = [] of String - current_basename = "" if "#{os_mount}" == "initramfs" - current_real_path = JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current") - current_basename = File.basename(current_real_path) + temp_initrds.concat(initramfs_initrds()) + else + temp_initrds.concat(nfs_cifs_initrds()) + end + + temp_initrds.concat(common_initrds()) + + self["initrds"] = temp_initrds.join(" ") + end + + private def initrd_basename + basenames = "" + + initrds.split(/ /).each do |initrd| + basenames += "initrd=#{File.basename(initrd)} " end + return basenames + end + + private def set_kernel_append_root + os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") + fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} initrd=#{lkp_basename}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrd_basename()}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr initrd=#{lkp_basename}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 initrd=#{current_basename}", - "container" => "" + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrd_basename()}", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrd_basename()}", + "container" => "", } self["kernel_append_root"] = fs2root[os_mount] end @@ -270,6 +334,7 @@ class Job end end end + self["initrd_deps"] = initrd_deps_arr.join(" ") self["initrd_pkg"] = initrd_pkg_arr.join(" ") end diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 309df04..990641c 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -488,61 +488,23 @@ class Sched return returned end - private def get_pp_initrd(job : Job) - initrd_deps = "" - initrd_pkg = "" - if job.os_mount == "initramfs" - initrd_deps += job.initrd_deps.split.join() { |item| "initrd #{item}\n" } - initrd_pkg += job.initrd_pkg.split.join() { |item| "initrd #{item}\n" } + private def get_initrds(job_initrds) + initrds = "" + job_initrds.each do |initrd| + initrds += "initrd #{initrd}\n" end - return initrd_deps, initrd_pkg + return initrds end private def get_boot_ipxe(job : Job) - initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" - - initrd_deps, initrd_pkg = get_pp_initrd(job) - - initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" - sched_http_prefix = "http://#{SCHED_HOST}:#{SCHED_PORT}" - os_http_prefix = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" - response = "#!ipxe\n\n" - if job.os_mount == "initramfs" - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/current")}\n" - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/run-ipconfig.cgz")}\n" - else - response += "initrd #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/initrd.lkp")}\n" - end - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/#{initrd_lkp_cgz}")}\n" - response += "initrd #{sched_http_prefix}/job_initrd_tmpfs/#{job.id}/job.cgz\n" - response += initrd_deps - response += initrd_pkg - if job.os_mount == "initramfs" - response += "kernel #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" - else - response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" - end + response += get_initrds(job.initrds.split(/ /)) + response += job.kernel response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" response += " #{job.kernel_append_root}" response += add_kernel_console_param(job.os_arch) - if job.os_mount == "initramfs" - response += " initrd=#{initrd_lkp_cgz} initrd=job.cgz" - job.initrd_deps.split.each do |initrd_dep| - response += " initrd=#{File.basename(initrd_dep)}" - end - response += " initrd=#{File.basename(JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/run-ipconfig.cgz"))}\n" - else - response += " initrd=#{initrd_lkp_cgz} initrd=job.cgz\n" - end - response += "boot\n" + response += "\nboot\n" puts %({"job_id": "#{job.id}", "job_state": "boot"}) return response diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 7ce49a1..839c241 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -24,4 +24,8 @@ OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i3 SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" +INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" +OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" +SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" + DEMO_JOB = %({"suite":"pixz","testcase":"pixz","category":"benchmark","nr_threads":1,"pixz":null,"job_origin":"jobs/pixz.yaml","testbox":"wfg-e595","arch":"x86_64","tbox_group":"wfg-e595","id":"100","kmsg":null,"boot-time":null,"uptime":null,"iostat":null,"heartbeat":null,"vmstat":null,"numa-numastat":null,"numa-vmstat":null,"numa-meminfo":null,"proc-vmstat":null,"proc-stat":null,"meminfo":null,"slabinfo":null,"interrupts":null,"kconfig":"x86_64-rhel-7.6","compiler":"gcc-7"}) -- 2.23.0

3 6

[PATCH v2 lkp-tests] lib: add transfer_port.sh
by Zhang Yale 09 Oct '20

09 Oct '20

Provide a login port for user, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- lib/transfer_port.sh | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 lib/transfer_port.sh diff --git a/lib/transfer_port.sh b/lib/transfer_port.sh new file mode 100755 index 00000000..61fb67b4 --- /dev/null +++ b/lib/transfer_port.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# uuid + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 + +get_port() +{ + [ -n "$uuid" ] || return + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%10+11333)) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] || continue + echo "$port" + break + done +} + -- 2.23.0

4 8

[PATCH v4 compass-ci 2/2] sparrow: open the masquerade at public zone
by Hu Xuejiao 09 Oct '20

09 Oct '20

Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/2-network/iptables | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 7183145..0fd82e5 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -28,3 +28,4 @@ DOCKER0_SUBNET=172.17.0.0/16 firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" +firewall-cmd --zone=public --add-masquerade -- 2.23.0

3 2

[PATCH v4 compass-ci 1/2] fix(sparrow): process will exit if firewalld.service was not installed
by Hu Xuejiao 09 Oct '20

09 Oct '20

When firewalld is inactive, return code of 'systemctl status firewalld' will be 3, and install-tiny has set -e, so this will cause this process exit. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/2-network/iptables | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..7183145 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -12,15 +12,19 @@ iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE -systemctl status firewalld | grep -q "running" && { +command -v firewall-cmd > /dev/null || { + echo "firewalld.service could not be found" + exit 0 +} - DOCKER0_IFACE=docker0 - DOCKER0_SUBNET=172.17.0.0/16 +systemctl start firewalld +[ "$(systemctl is-active firewalld)" == "active" ] || { + echo "start firewalld.service failed" + exit 0 +} - iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE - iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE +DOCKER0_SUBNET=172.17.0.0/16 - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -} +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -- 2.23.0

2 1

[PATCH lkp-tests 2/2] hosts: add vm-aarch64-*
by Wu Fengguang 30 Sep '20

30 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- hosts/vm-aarch64-1p1g | 6 ++++++ hosts/vm-aarch64-2p1g | 6 ++++++ hosts/vm-aarch64-2p4g | 6 ++++++ hosts/vm-aarch64-2p8g | 6 ++++++ hosts/vm-pxe-aarch64-1p1g | 6 ++++++ hosts/vm-pxe-aarch64-2p1g | 6 ++++++ hosts/vm-pxe-aarch64-2p4g | 6 ++++++ hosts/vm-pxe-aarch64-2p8g | 6 ++++++ 8 files changed, 48 insertions(+) create mode 100644 hosts/vm-aarch64-1p1g create mode 100644 hosts/vm-aarch64-2p1g create mode 100644 hosts/vm-aarch64-2p4g create mode 100644 hosts/vm-aarch64-2p8g create mode 100644 hosts/vm-pxe-aarch64-1p1g create mode 100644 hosts/vm-pxe-aarch64-2p1g create mode 100644 hosts/vm-pxe-aarch64-2p4g create mode 100644 hosts/vm-pxe-aarch64-2p8g diff --git a/hosts/vm-aarch64-1p1g b/hosts/vm-aarch64-1p1g new file mode 100644 index 00000000..b7a53601 --- /dev/null +++ b/hosts/vm-aarch64-1p1g @@ -0,0 +1,6 @@ +provider: qemu +template: kvm +arch: aarch64 +nr_node: 1 +nr_cpu: 1 +memory: 1G diff --git a/hosts/vm-aarch64-2p1g b/hosts/vm-aarch64-2p1g new file mode 100644 index 00000000..08d0c25c --- /dev/null +++ b/hosts/vm-aarch64-2p1g @@ -0,0 +1,6 @@ +provider: qemu +template: kvm +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 1G diff --git a/hosts/vm-aarch64-2p4g b/hosts/vm-aarch64-2p4g new file mode 100644 index 00000000..d57774b9 --- /dev/null +++ b/hosts/vm-aarch64-2p4g @@ -0,0 +1,6 @@ +provider: qemu +template: kvm +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 4G diff --git a/hosts/vm-aarch64-2p8g b/hosts/vm-aarch64-2p8g new file mode 100644 index 00000000..40fe2716 --- /dev/null +++ b/hosts/vm-aarch64-2p8g @@ -0,0 +1,6 @@ +provider: qemu +template: kvm +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 8G diff --git a/hosts/vm-pxe-aarch64-1p1g b/hosts/vm-pxe-aarch64-1p1g new file mode 100644 index 00000000..554e8fb1 --- /dev/null +++ b/hosts/vm-pxe-aarch64-1p1g @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +arch: aarch64 +nr_node: 1 +nr_cpu: 1 +memory: 1G diff --git a/hosts/vm-pxe-aarch64-2p1g b/hosts/vm-pxe-aarch64-2p1g new file mode 100644 index 00000000..27b8438b --- /dev/null +++ b/hosts/vm-pxe-aarch64-2p1g @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 1G diff --git a/hosts/vm-pxe-aarch64-2p4g b/hosts/vm-pxe-aarch64-2p4g new file mode 100644 index 00000000..c31dd953 --- /dev/null +++ b/hosts/vm-pxe-aarch64-2p4g @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 4G diff --git a/hosts/vm-pxe-aarch64-2p8g b/hosts/vm-pxe-aarch64-2p8g new file mode 100644 index 00000000..3687d460 --- /dev/null +++ b/hosts/vm-pxe-aarch64-2p8g @@ -0,0 +1,6 @@ +provider: qemu +template: pxe +arch: aarch64 +nr_node: 1 +nr_cpu: 2 +memory: 8G -- 2.23.0

1 1

[PATCH lkp-tests 1/2] hosts: rename dc-* to dc-aarch64-*
by Wu Fengguang 30 Sep '20

30 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- hosts/{dc-1g => dc-aarch64-1g} | 0 hosts/{dc-2g => dc-aarch64-2g} | 0 hosts/{dc-4g => dc-aarch64-4g} | 0 hosts/{dc-8g => dc-aarch64-8g} | 0 4 files changed, 0 insertions(+), 0 deletions(-) rename hosts/{dc-1g => dc-aarch64-1g} (100%) rename hosts/{dc-2g => dc-aarch64-2g} (100%) rename hosts/{dc-4g => dc-aarch64-4g} (100%) rename hosts/{dc-8g => dc-aarch64-8g} (100%) diff --git a/hosts/dc-1g b/hosts/dc-aarch64-1g similarity index 100% rename from hosts/dc-1g rename to hosts/dc-aarch64-1g diff --git a/hosts/dc-2g b/hosts/dc-aarch64-2g similarity index 100% rename from hosts/dc-2g rename to hosts/dc-aarch64-2g diff --git a/hosts/dc-4g b/hosts/dc-aarch64-4g similarity index 100% rename from hosts/dc-4g rename to hosts/dc-aarch64-4g diff --git a/hosts/dc-8g b/hosts/dc-aarch64-8g similarity index 100% rename from hosts/dc-8g rename to hosts/dc-aarch64-8g -- 2.23.0

1 0

[PATCH v4 compass-ci] container: consider the situation that GITCACHE_HOST is undefined
by Hu Xuejiao 30 Sep '20

30 Sep '20

- "[ -n $GITCACHE_HOST ] && ... " if $GITCACHE_HOST is none, process will exit 1 - "[ -z $GITCACHE_HOST ] || ... " if $GITCACHE_HOST is none, process will keep executing. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- container/crystal-shards/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/crystal-shards/Dockerfile b/container/crystal-shards/Dockerfile index 0c5713c..bdc5b66 100644 --- a/container/crystal-shards/Dockerfile +++ b/container/crystal-shards/Dockerfile @@ -7,7 +7,7 @@ MAINTAINER chief <taxcom(a)tom.com> ARG GITCACHE_HOST ARG GITCACHE_PORT -RUN [ -n "$GITCACHE_HOST" ] && echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig +RUN [ -z "$GITCACHE_HOST" ] || echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig copy shard.yml /usr/share/crystal/app/shard.yml -- 2.23.0

1 0

[PATCH v2 lkp-tests] sshd: send emails to provide login services for users
by Zhang Yale 30 Sep '20

30 Sep '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 61 +++++++++++++++++++++++++++++++++++------------------ 1 file changed, 41 insertions(+), 20 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..f921a7e5 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,10 @@ #!/bin/sh # pub_key # email +# uuid + +. $LKP_SRC/lib/transfer_port.sh +. $LKP_SRC/lib/http.sh run_ssh() { @@ -11,40 +15,57 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh +Subject: [NOTIFY Compass-ci] $testbox ready to use -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: +Dear $user_name: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: - Login: - ssh root@$PUB_IP - Due time: - $deadline + login: ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: $deadline - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch +Regards +Compass-ci +" +} + +data_failure() +{ + data="To: $email +Subject: [NOTIFY Compass-ci] Applying $testbox failed + +Dear $user_name: + Sorry to inform you that your application failed. + You may need to wait for a while and then thy again. Regards -compass-ci +Compass-ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$email" ] && [ -n "$uuid" ] || return 0 + user_name=$(echo "$email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +port=$(get_port) +report_ssh_port $port run_email -- 2.23.0

1 0

[PATCH v2 compass-ci] container/srv-http forbid access parent directory of job result_root
by Lu Weitao 30 Sep '20

30 Sep '20

by forbid access level 1, 2, 4 directory of /srv/result Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/root/etc/nginx/conf.d/default.conf | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/container/srv-http/root/etc/nginx/conf.d/default.conf b/container/srv-http/root/etc/nginx/conf.d/default.conf index 53833eb..2bf55b6 100644 --- a/container/srv-http/root/etc/nginx/conf.d/default.conf +++ b/container/srv-http/root/etc/nginx/conf.d/default.conf @@ -19,4 +19,8 @@ server { add_header Content-Type text/plain; } } + + location ~* ^/result/(([^/]+)?/?|[^/]+/[^/]+/[^/]+/?)$ { + deny all; + } } -- 2.23.0

1 0

[PATCH v2 compass-ci] fix: sched: submit idle jobs
by Ren Wen 30 Sep '20

30 Sep '20

1) submit all idles jobs once use '*' -> submit each idle job once until all done. 2) add two fields into job if job's filename is 'multi-qemu.yaml'. 3) add field 'idle_job' for idle jobs, value is always 'true'. Two extra fields are used for: monitoring logs of virtual machines started by physical machine. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 15f5c9e..984c7db 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -467,10 +467,20 @@ class Sched end def auto_submit_idle_job(tbox_group) - full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" - Jobfile::Operate.auto_submit_job( - full_path_patterns, - "testbox: #{tbox_group}") if Dir.glob(full_path_patterns).size > 0 + idle_job_path = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/" + + if Dir.exists?(idle_job_path) + jobs = Dir[idle_job_path + "*.yaml"] + + jobs.each do |job| + extra_job_fields = ["idle_job=true"] + if Path[job].basename == "multi-qemu.yaml" + extra_job_fields += ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", + "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}"] + end + Jobfile::Operate.auto_submit_job(job, "testbox: #{tbox_group}", extra_job_fields) + end + end end private def add_kernel_console_param(arch_tmp) @@ -520,7 +530,7 @@ class Sched "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" else response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" end response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" -- 2.23.0

3 7

[PATCH v3 compass-ci] src/extract-stats deal with potential exceptions
by Lu Weitao 30 Sep '20

30 Sep '20

background: extract-stats process is at risk of be interrupted, due to other dependent services(task-queue, es_client) may not working properly solution: 1. use begin ... rescue to handle potential exceptions - when ES and task-queue not working properly, sleep 10s and visit again - when "result_root" does not exist, skip and remove task from task-queue 2. refactor for consume_sched_queue() of stats_worker.cr Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 60 ++++++++++++++++++------------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index fcba6af..2af07d4 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -18,40 +18,50 @@ class StatsWorker def consume_sched_queue(queue_path : String) loop do begin - response = @tq.consume_task(queue_path) + job_id = get_job_id(queue_path) + if job_id # will consume the job by post-processing + job = @es.get_job_content(job_id) + result_root = job["result_root"]? + result_post_processing(job_id, result_root.to_s, queue_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + end rescue e STDERR.puts e.message - # incase of many error message when task-queue is busy + # incase of many error message when task-queue, ES does not work sleep(10) - next end + end + end - if response[0] == 200 - job_id = JSON.parse(response[1].to_json)["id"] - job = @es.get_job_content(job_id.to_s) - result_root = job["result_root"] - begin - # extract stats.json - system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" - # storage stats to job in es - store_stats_es(result_root.to_s, job_id.to_s) if result_root - # send mail to submitter for job results - system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" - rescue e - STDERR.puts e.message - next - end - - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - else - sleep(60) - end + # get job_id from task-queue + def get_job_id(queue_path : String) + response = @tq.consume_task(queue_path) + if response[0] == 200 + JSON.parse(response[1].to_json)["id"].to_s + else + # will sleep 60s if no task in task-queue + sleep(60) + nil end end - def store_stats_es(result_root : String, job_id : String) + def result_post_processing(job_id : String, result_root : String, queue_path : String) + return nil unless result_root && File.exists?(result_root) + + # extract stats.json + system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" + # storage stats to job in es + store_stats_es(result_root, job_id, queue_path) + # send mail to submitter for job results + system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" + end + + def store_stats_es(result_root : String, job_id : String, queue_path : String) stats_path = "#{result_root}/stats.json" - raise "#{stats_path} file not exists." unless File.exists?(stats_path) + unless File.exists?(stats_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + raise "#{stats_path} file not exists." + end stats = File.open(stats_path) do |file| JSON.parse(file) -- 2.23.0

1 0

[PATCH compass-ci] container: crystal-shards: fix build error
by Liu Yinsi 30 Sep '20

30 Sep '20

build crystal-shards depends gitcache, When build crystal-shards, error message as follows: Step 8/12 : RUN shards ---> Running in be90213caa48 Resolving dependencies Fetching https://github.com/kemalcr/kemal.git Failed to clone https://github.com/kemalcr/kemal.git The command '/bin/sh -c shards' returned a non-zero code: 1 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/crystal-shards/build-depends | 1 + 1 file changed, 1 insertion(+) diff --git a/container/crystal-shards/build-depends b/container/crystal-shards/build-depends index 6627b6f..6eed7ec 100755 --- a/container/crystal-shards/build-depends +++ b/container/crystal-shards/build-depends @@ -1 +1,2 @@ crystal-base +gitcache -- 2.23.0

1 0

[PATCH compass-ci] sparrow/install-tiny: fix process exit
by Liu Yinsi 30 Sep '20

30 Sep '20

-e will cause process exit when return code is not 0, only use when debugging. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-tiny | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index 1c0b5d2..a291549 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -1,4 +1,4 @@ -#!/bin/bash -e +#!/bin/bash # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # For your reference only. -- 2.23.0

1 0

[PATCH compass-ci] fix: sched: add idle job to idle queue
by Ren Wen 30 Sep '20

30 Sep '20

Add idle job to idle queue, then idle machine find job in idle queue. For example: when idle job under "allot/idle/vm-hi1620-2p8g", queue should be: "queues/sched/vm-hi1620-2p8g/idle" Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 309df04..47a2e54 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -266,7 +266,7 @@ class Sched }] unless tbox_group # only single job will has "idle job" and "execute rate limiter" - if %<job["job_origin"]>.includes?("allot/idle/") + unless job["idle_job"].empty? tbox_group = "#{tbox_group}/idle" else tbox_group += "#{job.get_uuid_tag}" -- 2.23.0

1 0

[PATCH lkp-tests 1/3] sshd: add build channel
by Zhang Yale 30 Sep '20

30 Sep '20

Auto send email to remind user to login with port/domain_name and send port to sched Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- daemon/sshd | 68 ++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 47 insertions(+), 21 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..d1fcae52 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,12 @@ #!/bin/sh # pub_key # email +# uuid + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 run_ssh() { @@ -14,37 +20,57 @@ run_ssh() compose_email() { deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") - data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh - -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: - - Login: - ssh root@$PUB_IP - Due time: - $deadline - - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch + if [ -n $port ]; then + data="To: $email +Subject: [NOTIFY compass-ci] $testbox ready to use + +Dear $user_name: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + Please use the following command to login: + + ssh root(a)api.compass-ci.openeuler.org -p $port + +Regards +compassi-ci +" + else + data="To: $email +Subject: [NOTIFY compass-ci] Applying $testbox failed + +Dear $user_name: + Sorry to inform you than your application failed. + + You may need to wait sometime and then have a second try. Regards compass-ci " + fi +} + +get_port() +{ + [ -n "$uuid" ] || return + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%10+11333)) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -Nf -R $port:$PUB_IP:22 sshr@"$sshr_ip" -p "$sshr_port" > out 2>&1 + [ -z "`grep -i "error" out`" ] || continue + return "$port" + break + done } run_email() { - [ -n "$email" ] || return 0 + [ -n "$email" ] && [ -n "$uuid" ] || return 0 + user_name=$(echo "$email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +get_port +report_ssh_port $port run_email -- 2.23.0

1 1

[PATCH lkp-tests 3/3] lib: add transfer_port.sh
by Zhang Yale 30 Sep '20

30 Sep '20

Provide a login port for user, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- lib/transfer_port.sh | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 lib/transfer_port.sh diff --git a/lib/transfer_port.sh b/lib/transfer_port.sh new file mode 100755 index 00000000..61fb67b4 --- /dev/null +++ b/lib/transfer_port.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# uuid + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 + +get_port() +{ + [ -n "$uuid" ] || return + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%10+11333)) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] || continue + echo "$port" + break + done +} + -- 2.23.0

1 0

[PATCH lkp-tests 2/3] sshd: send emails to provide login services for users
by Zhang Yale 30 Sep '20

30 Sep '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- daemon/sshd | 49 ++++++++++++++++++++++--------------------------- 1 file changed, 22 insertions(+), 27 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index d1fcae52..35809ffe 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -3,11 +3,9 @@ # email # uuid +. $LKP_SRC/lib/transfer_port.sh . $LKP_SRC/lib/http.sh -sshr_ip=$LKP_SERVER -sshr_port=5050 - run_ssh() { [ -n "$pub_key" ] || return @@ -17,49 +15,46 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") - if [ -n $port ]; then - data="To: $email + data="To: $email Subject: [NOTIFY compass-ci] $testbox ready to use Dear $user_name: Thanks for your participation in software ecosystem! According to your application, $testbox has been provisioned. - Please use the following command to login: + The datails are as follows: - ssh root(a)api.compass-ci.openeuler.org -p $port + login: ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: $deadline Regards -compassi-ci +compass-ci " - else - data="To: $email +} + +data_failure() +{ + data="To: $email Subject: [NOTIFY compass-ci] Applying $testbox failed Dear $user_name: - Sorry to inform you than your application failed. - - You may need to wait sometime and then have a second try. + Sorry to inform you that your application failed. + You may need to wait for a while and then have a second try. Regards compass-ci " - fi } -get_port() +compose_email() { - [ -n "$uuid" ] || return - for i in $(seq 1 10) - do - port=$(($(date +%s%N)%10+11333)) - ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -Nf -R $port:$PUB_IP:22 sshr@"$sshr_ip" -p "$sshr_port" > out 2>&1 - [ -z "`grep -i "error" out`" ] || continue - return "$port" - break - done + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi } run_email() @@ -71,6 +66,6 @@ run_email() } run_ssh -get_port +port=$(get_port) report_ssh_port $port run_email -- 2.23.0

1 0

[PATCH lkp-tests] fix: run-ipconfug: delete the ntpdate
by Wang Chenglong 30 Sep '20

30 Sep '20

the run_ntpdate function in bootstrap.sh here run ntpdate command is unnecessary Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- bin/run-ipconfig | 1 - 1 file changed, 1 deletion(-) diff --git a/bin/run-ipconfig b/bin/run-ipconfig index ad2a9808..0f80feaf 100755 --- a/bin/run-ipconfig +++ b/bin/run-ipconfig @@ -20,4 +20,3 @@ read_ip() [ -n "$IP" ] || read_ip configure_networking -ntpdate -u ntp.api.bz -- 2.23.0

1 0

[PATCH lkp-tests] fix: lib/bootstrap.sh: ntpdate from $LKP_SERVER
by Wang Chenglong 30 Sep '20

30 Sep '20

befor: [ "$LKP_SERVER"=inn ] || return ntpdate -b $LKP_SERVER the ntpdate command can't run after: [ -n "$LKP_SERVER" ] || return ntpdate -b $LKP_SERVER now, if $LKP_SERVER is ok, it can synchronised the time Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/bootstrap.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index f8a98de6..0be7ed3b 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -188,7 +188,7 @@ clearlinux_timesync() run_ntpdate() { [ -z "$NO_NETWORK" ] || return - [ "$LKP_SERVER" = inn ] || return + [ -n "$LKP_SERVER" ] || return is_clearlinux && clearlinux_timesync && return [ -x '/usr/sbin/ntpdate' ] || return -- 2.23.0

1 0

[PATCH compass-ci] container: manjaro: fix dependency issues
by Liu Yinsi 30 Sep '20

30 Sep '20

When build manjaro, error message as follows: util-linux-libs and libutil-linux are in conflict. Remove libutil-linux? [y/N] util-linux-libs and libutil-linux are in conflict error: unresolvable package conflicts detected error: failed to prepare transaction (conflicting dependencies) When installing packages in Arch, do not run pacman -Sy package_name instead of pacman -Syu package_name, as this could lead to dependency issues. See https://wiki.archlinux.org/index.php/Pacman Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/manjaro/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/manjaro/Dockerfile b/container/manjaro/Dockerfile index f6cec48..48c5ee6 100644 --- a/container/manjaro/Dockerfile +++ b/container/manjaro/Dockerfile @@ -7,7 +7,7 @@ MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> COPY root / RUN chmod 755 /etc /etc/pacman.d -RUN pacman --needed --noprogressbar --noconfirm -Sy && \ +RUN pacman --needed --noprogressbar --noconfirm -Syu && \ pacman --needed --noprogressbar --noconfirm -S bash zsh git openssh rsync make gcc tzdata sudo coreutils util-linux vim RUN ssh-keygen -t rsa -P '' -f /etc/ssh/ssh_host_rsa_key -- 2.23.0

1 0

[PATCH v2 compass-ci] feat: helper: snapshot of taskqueue
by Tong Qunfeng 30 Sep '20

30 Sep '20

Give a snapshot information about taskqueue (data at redis sorted set queue "queues/id2content"). It will generate a file named taskqueue_snap_yyyymmdd.md. snapshot content examples: |id---|add_time----------|current queue name| 14703, 2020-07-22 14:33, sched/vm-hi1620-2p8g-xxx/in_process 14831, 2020-07-22 17:51, sched/vm-hi1620-2p8g-xxx/in_process crystal.89630, 2020-09-23 11:19, sched/vm-hi1620-2p8g/in_process crystal.89667, 2020-09-23 11:20, sched/vm-hi1620-2p8g/in_process Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- .../helper/redis_op_gc/taskqueue_snap.rb | 56 +++++++++++++++++++ 1 file changed, 56 insertions(+) create mode 100755 user-client/helper/redis_op_gc/taskqueue_snap.rb diff --git a/user-client/helper/redis_op_gc/taskqueue_snap.rb b/user-client/helper/redis_op_gc/taskqueue_snap.rb new file mode 100755 index 0000000..f0a7034 --- /dev/null +++ b/user-client/helper/redis_op_gc/taskqueue_snap.rb @@ -0,0 +1,56 @@ +#!/usr/bin/env ruby + +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require_relative './basic_env' + +if !ARGV.empty? + puts 'Usage:' + puts "input: #{__FILE__}" + puts 'output: taskqueue_snap_yyyymmdd.md' + puts ' # id, first add time, current queue' + puts ' # 14703, 2020-07-22 14:33, sched/vm-hi1620-2p8g-xxx/in_process' + puts ' # 14831, 2020-07-22 17:51, sched/vm-hi1620-2p8g-xxx/in_process' + puts ' # crystal.89630, 2020-09-23 11:19, sched/vm-hi1620-2p8g/in_process' + puts ' # crystal.89667, 2020-09-23 11:20, sched/vm-hi1620-2p8g/in_process' + exit +end + +# redis hash key "queues/id2content" is used by taskqueue +cmd = "#{CMD_BASE} queues/id2content , hgetall" +result = `#{cmd}`.chomp +results = result.split("\n") + +task_info = [] + +# format time to "2020-09-29 14:23" +i = 0 +task_num = results.size / 2 +while i < task_num + task_id = results[i] + task_content = JSON.parse(results[i + 1]) + + time = task_content['add_time'] + task_content['add_time'] = Time.at(time.to_f).strftime('%Y-%m-%d %H:%M') unless time.nil? + + time = task_content['move_time'] + task_content['move_time'] = Time.at(time.to_f).strftime('%Y-%m-%d %H:%M') unless time.nil? + + task_info << { 'id' => task_id, 'value' => task_content } + + i += 2 + set_progress(i, task_num) +end + +# sort and put short information to file. +task_info.sort! { |a, b| a['value']['add_time'] <=> b['value']['add_time'] } +today = Time.now.strftime('%Y%m%d') +File.open("taskqueue_snap_#{today}.md", 'w') do |f| + task_info.each do |task| + f.write(task['id'] + ', ' + task['value']['add_time'] \ + + ', ' + task['value']['queue'] + "\n") + end +end -- 2.23.0

2 2

[PATCH compass-ci 3/4] sparrow: start the firewall in adapting openeuler
by Hu Xuejiao 30 Sep '20

30 Sep '20

When firewalld is inactive, return code of 'systemctl status firewalld' will be 3, and install-tiny has set -e, so this will cause this process exit. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/2-network/iptables | 26 +++++++++++++++++--------- 1 file changed, 17 insertions(+), 9 deletions(-) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..831c1a3 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -12,15 +12,23 @@ iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE -systemctl status firewalld | grep -q "running" && { +command -v firewall-cmd > /dev/null || { + echo "firewalld.service could not be found" + exit 0 +} + +systemctl start firewalld +[ "$(systemctl is-active firewalld)" == "active" ] || { + echo "start firewalld.service failed" + exit 0 +} - DOCKER0_IFACE=docker0 - DOCKER0_SUBNET=172.17.0.0/16 +DOCKER0_IFACE=docker0 +DOCKER0_SUBNET=172.17.0.0/16 - iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE - iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE +iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE +iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -} +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -- 2.23.0

3 2

[PATCH lab-z9] servers add disks, hosts file should update
by Zhang Dewan 30 Sep '20

30 Sep '20

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a11 | 19 ------------------- hosts/taishan200-2280-2s48p-256g--a24 | 19 ------------------- hosts/taishan200-2280-2s48p-256g--a25 | 19 ------------------- hosts/taishan200-2280-2s48p-256g--a66 | 19 ------------------- hosts/taishan200-2280-2s48p-256g--a67 | 19 ------------------- hosts/taishan200-2280-2s48p-256g--a68 | 19 ------------------- hosts/taishan200-2280-2s48p-256g--a69 | 19 ------------------- hosts/taishan200-2280-2s48p-256g--a70 | 19 ------------------- hosts/taishan200-2280-2s48p-256g--a71 | 13 ------------- hosts/taishan200-2280-2s64p-256g--a10 | 20 -------------------- hosts/taishan200-2280-2s64p-256g--a3 | 25 ------------------------- hosts/taishan200-2280-2s64p-256g--a4 | 25 ------------------------- hosts/taishan200-2280-2s64p-256g--a8 | 16 ---------------- hosts/taishan200-2280-2s64p-256g--a9 | 17 ----------------- 14 files changed, 268 deletions(-) delete mode 100644 hosts/taishan200-2280-2s48p-256g--a11 delete mode 100644 hosts/taishan200-2280-2s48p-256g--a24 delete mode 100644 hosts/taishan200-2280-2s48p-256g--a25 delete mode 100644 hosts/taishan200-2280-2s48p-256g--a66 delete mode 100644 hosts/taishan200-2280-2s48p-256g--a67 delete mode 100644 hosts/taishan200-2280-2s48p-256g--a68 delete mode 100644 hosts/taishan200-2280-2s48p-256g--a69 delete mode 100644 hosts/taishan200-2280-2s48p-256g--a70 delete mode 100644 hosts/taishan200-2280-2s48p-256g--a71 delete mode 100644 hosts/taishan200-2280-2s64p-256g--a10 delete mode 100644 hosts/taishan200-2280-2s64p-256g--a3 delete mode 100644 hosts/taishan200-2280-2s64p-256g--a4 delete mode 100644 hosts/taishan200-2280-2s64p-256g--a8 delete mode 100644 hosts/taishan200-2280-2s64p-256g--a9 diff --git a/hosts/taishan200-2280-2s48p-256g--a11 b/hosts/taishan200-2280-2s48p-256g--a11 deleted file mode 100644 index 9c0224e..0000000 --- a/hosts/taishan200-2280-2s48p-256g--a11 +++ /dev/null @@ -1,19 +0,0 @@ -nr_node: 4 -nr_cpu: 96 -memory: 256G -nr_hdd_partitions: 4 -hdd_partitions: - - /dev/disk/by-id/scsi-350000399b8891431-part1 - - /dev/disk/by-id/scsi-350000399b8890e09 - - /dev/disk/by-id/scsi-350000399b8890cd1 - - /dev/disk/by-id/scsi-350000399b8727929 -mac_addr: - - cc:05:77:ab:97:62 - - cc:05:77:ab:97:63 - - cc:05:77:ab:97:64 - - cc:05:77:ab:97:65 - - cc:05:77:ab:97:66 - - cc:05:77:ab:97:67 - - cc:05:77:ab:97:68 - - cc:05:77:ab:97:69 -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a24 b/hosts/taishan200-2280-2s48p-256g--a24 deleted file mode 100644 index 7dda921..0000000 --- a/hosts/taishan200-2280-2s48p-256g--a24 +++ /dev/null @@ -1,19 +0,0 @@ -nr_node: 4 -nr_cpu: 96 -memory: 256G -nr_hdd_partitions: 4 -hdd_partitions: - - /dev/disk/by-id/scsi-350000399b8917d59 - - /dev/disk/by-id/scsi-350000399b891790d - - /dev/disk/by-id/scsi-350000399b89185d9 - - /dev/disk/by-id/scsi-350000399b8917ef1 -mac_addr: - - 44:67:47:c9:d8:f4 - - 44:67:47:c9:d8:f5 - - 44:67:47:c9:d8:f6 - - 44:67:47:c9:d8:f7 - - 44:67:47:c9:d8:f8 - - 44:67:47:c9:d8:f9 - - 44:67:47:c9:d8:fa - - 44:67:47:c9:d8:fb -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a25 b/hosts/taishan200-2280-2s48p-256g--a25 deleted file mode 100644 index 5d075b9..0000000 --- a/hosts/taishan200-2280-2s48p-256g--a25 +++ /dev/null @@ -1,19 +0,0 @@ -nr_node: 4 -nr_cpu: 96 -memory: 256G -nr_hdd_partitions: 4 -hdd_partitions: - - /dev/disk/by-id/scsi-350000399b88b4499 - - /dev/disk/by-id/scsi-350000399b88b4455 - - /dev/disk/by-id/scsi-350000399b88b442d - - /dev/disk/by-id/scsi-350000399b88b44f9 -mac_addr: - - 44:67:47:c9:ea:08 - - 44:67:47:c9:ea:09 - - 44:67:47:c9:ea:0a - - 44:67:47:c9:ea:0b - - 44:67:47:c9:ea:0c - - 44:67:47:c9:ea:0d - - 44:67:47:c9:ea:0e - - 44:67:47:c9:ea:0f -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a66 b/hosts/taishan200-2280-2s48p-256g--a66 deleted file mode 100644 index 88ef791..0000000 --- a/hosts/taishan200-2280-2s48p-256g--a66 +++ /dev/null @@ -1,19 +0,0 @@ -nr_node: 4 -nr_cpu: 96 -memory: 256G -nr_hdd_partitions: 4 -hdd_partitions: - - /dev/disk/by-id/scsi-35000c5009a671773 - - /dev/disk/by-id/scsi-35000c5009a690c53 - - /dev/disk/by-id/scsi-35000c5009a6713fb - - /dev/disk/by-id/scsi-35000c5009a65963f -mac_addr: - - 5c:e8:83:31:c8:8e - - 5c:e8:83:31:c8:8f - - 5c:e8:83:31:c8:90 - - 5c:e8:83:31:c8:91 - - 5c:e8:83:31:c8:92 - - 5c:e8:83:31:c8:93 - - 5c:e8:83:31:c8:94 - - 5c:e8:83:31:c8:95 -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a67 b/hosts/taishan200-2280-2s48p-256g--a67 deleted file mode 100644 index 97ea498..0000000 --- a/hosts/taishan200-2280-2s48p-256g--a67 +++ /dev/null @@ -1,19 +0,0 @@ -nr_node: 4 -nr_cpu: 96 -memory: 256G -nr_hdd_partitions: 4 -hdd_partitions: - - /dev/disk/by-id/scsi-350000399b88aed21 - - /dev/disk/by-id/scsi-350000399b88af325 - - /dev/disk/by-id/scsi-350000399b88af699 - - /dev/disk/by-id/scsi-350000399b88af421 -mac_addr: - - 44:67:47:c9:e2:2c - - 44:67:47:c9:e2:2d - - 44:67:47:c9:e2:2e - - 44:67:47:c9:e2:2f - - 44:67:47:c9:e2:30 - - 44:67:47:c9:e2:31 - - 44:67:47:c9:e2:32 - - 44:67:47:c9:e2:33 -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a68 b/hosts/taishan200-2280-2s48p-256g--a68 deleted file mode 100644 index f3d50f4..0000000 --- a/hosts/taishan200-2280-2s48p-256g--a68 +++ /dev/null @@ -1,19 +0,0 @@ -nr_node: 4 -nr_cpu: 96 -memory: 256G -nr_hdd_partitions: 4 -hdd_partitions: - - /dev/disk/by-id/scsi-350000399b8909f81 - - /dev/disk/by-id/scsi-350000399b8909f79 - - /dev/disk/by-id/scsi-350000399b8909605 - - /dev/disk/by-id/scsi-350000399b8909b85 -mac_addr: - - 84:46:fe:89:b6:e6 - - 84:46:fe:89:b6:e7 - - 84:46:fe:89:b6:e8 - - 84:46:fe:89:b6:e9 - - 84:46:fe:89:b6:ea - - 84:46:fe:89:b6:eb - - 84:46:fe:89:b6:ec - - 84:46:fe:89:b6:ed -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a69 b/hosts/taishan200-2280-2s48p-256g--a69 deleted file mode 100644 index 9e5734e..0000000 --- a/hosts/taishan200-2280-2s48p-256g--a69 +++ /dev/null @@ -1,19 +0,0 @@ -nr_node: 4 -nr_cpu: 96 -memory: 256G -nr_hdd_partitions: 4 -hdd_partitions: - - /dev/disk/by-id/scsi-350000399b88be92d - - /dev/disk/by-id/scsi-350000399b88befc5 - - /dev/disk/by-id/scsi-350000399b88be701 - - /dev/disk/by-id/scsi-350000399b88bea5d -mac_addr: - - 44:67:47:c9:d9:4e - - 44:67:47:c9:d9:4f - - 44:67:47:c9:d9:50 - - 44:67:47:c9:d9:51 - - 44:67:47:c9:d9:52 - - 44:67:47:c9:d9:53 - - 44:67:47:c9:d9:54 - - 44:67:47:c9:d9:55 -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a70 b/hosts/taishan200-2280-2s48p-256g--a70 deleted file mode 100644 index 0dd8208..0000000 --- a/hosts/taishan200-2280-2s48p-256g--a70 +++ /dev/null @@ -1,19 +0,0 @@ -nr_node: 4 -nr_cpu: 96 -memory: 256G -nr_hdd_partitions: 4 -hdd_partitions: - - /dev/disk/by-id/scsi-350000399b890931d - - /dev/disk/by-id/scsi-350000399b8909d79 - - /dev/disk/by-id/scsi-350000399b8909ac5 - - /dev/disk/by-id/scsi-350000399b8909c99 -mac_addr: - - 84:46:fe:89:b7:22 - - 84:46:fe:89:b7:23 - - 84:46:fe:89:b7:24 - - 84:46:fe:89:b7:25 - - 84:46:fe:89:b7:26 - - 84:46:fe:89:b7:27 - - 84:46:fe:89:b7:28 - - 84:46:fe:89:b7:29 -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s48p-256g--a71 b/hosts/taishan200-2280-2s48p-256g--a71 deleted file mode 100644 index dde6205..0000000 --- a/hosts/taishan200-2280-2s48p-256g--a71 +++ /dev/null @@ -1,13 +0,0 @@ -nr_node: 4 -nr_cpu: 96 -memory: 256G -mac_addr: - - 44:67:47:d7:6d:e6 - - 44:67:47:d7:6d:e7 - - 44:67:47:d7:6d:e8 - - 44:67:47:d7:6d:e9 - - 44:67:47:d7:6d:ea - - 44:67:47:d7:6d:eb - - 44:67:47:d7:6d:ec - - 44:67:47:d7:6d:ed -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s64p-256g--a10 b/hosts/taishan200-2280-2s64p-256g--a10 deleted file mode 100644 index 2784a67..0000000 --- a/hosts/taishan200-2280-2s64p-256g--a10 +++ /dev/null @@ -1,20 +0,0 @@ -nr_node: 4 -nr_cpu: 128 -memory: 256G -nr_ssd_partitions: 1 -ssd_partitions: - - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB44333 -mac_addr: - - 44:67:47:86:2c:95 - - 44:67:47:86:2c:96 - - 44:67:47:86:2c:97 - - 44:67:47:86:2c:98 - - ac:b3:b5:1e:d6:8b - - ac:b3:b5:1e:d6:8c - - ac:b3:b5:1e:d6:8d - - ac:b3:b5:1e:d6:8e - - ac:b3:b5:1e:d5:4b - - ac:b3:b5:1e:d5:4c - - ac:b3:b5:1e:d5:4d - - ac:b3:b5:1e:d5:4e -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s64p-256g--a3 b/hosts/taishan200-2280-2s64p-256g--a3 deleted file mode 100644 index 4bfa8ed..0000000 --- a/hosts/taishan200-2280-2s64p-256g--a3 +++ /dev/null @@ -1,25 +0,0 @@ -nr_node: 4 -nr_cpu: 128 -memory: 256G -nr_hdd_partitions: 1 -nr_ssd_partitions: 3 -hdd_partitions: - - /dev/disk/by-id/scsi-350000399981347d5 -ssd_partitions: - - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part1 - - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part2 - - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part3 -mac_addr: - - 68:4a:ae:ca:e8:ca - - 68:4a:ae:ca:e8:cb - - 68:4a:ae:ca:e8:cc - - 68:4a:ae:ca:e8:cd - - ac:b3:b5:1f:74:de - - ac:b3:b5:1f:74:df - - ac:b3:b5:1f:74:e0 - - ac:b3:b5:1f:74:e1 - - ac:b3:b5:1f:74:ee - - ac:b3:b5:1f:74:ef - - ac:b3:b5:1f:74:f0 - - ac:b3:b5:1f:74:f1 -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s64p-256g--a4 b/hosts/taishan200-2280-2s64p-256g--a4 deleted file mode 100644 index e0062d9..0000000 --- a/hosts/taishan200-2280-2s64p-256g--a4 +++ /dev/null @@ -1,25 +0,0 @@ -nr_node: 4 -nr_cpu: 128 -memory: 256G -nr_hdd_partitions: 1 -nr_ssd_partitions: 3 -hdd_partitions: - - /dev/disk/by-id/scsi-350000399c802cfb9 -ssd_partitions: - - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43892-part1 - - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43892-part2 - - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43892-part3 -mac_addr: - - 68:4a:ae:ca:e8:a2 - - 68:4a:ae:ca:e8:a3 - - 68:4a:ae:ca:e8:a4 - - 68:4a:ae:ca:e8:a5 - - ac:b3:b5:1f:74:6e - - ac:b3:b5:1f:74:6f - - ac:b3:b5:1f:74:70 - - ac:b3:b5:1f:74:71 - - ac:b3:b5:1f:7d:4e - - ac:b3:b5:1f:7d:4f - - ac:b3:b5:1f:7d:50 - - ac:b3:b5:1f:7d:51 -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s64p-256g--a8 b/hosts/taishan200-2280-2s64p-256g--a8 deleted file mode 100644 index 9386034..0000000 --- a/hosts/taishan200-2280-2s64p-256g--a8 +++ /dev/null @@ -1,16 +0,0 @@ -nr_node: 4 -nr_cpu: 128 -memory: 256G -nr_ssd_partitions: 1 -ssd_partitions: - - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43325 -mac_addr: - - 84:46:fe:26:12:f5 - - 84:46:fe:26:12:f6 - - 84:46:fe:26:12:f7 - - 84:46:fe:26:12:f8 - - ac:b3:b5:1e:b6:7b - - ac:b3:b5:1e:b6:7c - - ac:b3:b5:1e:b6:7d - - ac:b3:b5:1e:b6:7e -model_name: Kunpeng-920 diff --git a/hosts/taishan200-2280-2s64p-256g--a9 b/hosts/taishan200-2280-2s64p-256g--a9 deleted file mode 100644 index 19f4d75..0000000 --- a/hosts/taishan200-2280-2s64p-256g--a9 +++ /dev/null @@ -1,17 +0,0 @@ -nr_node: 4 -nr_cpu: 128 -memory: 128G -nr_hdd_partitions: 2 -hdd_partitions: - - /dev/disk/by-id/scsi-35000c500bd6a1347 - - /dev/disk/by-id/scsi-35000c500bd62acb3 -mac_addr: - - cc:05:77:fe:d4:0e - - cc:05:77:fe:d4:0f - - cc:05:77:fe:d4:10 - - cc:05:77:fe:d4:11 - - 44:a1:91:35:1a:70 - - 44:a1:91:35:1a:71 - - 44:a1:91:35:1a:72 - - 44:a1:91:35:1a:73 -model_name: Kunpeng-920 -- 2.23.0

1 0

[PATCH compass-ci] fix: helper: delete_task "-h" option and bugs
by Tong Qunfeng 30 Sep '20

30 Sep '20

when input single task_id, should use "<< task_id" not "<< [task_id]" Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- user-client/helper/redis_op_gc/delete_task.rb | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/user-client/helper/redis_op_gc/delete_task.rb b/user-client/helper/redis_op_gc/delete_task.rb index 643d2b3..6534f95 100755 --- a/user-client/helper/redis_op_gc/delete_task.rb +++ b/user-client/helper/redis_op_gc/delete_task.rb @@ -6,17 +6,21 @@ require_relative './force_delete' -if ARGV.empty? +if ARGV.empty? || ARGV.include?("-h") || ARGV.include?("--help") puts "Usage: #{__FILE__} task_id[|task_ids]" puts ' delete special task with [task_id]' - puts ' or delete special task form a [task_ids] file' + puts ' or delete special task from [task_ids] file' + puts "\n--- example ---" + puts " input: #{__FILE__} z9.11734 nolab.11342" + puts 'output: <none>' + puts ' # delete (z9.11734 nolab.11342) at taskqueue and scheduler' exit end task_ids = [] ARGV.each do |id| task_ids += IO.readlines(id) if File.exist?(id) - task_ids << [id] unless File.exist?(id) + task_ids << id unless File.exist?(id) end i = 0 -- 2.23.0

1 0

[PATCH v3 lkp-tests] fix: run-ipconfig: add ntpdat
by Wang Chenglong 30 Sep '20

30 Sep '20

initramfs start, need to synchronize time after get the ip address Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- bin/run-ipconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/run-ipconfig b/bin/run-ipconfig index 0f80feaf..ad2a9808 100755 --- a/bin/run-ipconfig +++ b/bin/run-ipconfig @@ -20,3 +20,4 @@ read_ip() [ -n "$IP" ] || read_ip configure_networking +ntpdate -u ntp.api.bz -- 2.23.0

2 2

[PATCH v4 compass-ci 6/7] container/mail-robot: conf/sources.list
by Luan Shengde 30 Sep '20

30 Sep '20

sources list file for the image install packages Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/conf/sources.list | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 container/mail-robot/conf/sources.list diff --git a/container/mail-robot/conf/sources.list b/container/mail-robot/conf/sources.list new file mode 100644 index 0000000..8179e01 --- /dev/null +++ b/container/mail-robot/conf/sources.list @@ -0,0 +1,9 @@ +deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib +deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib + +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib +deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib -- 2.23.0

1 0

[PATCH v4 compass-ci 5/7] container/mail-robot: start file
by Luan Shengde 30 Sep '20

30 Sep '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/start | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100755 container/mail-robot/start diff --git a/container/mail-robot/start b/container/mail-robot/start new file mode 100755 index 0000000..56c4259 --- /dev/null +++ b/container/mail-robot/start @@ -0,0 +1,28 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +require_relative '../defconfig.rb' +docker_rm 'mail-robot' + +JUMPER_IP = '192.168.132.2' +JUMPER_PORT = '29999' +SEND_MAIL_PORT = '11311' + +cmd = %W[ + docker run + --restart=always + --name=mail-robot + -d + -e JUMPER_IP=#{JUMPER_IP} + -e JUMPER_PORT=#{JUMPER_PORT} + -e SEND_MAIL_PORT=#{SEND_MAIL_PORT} + -v /c/upstream-repos:/upstream-repos:ro + -v /srv/cci/Maildir:/Maildir:rw + -v /c/compass-ci:/compass-ci:ro + mail-robot +] + +cmd += ['bash', '-c', 'umask 002 && ruby ./mail-robot.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci] doc/code-spec: add 文档与易用性
by Wu Fengguang 30 Sep '20

30 Sep '20

From: Wu Fengguang <wufengguang(a)huawei.com> Signed-off-by: Wu Fengguang <wufengguang(a)huawei.com> --- doc/code-spec.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index e783a0c..9805182 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -40,3 +40,16 @@ here: 2) 描述你的改动 3) 拆分你的改动 8）回复评审意见 + +文档与易用性 +============ + +- 每个feature都应当有 + - 用户文档 + - 设计文档 + - 指示对应的代码在哪里(class/func); 或者如果有多处代码互相配合, 指示用什么关键字grep能把它们串起来看 + +- 每个有命令行参数的脚本, 都应当支持help output (-h|--help|参数不够时显示帮助信息) + - dependencies/assumptions + - input parameters + - input/output example -- 2.23.0

1 0

[PATCH compass-ci] feat: helper: snapshot of taskqueue
by Tong Qunfeng 30 Sep '20

30 Sep '20

Give a snapshot information about taskqueue (data at redis sorted set queue "queues/id2content"). It will generate a file named taskqueue_snap_yyyymmdd.md. snapshot content examples: |id---|add_time----------|current queue name| 14703, 2020-07-22 14:33, sched/vm-hi1620-2p8g-xxx/in_process 14831, 2020-07-22 17:51, sched/vm-hi1620-2p8g-xxx/in_process crystal.89630, 2020-09-23 11:19, sched/vm-hi1620-2p8g/in_process crystal.89667, 2020-09-23 11:20, sched/vm-hi1620-2p8g/in_process Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- .../helper/redis_op_gc/taskqueue_snap.rb | 44 +++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100755 user-client/helper/redis_op_gc/taskqueue_snap.rb diff --git a/user-client/helper/redis_op_gc/taskqueue_snap.rb b/user-client/helper/redis_op_gc/taskqueue_snap.rb new file mode 100755 index 0000000..ebf32b5 --- /dev/null +++ b/user-client/helper/redis_op_gc/taskqueue_snap.rb @@ -0,0 +1,44 @@ +#!/usr/bin/env ruby + +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require_relative './basic_env' + +# redis hash key "queues/id2content" is used by taskqueue +cmd = "#{CMD_BASE} queues/id2content , hgetall" +result = `#{cmd}`.chomp +results = result.split("\n") + +task_info = [] + +# format time to "2020-09-29 14:23" +i = 0 +task_num = results.size / 2 +while i < task_num + task_id = results[i] + task_content = JSON.parse(results[i + 1]) + + time = task_content['add_time'] + task_content['add_time'] = Time.at(time.to_f).strftime('%Y-%m-%d %H:%M') unless time.nil? + + time = task_content['move_time'] + task_content['move_time'] = Time.at(time.to_f).strftime('%Y-%m-%d %H:%M') unless time.nil? + + task_info << { 'id' => task_id, 'value' => task_content } + + i += 2 + set_progress(i, task_num) +end + +# sort and put short information to file. +task_info.sort! { |a, b| a['value']['add_time'] <=> b['value']['add_time'] } +today = Time.now.strftime('%Y%m%d') +File.open("taskqueue_snap_#{today}.md", 'w') do |f| + task_info.each do |task| + f.write(task['id'] + ', ' + task['value']['add_time'] \ + + ', ' + task['value']['queue'] + "\n") + end +end -- 2.23.0

2 1

[PATCH compass-ci] modify the origin address for fluentd-base image
by Xiao Shenwei 30 Sep '20

30 Sep '20

error message: ERROR: libcurl-7.67.0-r0: package mentioned in index not found refer to: https://mirrors.huaweicloud.com/alpine Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/fluentd-base/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/fluentd-base/Dockerfile b/container/fluentd-base/Dockerfile index ef671c2..62f55fd 100644 --- a/container/fluentd-base/Dockerfile +++ b/container/fluentd-base/Dockerfile @@ -5,7 +5,7 @@ FROM alpine:3.11 MAINTAINER Wu Zhende <wuzhende666(a)163.com> -RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|https://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk update -- 2.23.0

3 5

[PATCH v2 compass-ci] sparrow: start the firewall in adapting openeuler
by Hu Xuejiao 30 Sep '20

30 Sep '20

When firewalld is inactive, return code of 'systemctl status firewalld' will be 3, and install-tiny has set -e, so this will cause this process exit. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/2-network/iptables | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..b4dc2d2 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -12,15 +12,18 @@ iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE -systemctl status firewalld | grep -q "running" && { +if command -v "firewall-cmd" > /dev/null; then + systemctl start firewalld && { + DOCKER0_IFACE=docker + DOCKER0_SUBNET=172.17.0.0/16 - DOCKER0_IFACE=docker0 - DOCKER0_SUBNET=172.17.0.0/16 + iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE + iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE - iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE - iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE - - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -} + firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" + firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" + firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" + } +else + echo "firewall.service could not be found" +fi -- 2.23.0

3 3

[PATCH compass-ci 2/4] sparrow: add domain name resolution for Docker images
by Hu Xuejiao 30 Sep '20

30 Sep '20

When executing ./crystal-shards/build appear error: http://mirrors.huaweicloud.com/alpine/v3.11/main: temporary error(try again later) Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/3-code/dev-env | 3 +++ sparrow/3-code/os/debian | 4 ++++ 2 files changed, 7 insertions(+) create mode 100644 sparrow/3-code/os/debian diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index ae267bd..80ee2cf 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -32,3 +32,6 @@ export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +source /etc/os-release + +. $(dirname ${BASH_SOURCE[0]})/os/${ID} diff --git a/sparrow/3-code/os/debian b/sparrow/3-code/os/debian new file mode 100644 index 0000000..171b979 --- /dev/null +++ b/sparrow/3-code/os/debian @@ -0,0 +1,4 @@ +#!/bin/bash +cat >> /etc/default/docker <<EOF +DOCKER_OPTS="--dns 114.114.114.114" +EOF -- 2.23.0

2 1

[PATCH compass-ci 1/4] container: consider the situation that GITCACHE_HOST is undefined
by Hu Xuejiao 30 Sep '20

30 Sep '20

Error Msg: Step 5/12 : RUN [ -n "$GITCACCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig ---> Running in 52ffe9f8ecd1 The command '/bin/sh -c [ -n "$GITCACHE_HOST" ] && echo -e "[url\"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig' reton-zero code: 1 Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- container/crystal-shards/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/crystal-shards/Dockerfile b/container/crystal-shards/Dockerfile index 0c5713c..bdc5b66 100644 --- a/container/crystal-shards/Dockerfile +++ b/container/crystal-shards/Dockerfile @@ -7,7 +7,7 @@ MAINTAINER chief <taxcom(a)tom.com> ARG GITCACHE_HOST ARG GITCACHE_PORT -RUN [ -n "$GITCACHE_HOST" ] && echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig +RUN [ -z "$GITCACHE_HOST" ] || echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig copy shard.yml /usr/share/crystal/app/shard.yml -- 2.23.0

2 1

[PATCH compass-ci 1/2] feat(dnsmasq): allow the use of custom dnsmasq conf file
by Yu Chuan 29 Sep '20

29 Sep '20

The custom dnsmasq conf files should be placed under the follow dir: /etc/compass-ci/dnsmasq.d/ Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/dnsmasq/start | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/container/dnsmasq/start b/container/dnsmasq/start index 3cad801ec81c..8e0e05c953fb 100755 --- a/container/dnsmasq/start +++ b/container/dnsmasq/start @@ -8,6 +8,10 @@ docker_rm dnsmasq kill $(ps -ef| grep dnsmasq| grep libvirt| awk '{print $2}') 2> /dev/null +# load custom dnsmasq.conf +[ -d "/etc/compass-ci/dnsmasq.d/" ] && + cp /etc/compass-ci/dnsmasq.d/* ${PWD}/dnsmasq.d/ + cmd=( docker run --restart=always -- 2.23.0

4 5

[PATCH v2 compass-ci] sparrow: delete the same name install package
by Zhang Yu 29 Sep '20

29 Sep '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- sparrow/0-package/os/debian | 1 - 1 file changed, 1 deletion(-) diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 8301500..68423d7 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -12,9 +12,8 @@ pkgs=( make docker.io ruby-full ruby-dev - bridge-utils qemu dmidecode lockfile-progs bridge-utils -- 2.23.0

1 0

[PATCH compass-ci 4/4] sparrow: open the masquerade at public zone
by Hu Xuejiao 29 Sep '20

29 Sep '20

Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/2-network/iptables | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 831c1a3..0af422e 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -32,3 +32,4 @@ iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERAD firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" +firewall-cmd --zone=public --add-masquerade -- 2.23.0

1 0

[PATCH compass-ci] add "host" field to job
by Zhang Yuhang 29 Sep '20

29 Sep '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/job.cr | 22 ++++++++++++++++++++++ src/lib/sched.cr | 1 + 2 files changed, 23 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 53a560c..5008638 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -28,6 +28,19 @@ module JobHelper return tbox_group end + def self.match_host(testbox : String) + # "xxx-xxx-123" remove "-123" + find = testbox.match(/(.*)-\d+$/) + if testbox[0, 3] == "vm-" + # "vm-xxx-xxx--xxx" remove "--xxx" + # "vm-xxx-2xx-xxx" remove "-xxx" + find = testbox.match(/(.*)--.+$/) || + testbox.match(/(.*-\d\w*)-[a-zA-Z]+$/) || + find + end + find ? find[1] : testbox + end + def self.service_path(path) temp_path = File.real_path(path) return temp_path.split("/srv")[-1] @@ -161,6 +174,7 @@ class Job private def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group + update_host_from_testbox # id must exists, need update host date = Time.local.to_s("%F") self["result_root"] = "/result/#{suite}/#{tbox_group}/#{date}/#{id}" @@ -186,6 +200,14 @@ class Job end end + # if not assign host, set it to a match result from testbox + # if job special host, should we just set host=host + private def update_host_from_testbox + if self["host"] == "" + @hash["host"] = JSON::Any.new(JobHelper.match_host(testbox)) + end + end + def [](key : String) : String "#{@hash[key]?}" end diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 15f5c9e..4e71959 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -226,6 +226,7 @@ class Sched # add to job content when multi-test job["testbox"] = tbox_group job.update_tbox_group(tbox_group) + job.update_host(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") job["node_macs"] = config["macs"].as_a.join(" ") -- 2.23.0

1 0

help star 2020-09-25
by Wu Fengguang 29 Sep '20

29 Sep '20

Jihui is our help star, thanks to kind and helpful Jihui! wfg@crystal ~/todo% bin/help-star people/*/credit/2020-09-25.md 25.7 Shi Zhichao 21.9 Wu Fengguang 12.1 Wei Jihui 11.0 Zhang Yu 10.8 Yu Chuan 10.2 Du Kaitian 9.7 Lu Weitao 9.2 Wang Chenglong 8.6 Wu Zhende 7.7 Bai Jing 7.7 Wang Yong 7.7 Zhang Dewan 6.5 Sun Yukui 6.5 Cui Yili 6.1 Cao Xueliang 5.2 Li Yuanchao 5.1 Zhou Shengrui 4.7 Xu Xijian 4.5 Tong Qunfeng 4.5 Xiao Shenwei 4.3 Zhang Yale 3.9 Zhang Yuhang 3.7 Luan Shengde 3.0 Liu Shaofei 2.8 Liu Yinsi 2.4 Lu Kaiyi 1.6 Lin Jiaxin 1.3 Hu Xuejiao 1.0 Li Ping 0.9 Ren Wen 0.0 Ren Zhiwen 0.0 Sun Lijun 0.0 Liang Tao 0.0 Wang Shuai 0.0 Cui Xucui 0.0 Xie Zheng

1 0

review star 2020-09-29
by Wu Fengguang 29 Sep '20

29 Sep '20

Yuchuan is our review star, thanks Yuchuan! wfg@crystal ~/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 401 212 425 75 Wu Fengguang 129 23 56 27 Tong Qunfeng 28 17 52 12 Shi Zhichao 49 22 43 24 Du Kaitian 67 14 41 10 Yu Chuan 28 18 27 7 Xu Xijian 41 12 21 7 Xiao Shenwei 24 14 20 5 Wei Jihui 21 11 20 6 Li Yuanchao 20 10 15 0 Zhang Yu 8 8 14 2 Sun Yukui 32 10 14 1 Liu Yinsi 54 5 11 3 Hu Xuejiao 33 4 10 1 Cao Xueliang 90 5 9 1 Luan Shengde 44 7 8 0 Wu Zhende 13 5 7 0 Xie Zheng 116 5 7 0 Ren Wen 13 7 7 0 Wang Yong 9 3 6 3 Zhou Shengrui 17 3 6 0 Liu Shaofei 69 1 5 1 Lu Weitao 67 1 5 3 Zhang Yuhang 20 3 3 1 Zhang Yale 29 1 3 1 Li Ping 30 1 2 6 Cui Yili 15 6 2 2 Lu Kaiyi 13 1 1 0 Lin Jiaxin 29 1 1 0 Wang Chenglong 1 0 0 0 Liang Tao 8 0 0 0 Zhang Dewan 22 3 0 1 Bai Jing

1 0

help star 2020-09-18
by Wu Fengguang 29 Sep '20

29 Sep '20

Jihui is our help star, thanks to kind and helpful Jihui! wfg@crystal ~/todo% bin/help-star people/*/credit/2020-09-18.md 32.6 Wu Fengguang 20.1 Shi Zhichao 18.1 Wei Jihui 17.6 Xu Xijian 15.1 Bai Jing 13.4 Yu Chuan 12.2 Zhang Dewan 11.8 Lu Weitao 10.4 Tong Qunfeng 9.8 Xiao Shenwei 9.6 Cui Yili 9.0 Liu Shaofei 8.5 Du Kaitian 8.4 Liu Yinsi 7.7 Zhang Yu 7.5 Wu Zhende 7.3 Wang Yong 7.1 Cao Xueliang 6.9 Sun Yukui 6.6 Hu Xuejiao 6.5 Wang Chenglong 6.1 Li Yuanchao 5.8 Zhou Shengrui 4.7 Lin Jiaxin 3.6 Liang Tao 3.5 Luan Shengde 3.1 Lu Kaiyi 2.5 Zhang Yale 1.4 Lu Shengde 1.2 Ren Wen 1.1 Zhang Yuhang 0.8 Li Ping 0.0 Ren Zhiwen 0.0 Sun Lijun 0.0 Wang Shuai 0.0 Cui Xucui 0.0 Xie Zheng

1 0

git count 2020-09-29
by Wu Fengguang 29 Sep '20

29 Sep '20

Renwen is code star of the week, thanks Renwen! wfg@crystal ~/todo% bin/git-count 11 40 Ren Wen 18 Wu Fengguang 12 Lu Weitao 11 Wu Zhende 10 Hu Xuejiao 8 Zhang Yuhang 7 Cao Xueliang 7 Liu Yinsi 7 Xiao Shenwei 6 Yu Chuan 5 Bai Jing 5 Zhang Yu 4 Xu Xijian 4 Wei Jihui 4 Cui Yili 4 Wang Yong 4 Li Yuanchao 3 Wang Chenglong 3 Luan Shengde 3 Lin Jiaxin 3 Xie Zheng 2 Zhang Yale 2 Li Ping 2 Lu Kaiyi 1 Du Kaitian 1 Liu Shaofei 1 Liang Tao 1 Zhang Dewan 0 chief 0 Ren Zhiwen 0 Sun Lijun 0 Sun Yukui 0 Shi Zhichao 0 Zhou Shengrui

1 0

[PATCH v2 compass-ci] sparrow/0-package: delete the repeate install package
by Zhang Yu 29 Sep '20

29 Sep '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- sparrow/0-package/os/debian | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 8301500..ef90e50 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -13,11 +13,10 @@ pkgs=( docker.io ruby-full ruby-dev - bridge-utils qemu + bridge-utils dmidecode lockfile-progs - bridge-utils ) apt-get install -y "${pkgs[@]}" -- 2.23.0

3 2

[PATCH compass-ci] git_mirror: feat: reload fork_info when /upstream-repos update
by Li Yuanchao 29 Sep '20

29 Sep '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index ad27880..88a94ac 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -147,6 +147,8 @@ class MirrorMain end def send_message(feedback_info) + feedback_info.merge!(@git_info[feedback_info[:git_repo]]) + feedback_info.delete(:cur_refs) message = feedback_info.to_json @message_queue.publish(message) end @@ -158,12 +160,12 @@ class MirrorMain @fork_stat[feedback_info[:git_repo]][:queued] = false return unless feedback_info[:possible_new_refs] + return reload_fork_info if feedback_info[:git_repo] == 'upstream-repos/upstream-repos' + new_refs = check_new_refs(feedback_info[:git_repo]) return if new_refs[:heads].empty? feedback_info[:new_refs] = new_refs - feedback_info.merge!(@git_info[feedback_info[:git_repo]]) - feedback_info.delete(:cur_refs) send_message(feedback_info) end @@ -219,4 +221,27 @@ class MirrorMain @git_info[git_repo][:cur_refs] = cur_refs return new_refs end + + def reload_fork_info + upstream_repos = 'upstream-repos/upstream-repos' + if @git_info[upstream_repos][:cur_refs].empty? + @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) + else + old_commit = @git_info[upstream_repos][:cur_refs][:heads]['refs/heads/master'] + new_refs = check_new_refs(upstream_repos) + new_commit = new_refs[:heads]['refs/heads/master'] + changed_files = %x(git -C /srv/git/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) + reload(changed_files) + end + end + + def reload(file_list) + system("git -C #{REPO_DIR} pull") + file_list.each_line do |file| + next if File.basename(file) == '.ignore' + + repo_dir = "#{REPO_DIR}/#{file}" + load_repo_file(repo_dir, File.dirname(file), File.basename(file)) if File.file?(repo_dir) + end + end end -- 2.23.0

2 1

[PATCH v4 compass-ci] fix: sched: submit idle jobs
by Ren Wen 29 Sep '20

29 Sep '20

1) add field 'idle_job' for idle jobs, value is always 'true'. 2) add two fields 'FLUENTD_SERVER_*' for monitoring logs of virtual machines started by physical machine. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 15f5c9e..309df04 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -468,9 +468,16 @@ class Sched def auto_submit_idle_job(tbox_group) full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" + extra_job_fields = [ + "idle_job=true", + "FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", + "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}", + ] + Jobfile::Operate.auto_submit_job( full_path_patterns, - "testbox: #{tbox_group}") if Dir.glob(full_path_patterns).size > 0 + "testbox: #{tbox_group}", + extra_job_fields) if Dir.glob(full_path_patterns).size > 0 end private def add_kernel_console_param(arch_tmp) @@ -520,7 +527,7 @@ class Sched "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" else response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" end response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" -- 2.23.0

1 0

[PATCH compass-ci] check path of the upstream_repo under /srv/git/
by Li Ping 29 Sep '20

29 Sep '20

--- lib/git.rb | 28 +++++++++++++++++++++++++++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/lib/git.rb b/lib/git.rb index db97c47..a181ec9 100644 --- a/lib/git.rb +++ b/lib/git.rb @@ -3,8 +3,34 @@ # wrap common git commands class GitCommit def initialize(repo, commit) - @git_prefix = "git -C /srv/git/#{repo}.git" + @repo = repo @commit = commit + @git_prefix = cd_repo_path + end + + def check_repo + fork_name = @repo.split('/')[0] + project = @repo.split('/')[1] + first_letter = project[0] + fork_name = 'linus' if project == 'linux' + path_a = find_path("/srv/git/#{first_letter}/#{project}/#{project}.git") + path_b = find_path("/srv/git/#{first_letter}/#{project}/#{fork_name}.git") + path_c = find_path("/srv/git/#{project}/#{fork_name}.git") + path_d = find_path("/srv/git/#{project}/#{project}.git") + path_a || path_b || path_c || path_d + end + + def find_path(path) + return false if Dir.glob(path).empty? + + Dir.glob(path).first + end + + def cd_repo_path + find_repo = check_repo + return unless find_repo + + "git -C #{find_repo}" end def author_name -- 2.23.0

5 13

[PATCH v3 compass-ci] fix: sched: submit idle jobs
by Ren Wen 29 Sep '20

29 Sep '20

1) submit all idles jobs once use '*' -> submit each idle job once until all done. 2) add two fields into job if job's filename is 'multi-qemu.yaml'. 3) add field 'idle_job' for idle jobs, value is always 'true'. Two extra fields are used for: monitoring logs of virtual machines started by physical machine. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 15f5c9e..6ca20d7 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -467,10 +467,18 @@ class Sched end def auto_submit_idle_job(tbox_group) - full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" - Jobfile::Operate.auto_submit_job( - full_path_patterns, - "testbox: #{tbox_group}") if Dir.glob(full_path_patterns).size > 0 + idle_job_path = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/" + + return nil unless Dir.exists?(idle_job_path) + jobs = Dir[idle_job_path + "*.yaml"] + jobs.each do |job| + extra_job_fields = ["idle_job=true"] + if Path[job].basename == "multi-qemu.yaml" + extra_job_fields += ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", + "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}"] + end + Jobfile::Operate.auto_submit_job(job, "testbox: #{tbox_group}", extra_job_fields) + end end private def add_kernel_console_param(arch_tmp) @@ -520,7 +528,7 @@ class Sched "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" else response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" end response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" -- 2.23.0

2 3

[PATCH compass-ci] fix: sched: submit idle jobs
by Ren Wen 29 Sep '20

29 Sep '20

1) submit all idles jobs once use '*' -> submit each idle job once until all done. 2) add two fields into job if job's name is multi-qemu. job's name 'multi-qemu' is case-insensitive matching now. 3) `.yaml` and `.yml` can both match. Two extra fields are used for: monitoring logs of virtual machines started by physical machine. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 15f5c9e..c27dd99 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -467,10 +467,20 @@ class Sched end def auto_submit_idle_job(tbox_group) - full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" - Jobfile::Operate.auto_submit_job( - full_path_patterns, - "testbox: #{tbox_group}") if Dir.glob(full_path_patterns).size > 0 + idle_job_path = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/" + + if Dir.exists?(idle_job_path) + jobs = Dir[idle_job_path + "*.y{a,}ml"] + extra_job_fields = ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", + "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}"] + + jobs.each do |job| + if Path[job].basename =~ /^multi-qemu\.ya?ml/i + Jobfile::Operate.auto_submit_job(job, "testbox: #{tbox_group}", extra_job_fields) + end + Jobfile::Operate.auto_submit_job(job, "testbox: #{tbox_group}") + end + end end private def add_kernel_console_param(arch_tmp) @@ -520,7 +530,7 @@ class Sched "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" else response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" end response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" -- 2.23.0

3 10

[PATCH compass-ci] check if author's email address exist in upstream repos
by Li Ping 29 Sep '20

29 Sep '20

--- lib/git.rb | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/lib/git.rb b/lib/git.rb index a181ec9..4939c94 100644 --- a/lib/git.rb +++ b/lib/git.rb @@ -2,7 +2,7 @@ # wrap common git commands class GitCommit - def initialize(repo, commit) + def initialize(repo, commit = nil) @repo = repo @commit = commit @git_prefix = cd_repo_path @@ -28,32 +28,42 @@ class GitCommit def cd_repo_path find_repo = check_repo - return unless find_repo + message = "The repo is not found in the upstream-repo" + + raise StandardError, message unless find_repo "git -C #{find_repo}" end + def check_author_in_upstream_repos(email) + author_email = %x(#{@git_prefix} log | grep '^Author'| grep #{email}).chomp + + message = "The author's email address is not found in the upstream-repo" + + raise StandardError, message if author_email.empty? + end + def author_name - `#{@git_prefix} log -n1 --pretty=format:'%an' #{(a)commit}`.chomp + %x(#{git_prefix} log -n1 --pretty=format:'%an' #{(a)commit}).chomp end def author_email - `#{@git_prefix} log -n1 --pretty=format:'%ae' #{(a)commit}`.chomp + %x(#{@git_prefix} log -n1 --pretty=format:'%ae' #{(a)commit}).chomp end def subject - `#{@git_prefix} log -n1 --pretty=format:'%s' #{(a)commit}`.chomp + %x(#{@git_prefix} log -n1 --pretty=format:'%s' #{(a)commit}).chomp end def commit_time - `#{@git_prefix} log -n1 --pretty=format:'%ci' #{(a)commit}`.chomp + %x(#{@git_prefix} log -n1 --pretty=format:'%ci' #{(a)commit}).chomp end def url - `#{@git_prefix} remote -v`.split[1] + %x(#{@git_prefix} remote -v).split[1] end def diff - `#{@git_prefix} diff -1 #{@commit}~..#{@commit}`.chomp + %x(#{@git_prefix} diff -1 #{@commit}~..#{@commit}).chomp end end -- 2.23.0

1 0

[PATCH compass-ci] sparrow: ignore containers when deploy
by Liu Yinsi 29 Sep '20

29 Sep '20

scheduler-https, conserver, web-backend-nginx no need to deploy. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 230c6ac..7698c08 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -29,7 +29,10 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd $container - [ "$container_name" == 'sub-fluentd' ] && exit + [ "$container_name" == 'sub-fluentd' ] || \ + [ "$container_name" == 'conserver' ] || \ + [ "$container_name" == 'web-backend-nginx' ] || \ + [ "$container_name" == 'scheduler-https' ] && exit [ -x build ] && ./build [ -x install ] && ./install [ -x first-run ] && ./first-run -- 2.23.0

3 4

[PATCH v3 compass-ci 2/2] container/extract-stats: add 'LAB' in environment variables
by Lu Weitao 29 Sep '20

29 Sep '20

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/extract-stats/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/extract-stats/start b/container/extract-stats/start index 25bdd25..6bb9f10 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -15,6 +15,7 @@ names = Set.new %w[ FLUENTD_PORT MAIL_HOST MAIL_PORT + LAB ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] src/extract-stats: fix can not send delimiter task
by Lu Weitao 29 Sep '20

29 Sep '20

problem: stats_worker call add_task() of TaskQueueAPI to send delimiter task, but no task added in delimiter task queue. cause analysis: the latest TaskQueueAPI.add_task(service_queue_path, task), require {"lab" => $LAB} in task content, otherwise return error message: "409, need lab in task content" in fact, we miss "lab", so can not send task to task queue. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..fcba6af 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -76,7 +76,8 @@ class StatsWorker unless new_error_ids.empty? STDOUT.puts "send a delimiter task: job_id is #{job_id}" @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => new_error_ids.sample, - "job_id" => job_id}.to_json)) + "job_id" => job_id, + "lab" => LAB}.to_json)) end msg = %({"job_id": "#{job_id}", "job_state": "extract_finished"}) system "echo '#{msg}'" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] container/extract-stats prefix set lab in ENV
by Lu Weitao 29 Sep '20

29 Sep '20

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/extract-stats/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/extract-stats/start b/container/extract-stats/start index 25bdd25..6bb9f10 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -15,6 +15,7 @@ names = Set.new %w[ FLUENTD_PORT MAIL_HOST MAIL_PORT + LAB ] defaults = relevant_defaults(names) -- 2.23.0

2 5

[PATCH compass-ci 2/2] sparrow: run 3-code/os/debian
by Hu Xuejiao 29 Sep '20

29 Sep '20

Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/3-code/dev-env | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index ae267bd..80ee2cf 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -32,3 +32,6 @@ export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +source /etc/os-release + +. $(dirname ${BASH_SOURCE[0]})/os/${ID} -- 2.23.0

2 1

[PATCH compass-ci] sparrow: delete the same name install package
by Zhang Yu 29 Sep '20

29 Sep '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- sparrow/0-package/os/debian | 1 - 1 file changed, 1 deletion(-) diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 8301500..68423d7 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -13,7 +13,6 @@ pkgs=( docker.io ruby-full ruby-dev - bridge-utils qemu dmidecode lockfile-progs -- 2.23.0

2 2

[PATCH compass-ci 4/4] container: consider the situation that is no GITCACHE_HOST
by Hu Xuejiao 29 Sep '20

29 Sep '20

When manually ./build in VM, the GITCACHE_HOST variable is not found. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- container/crystal-shards/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/crystal-shards/Dockerfile b/container/crystal-shards/Dockerfile index 0c5713c..bdc5b66 100644 --- a/container/crystal-shards/Dockerfile +++ b/container/crystal-shards/Dockerfile @@ -7,7 +7,7 @@ MAINTAINER chief <taxcom(a)tom.com> ARG GITCACHE_HOST ARG GITCACHE_PORT -RUN [ -n "$GITCACHE_HOST" ] && echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig +RUN [ -z "$GITCACHE_HOST" ] || echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig copy shard.yml /usr/share/crystal/app/shard.yml -- 2.23.0

3 3

[PATCH compass-ci 1/4] sparrow: start the firewall in adapting openeuler
by Hu Xuejiao 29 Sep '20

29 Sep '20

When firewalld is inactive, return code of 'systemctl status firewalld' will be 3, and install-tiny has set -e, so this will cause this process exit. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/2-network/iptables | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..a9c6cf8 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -12,15 +12,19 @@ iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE -systemctl status firewalld | grep -q "running" && { +systemctl start firewalld - DOCKER0_IFACE=docker0 - DOCKER0_SUBNET=172.17.0.0/16 +[ "$(systemctl is-active firewalld)" == "active" ] || { + echo "firewalld start failed" + exit 0 +} - iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE - iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE +DOCKER0_IFACE=docker0 +DOCKER0_SUBNET=172.17.0.0/16 - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -} +iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE +iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE + +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -- 2.23.0

3 3

[PATCH compass-ci] lib/commit_status.rb: optimize jobs_list extract
by Lu Kaiyi 29 Sep '20

29 Sep '20

optimize jobs_list extract and reduce unnecessary reliance on params_group.rb Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/commit_status.rb | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/commit_status.rb b/lib/commit_status.rb index 19473f4..81aeb29 100644 --- a/lib/commit_status.rb +++ b/lib/commit_status.rb @@ -4,7 +4,6 @@ require_relative './constants' require_relative './es_query' -require_relative './params_group' def query_commit_status(job, error_id) items = { @@ -29,8 +28,8 @@ end def query_jobs_from_es(items) es = ESQuery.new(ES_HOST, ES_PORT) result = es.multi_field_query items - jobs = result['hits']['hits'] - jobs_list = extract_jobs_list(jobs) + jobs_list = result['hits']['hits'] + jobs_list.map! { |job| job['_source'] } return jobs_list end -- 2.23.0

1 0

[PATCH compass-ci 1/2] sparrow: add domain name resolution for Docker images
by Hu Xuejiao 29 Sep '20

29 Sep '20

When executing ./crystal-shards/build appear error: http://mirrors.huaweicloud.com/alpine/v3.11/main: temporary error(try again later) Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/3-code/os/debian | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 sparrow/3-code/os/debian diff --git a/sparrow/3-code/os/debian b/sparrow/3-code/os/debian new file mode 100644 index 0000000..171b979 --- /dev/null +++ b/sparrow/3-code/os/debian @@ -0,0 +1,4 @@ +#!/bin/bash +cat >> /etc/default/docker <<EOF +DOCKER_OPTS="--dns 114.114.114.114" +EOF -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: change tbox_group filter regex
by Zhang Yuhang 29 Sep '20

29 Sep '20

change to remove /-(\d+)$/ Exmaple: "taishan-xxx--xxx-12" remove "-12" Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 4544690..419ef86 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -41,16 +41,19 @@ def es_count(query) ES_CLIENT.count(index: 'jobs*', body: query)['count'] end -# "vm-hi1620-2p8g-212" remove "-212" -# "vm-hi1620-2p8g-zzz" remove "-zzz" -# "vm-git-bisect" don't remove "-bisect" +# vm-hi1620-2p4g-12 remove -12 +# taishan-xxx--xxx-12 remove -12 +# vm-hi1620-2p4g--0x remove --0x +# vm-hi1620-2p4g-abc remove -abc +# vm-git-bisect don't remove -bisect def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ - index = r.index('--') || r.rindex('-') - r = r[0, index] + remove_index = r =~ /-(\d+)$/ + if r[0, 3] == 'vm-' + remove_index = r =~ /(--.+$)|(-\d\w*-[a-zA-Z]+$)/ || remove_index end + r = r[0, remove_index] if remove_index result.add r end result.to_a @@ -95,7 +98,7 @@ def compare_candidates_body OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, - { os: 'debian', os_version: ['10', 'sid'] }, + { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], os_arch: %w[aarch64 x86_64], -- 2.23.0

2 2

[PATCH v2 compass-ci 1/2] src/extract-stats fix can not send delimiter task
by Lu Weitao 29 Sep '20

29 Sep '20

cause analysis: return 409 when add_task without "lab" in input parameters Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..fcba6af 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -76,7 +76,8 @@ class StatsWorker unless new_error_ids.empty? STDOUT.puts "send a delimiter task: job_id is #{job_id}" @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => new_error_ids.sample, - "job_id" => job_id}.to_json)) + "job_id" => job_id, + "lab" => LAB}.to_json)) end msg = %({"job_id": "#{job_id}", "job_state": "extract_finished"}) system "echo '#{msg}'" -- 2.23.0

2 2

[PATCH compass-ci] feat: helper: delete_task
by Tong Qunfeng 29 Sep '20

29 Sep '20

Usage: delete_task task_id[|task_ids file] case A (delete task_ids) - helper/redis_op_gc/consistency.rb -o : output manually delete task to man_del.id, and live too long task to too_long.id - then use delete_task man_del.id too_long.id : delete all this task case B (delete a signle task) - helper/1_showjob.sh task_id : show the job content in es. - delete_task task_id : delete this task Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- user-client/helper/redis_op_gc/delete_task.rb | 36 ++++++++++++++++ .../helper/redis_op_gc/force_delete.rb | 41 +++++++++++++++++++ 2 files changed, 77 insertions(+) create mode 100755 user-client/helper/redis_op_gc/delete_task.rb create mode 100644 user-client/helper/redis_op_gc/force_delete.rb diff --git a/user-client/helper/redis_op_gc/delete_task.rb b/user-client/helper/redis_op_gc/delete_task.rb new file mode 100755 index 0000000..4a67a2b --- /dev/null +++ b/user-client/helper/redis_op_gc/delete_task.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby + +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './force_delete' + +if ARGV.size < 1 + puts "Usage: #{__FILE__} task_id[|task_ids]" + puts " delete special task with [task_id]" + puts " or delete special task form a [task_ids] file" + exit +end + +ARGV.each do |id| + flag = FileTest::exists?(id) + if flag == true + task_ids = IO.readlines(id) + else + task_ids = [id] + end + + i = 0 + task_num = task_ids.size + while i < task_num + task_id = task_ids[i] + fd = ForceDelete.new(task_id) + fd.run + + i += 1 + set_progress(i, task_num) + end + + puts "" +end diff --git a/user-client/helper/redis_op_gc/force_delete.rb b/user-client/helper/redis_op_gc/force_delete.rb new file mode 100644 index 0000000..d8e973a --- /dev/null +++ b/user-client/helper/redis_op_gc/force_delete.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require_relative './basic_env' + +# force delete for job (with job_id) +# - taskqueue hash key: "queues/sched/*/reday?in_process?idle?uuid..." +# - taskqueue hash key: "queues/id2content" +# - scheduler hash key: "sched/id2job" +class ForceDelete + def initialize(id) + @task_id = id + end + + def run + fd4jobqueue + gc4taskqueue + gc4scheduler + end + + def fd4jobqueue + task_content = get_taskqueue_content4id(nil, @task_id) + if task_content + queue_name = task_content["queue"] + cmd = "#{CMD_BASE} queues/#{queue_name} , zrem #{@task_id}" + `#{cmd}`.chomp + end + end + + def gc4taskqueue + cmd = "#{CMD_BASE} queues/id2content , hdel #{@task_id}" + `#{cmd}`.chomp + end + + def gc4scheduler + cmd = "#{CMD_BASE} sched/id2job , hdel #{@task_id}" + `#{cmd}`.chomp + end +end -- 2.23.0

2 2

[PATCH v2 compass-ci] feat: helper: delete_task
by Tong Qunfeng 29 Sep '20

29 Sep '20

Usage: delete_task task_id[|task_ids file] case A (delete task_ids) - helper/redis_op_gc/consistency.rb -o : output manually delete task to man_del.id, and live too long task to too_long.id - then use delete_task man_del.id too_long.id : delete all this task case B (delete a signle task) - helper/1_showjob.sh task_id : show the job content in es. - delete_task task_id : delete this task Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- user-client/helper/redis_op_gc/delete_task.rb | 31 ++++++++++++++ .../helper/redis_op_gc/force_delete.rb | 41 +++++++++++++++++++ 2 files changed, 72 insertions(+) create mode 100755 user-client/helper/redis_op_gc/delete_task.rb create mode 100644 user-client/helper/redis_op_gc/force_delete.rb diff --git a/user-client/helper/redis_op_gc/delete_task.rb b/user-client/helper/redis_op_gc/delete_task.rb new file mode 100755 index 0000000..643d2b3 --- /dev/null +++ b/user-client/helper/redis_op_gc/delete_task.rb @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby + +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './force_delete' + +if ARGV.empty? + puts "Usage: #{__FILE__} task_id[|task_ids]" + puts ' delete special task with [task_id]' + puts ' or delete special task form a [task_ids] file' + exit +end + +task_ids = [] +ARGV.each do |id| + task_ids += IO.readlines(id) if File.exist?(id) + task_ids << [id] unless File.exist?(id) +end + +i = 0 +task_num = task_ids.size +while i < task_num + task_id = task_ids[i] + fd = ForceDelete.new(task_id) + fd.run + + i += 1 + set_progress(i, task_num) +end diff --git a/user-client/helper/redis_op_gc/force_delete.rb b/user-client/helper/redis_op_gc/force_delete.rb new file mode 100644 index 0000000..f5e52d2 --- /dev/null +++ b/user-client/helper/redis_op_gc/force_delete.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require_relative './basic_env' + +# force delete for job (with job_id) +# - taskqueue hash key: "queues/sched/*/reday?in_process?idle?uuid..." +# - taskqueue hash key: "queues/id2content" +# - scheduler hash key: "sched/id2job" +class ForceDelete + def initialize(id) + @task_id = id.chomp + end + + def run + fd4jobqueue + gc4taskqueue + gc4scheduler + end + + def fd4jobqueue + task_content = get_taskqueue_content4id(nil, @task_id) + return unless task_content + + queue_name = task_content['queue'] + cmd = "#{CMD_BASE} queues/#{queue_name} , zrem #{@task_id}" + `#{cmd}`.chomp + end + + def gc4taskqueue + cmd = "#{CMD_BASE} queues/id2content , hdel #{@task_id}" + `#{cmd}`.chomp + end + + def gc4scheduler + cmd = "#{CMD_BASE} sched/id2job , hdel #{@task_id}" + `#{cmd}`.chomp + end +end -- 2.23.0

1 0

[PATCH compass-ci 3/4] sparrow: add domain name resolution for Docker images
by Hu Xuejiao 28 Sep '20

28 Sep '20

When executing ./crystal-shards/build appear error: http://mirrors.huaweicloud.com/alpine/v3.11/main: temporary error(try aggin later) Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/3-code/dev-env | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 00d19c0..6bd7f21 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -31,3 +31,6 @@ export CCI_SRC=/c/compass-ci export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +cat >> /etc/default/docker <<EOF +DOCKER_OPTS="--dns 114.114.114.114" +EOF -- 2.23.0

2 1

[PATCH compass-ci] Change crystal-ci to compass-ci
by Zhang Yu 28 Sep '20

28 Sep '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/tutorial.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index d366580..d9f6158 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -1,4 +1,4 @@ -# Crystal-CI Introduction +# Compass-CI Introduction ## 简单说明 @@ -659,7 +659,7 @@ Compass-CI 集开发调测、测试服务、测试结果分析、辅助定位为 b) 将测试用例所需debian下的包名填入创建的文件中 echo "apache2" >> lkp-tests/distro/depends/test c) 生成lkp-aarch64.cgz - ./crystal-ci/container/lkp-initrd/run + ./compass-ci/container/lkp-initrd/run d) 提交测试用例 submit test.yaml e) 查看结果 -- 2.23.0

2 1

[PATCH compass-ci 2/4] sparrow: use useradd to add a lkp user instead of adduser
by Hu Xuejiao 28 Sep '20

28 Sep '20

Executing "adduser -u 1090 lkp" appear: New password: After usering "adduser ** --disabled-password **" appear: Full Name []: So use "adduser" to need too much user-information. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 1bb6861..05becdf 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -4,7 +4,7 @@ gem install git activesupport rest-client cucumber json faye-websocket elasticsearch -grep -q '^lkp:' /etc/passwd || adduser -u 1090 lkp +grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^team:' /etc/group || groupadd team grep -q '^committer:' /etc/group || groupadd -g 1999 committer -- 2.23.0

1 0

[PATCH v2 compass-ci] src/extract-stats prefix exception handing
by Lu Weitao 28 Sep '20

28 Sep '20

background: extract-stats process is at risk of be interrupted, due to other dependent services(task-queue, es_client) may not working properly solution: 1. use begin ... rescue to handle possible exceptions 2. skip and remove the task from task-queue if the job data does not have post-processing conditions 3. refactor for stats_worker.cr Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 60 ++++++++++++++++++------------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..edebf27 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -18,40 +18,50 @@ class StatsWorker def consume_sched_queue(queue_path : String) loop do begin - response = @tq.consume_task(queue_path) + job_id = get_job_id(queue_path).to_s + if job_id # consume the job by post-processing + job = @es.get_job_content(job_id) + result_root = job["result_root"]? + result_post_processing(job_id, result_root.to_s, queue_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + end rescue e STDERR.puts e.message - # incase of many error message when task-queue is busy + # incase of many error message when task-queue, ES does not work sleep(10) - next end + end + end - if response[0] == 200 - job_id = JSON.parse(response[1].to_json)["id"] - job = @es.get_job_content(job_id.to_s) - result_root = job["result_root"] - begin - # extract stats.json - system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" - # storage stats to job in es - store_stats_es(result_root.to_s, job_id.to_s) if result_root - # send mail to submitter for job results - system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" - rescue e - STDERR.puts e.message - next - end - - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - else - sleep(60) - end + # get job_id from task-queue + def get_job_id(queue_path : String) + response = @tq.consume_task(queue_path) + if response[0] == 200 + JSON.parse(response[1].to_json)["id"] + else + # will sleep 60s if no task in task-queue + sleep(60) + nil end end - def store_stats_es(result_root : String, job_id : String) + def result_post_processing(job_id : String, result_root : String, queue_path : String) + return nil unless result_root + + # extract stats.json + system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" + # storage stats to job in es + store_stats_es(result_root, job_id, queue_path) + # send mail to submitter for job results + system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" + end + + def store_stats_es(result_root : String, job_id : String, queue_path : String) stats_path = "#{result_root}/stats.json" - raise "#{stats_path} file not exists." unless File.exists?(stats_path) + unless File.exists?(stats_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + raise "#{stats_path} file not exists." + end stats = File.open(stats_path) do |file| JSON.parse(file) -- 2.23.0

1 0

[PATCH compass-ci] feat: scheduler: add optional fields into job
by Ren Wen 28 Sep '20

28 Sep '20

Only `multi-qemu` job needs `FLUENTD_SERVER_*`, so move them to the place where calls this function. When calls this function, an array can be passed, like: ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}",] these elements will add into job as job fields. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/jobfile_operate.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index df5af1b..0533346 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -200,14 +200,13 @@ module Jobfile::Operate return target_path end - def self.auto_submit_job(job_file, override_parameter) + def self.auto_submit_job(job_file, override_parameter, other_parameters = nil) job_fields = ["SCHED_HOST=#{ENV["SCHED_HOST"]}", "SCHED_PORT=#{ENV["SCHED_PORT"]}", - "FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", - "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}", "GIT_SERVER=#{ENV["GIT_SERVER"]}", "lab=#{ENV["lab"]}"] + job_fields.concat(other_parameters) if other_parameters cmd = "#{ENV["LKP_SRC"]}/sbin/submit " cmd += job_fields.join(" ") cmd += " -s '#{override_parameter}' #{job_file}" -- 2.23.0

1 0

[PATCH compass-ci] sparrow: container: fix git-mirror start error
by Liu Yinsi 28 Sep '20

28 Sep '20

1. git clone upstream-repo and export REPO_SRC to fix error: No such file or directory @ rb_sysopen - /c/upstream-repos 2. chown lkp:committer /srv/git to fix error: Permission denied @ dir_s_mkdir - /srv/git/b Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 1 + sparrow/3-code/dev-env | 1 + sparrow/3-code/git | 4 ++++ 3 files changed, 6 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 35456a8..f5c1646 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -6,6 +6,7 @@ chown -R lkp:lkp /srv/es chown lkp:lkp /srv/cache/netdata_cache chown lkp:lkp /srv/cache/netdata_lib chown lkp:lkp /srv/cci/serial/fluentd-pos +chown lkp:committer /srv/git chgrp team /srv/cci/serial/logs chmod 775 /srv/result chmod 775 /srv/es/logging-es diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 00d19c0..ae267bd 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -27,6 +27,7 @@ EOF cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci +export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF diff --git a/sparrow/3-code/git b/sparrow/3-code/git index b1d2c03..5588696 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -16,3 +16,7 @@ cd /c || exit [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git } + +[ -d "upstream-repos/.git" ] || { + git clone https://gitee.com/wu_fengguang/upstream-repos.git +} -- 2.23.0

1 0

[PATCH compass-ci] Makefile: add ctags
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- Makefile | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 Makefile diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..c7505e8 --- /dev/null +++ b/Makefile @@ -0,0 +1,2 @@ +ctags: + ctags -R --links=no --langmap=ruby:.rb.cr -- 2.23.0

1 1

[PATCH compass-ci] fix: system rebooting when executed 2-network/br0
by Xu Xijian 28 Sep '20

28 Sep '20

When 2-network/br0 executed, program will hang up fo a while and then go rebooting. The cause is iptables need time to take effect, echo 1 to /proc/sys/net/ipv4/ip_forward to make it taking effect immediately. Logs(openeuler 20.03): [ 161.565451] + 2-network/iptables [ 161.565453] [ 161.568088] ++ ip route get 1.2.3.4 [ 161.568090] ... [ 168.107379] LKP: rebooting Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- sparrow/2-network/iptables | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..0bf1235 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -11,6 +11,8 @@ BR0_SUBNET=172.18.0.0/16 iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE +# To make upon rules taking effect immediately +echo 1 > /proc/sys/net/ipv4/ip_forward systemctl status firewalld | grep -q "running" && { -- 2.23.0

2 1

[PATCH compass-ci] code-spec: refactor friendly principles
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index e783a0c..f805aec 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -7,6 +7,23 @@ code注意事项 - 一个函数, 不要超过10行代码 - 起准确和有意义的变量/函数名字, 名字要让人看出赋予他们的确切意义 +重构友好编码 +============ + +基本原则: +- 预防自己的代码造成他人的几大困难 + - 难以理解不敢下手 + - 不知道怎么找到*所有*关联逻辑 + - 不知道自己不知道 (unknown unknown) +- 设想将来做重构, 或者需要理解任何一个高层次的功能/流程, + 一个新人应当可以通过 git grep 一两个关键字或者pattern, + 就能找到所有相关底层代码与文档, 而不必担心有所遗漏. + +基本方法: +- 隐式场景显式化: 加注释明确代码的各类隐含目的, 意义, 场景, 数据示例等等 +- 隐式依赖显式化: 定义相同的ID, 把所有的关联文档与各处代码串起来 +- 分散依赖集中化: 定义类/函数/assert, 封装/强制各类设计中的假设, 约定与规则 + PATCH注意事项 ============= -- 2.23.0

1 0

[PATCH lkp-tests] Makefile: support ctags
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- Makefile | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Makefile b/Makefile index c40d914b..37387809 100644 --- a/Makefile +++ b/Makefile @@ -23,3 +23,6 @@ doc: tests/%.md: tests/%.yaml lkp gen-doc $< + +ctags: + ctags -R --links=no -- 2.23.0

1 0

[PATCH compass-ci] doc: add code-spec.md
by Wu Fengguang 27 Sep '20

27 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 42 ++++++++++++++++++++++++++++++++++++++ doc/develop-environment.md | 1 + 2 files changed, 43 insertions(+) create mode 100644 doc/code-spec.md diff --git a/doc/code-spec.md b/doc/code-spec.md new file mode 100644 index 0000000..e783a0c --- /dev/null +++ b/doc/code-spec.md @@ -0,0 +1,42 @@ +code注意事项 +============ + +- 代码应当放进函数里不要在脚本顶层直接写逻辑 +- 代码应当放进lib/里脚本本身最小化, 仅解析命令行参数, 调用lib/入口函数 +- 如果出现了全局变量一般意味着需要写一个 class/module 来放这个状态量了 +- 一个函数, 不要超过10行代码 +- 起准确和有意义的变量/函数名字, 名字要让人看出赋予他们的确切意义 + +PATCH注意事项 +============= + +- PATCH title 应当简要说明做了什么 (what) +- PATCH changelog / code comment 应当解释读者无法从代码简单推断的 + - 背景 + - 目的 + - 效果 + - 设计选项和权衡 + - 实际例子 + - 改进数字 + - bug场景, 出错消息原文 +- PATCH 应当加上作者签名 (Signed-off-by: Name <email>) +- PATCH 最好不超过100行, 一个PATCH只做一件事, 避免混杂多个逻辑变更 + +PATCH参考材料 +============= + +The patch subject and changelog should be well written according to suggestions +here: + + https://github.com/thoughtbot/dotfiles/blob/master/gitmessage + https://www.cnblogs.com/cpselvis/p/6423874.html + + http://www.ozlabs.org/~akpm/stuff/tpp.txt + 2: Subject: + 3: Attribution + 4: Changelog + + https://www.kernel.org/doc/html/latest/translations/zh_CN/process/submittin… + 2) 描述你的改动 + 3) 拆分你的改动 + 8）回复评审意见 diff --git a/doc/develop-environment.md b/doc/develop-environment.md index 3af44ac..8760878 100644 --- a/doc/develop-environment.md +++ b/doc/develop-environment.md @@ -50,6 +50,7 @@ Then read through documents lkp-tests/doc/INSTALL.md compass-ci/doc/INSTALL.md + compass-ci/doc/code-spec.md compass-ci/doc/learning-resources.md # crystal compiler -- 2.23.0

1 0

[PATCH todo] credit: fix typos
by Wu Fengguang 26 Sep '20

26 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- people/LiangTao/credit/2020-09-11.md | 2 +- people/LiuShaofei/credit/2020-08-14.md | 2 +- people/LiuYinsi/credit/2020-09-25.md | 1 - people/LuWeitao/credit/2020-09-18.md | 2 +- people/LuWeitao/credit/2020-09-25.md | 2 +- people/LuanShengde/credit/2020-09-18.md | 1 - people/WangChenglong/credit/2020-09-25.md | 1 - people/XuXijian/credit/2020-07-31.md | 2 +- people/XuXijian/credit/2020-08-07.md | 2 +- people/XuXijian/credit/2020-08-14.md | 2 +- 10 files changed, 7 insertions(+), 10 deletions(-) diff --git a/people/LiangTao/credit/2020-09-11.md b/people/LiangTao/credit/2020-09-11.md index 87afa33..464173b 100644 --- a/people/LiangTao/credit/2020-09-11.md +++ b/people/LiangTao/credit/2020-09-11.md @@ -1,4 +1,4 @@ Shi Zhichao Xiao Shenwei Wang Chenglong -Wei Jihiu +Wei Jihui diff --git a/people/LiuShaofei/credit/2020-08-14.md b/people/LiuShaofei/credit/2020-08-14.md index 66d1d24..a5a3f35 100644 --- a/people/LiuShaofei/credit/2020-08-14.md +++ b/people/LiuShaofei/credit/2020-08-14.md @@ -4,4 +4,4 @@ Wang Yong Bai Jing Yu Chuan Lin Jiaxin -Sun yukui +Sun Yukui diff --git a/people/LiuYinsi/credit/2020-09-25.md b/people/LiuYinsi/credit/2020-09-25.md index c7fb87e..f7652f6 100644 --- a/people/LiuYinsi/credit/2020-09-25.md +++ b/people/LiuYinsi/credit/2020-09-25.md @@ -3,4 +3,3 @@ Yu Chuan Luan Shengde Zhang Dewan Shi Zhichao - diff --git a/people/LuWeitao/credit/2020-09-18.md b/people/LuWeitao/credit/2020-09-18.md index 5f9f6ef..5678b6d 100644 --- a/people/LuWeitao/credit/2020-09-18.md +++ b/people/LuWeitao/credit/2020-09-18.md @@ -1,4 +1,4 @@ -Luan Shengden +Luan Shengde Wu Fengguang Zhang Yuhang Cao Xueliang diff --git a/people/LuWeitao/credit/2020-09-25.md b/people/LuWeitao/credit/2020-09-25.md index decb297..1f2515d 100644 --- a/people/LuWeitao/credit/2020-09-25.md +++ b/people/LuWeitao/credit/2020-09-25.md @@ -2,7 +2,7 @@ Wu Fengguang Du Kaitian Cao Xueliang Zhang Yuhang -Luan Shengden +Luan Shengde Sun Yukui Liu Shaofei Li Yuanchao diff --git a/people/LuanShengde/credit/2020-09-18.md b/people/LuanShengde/credit/2020-09-18.md index 574c682..4a81084 100644 --- a/people/LuanShengde/credit/2020-09-18.md +++ b/people/LuanShengde/credit/2020-09-18.md @@ -3,4 +3,3 @@ Lu Weitao Shi Zhichao Tong Qunfeng Zhang Dewan - diff --git a/people/WangChenglong/credit/2020-09-25.md b/people/WangChenglong/credit/2020-09-25.md index a0e4736..5239ca1 100644 --- a/people/WangChenglong/credit/2020-09-25.md +++ b/people/WangChenglong/credit/2020-09-25.md @@ -3,4 +3,3 @@ Shi Zhichao Wu Zhende Wang Yong Wei Jihui - diff --git a/people/XuXijian/credit/2020-07-31.md b/people/XuXijian/credit/2020-07-31.md index 5cf1aa4..36fea5e 100644 --- a/people/XuXijian/credit/2020-07-31.md +++ b/people/XuXijian/credit/2020-07-31.md @@ -1,4 +1,4 @@ -Zhang Dewang +Zhang Dewan Zhang Yale Yu Chuan Wang Yong diff --git a/people/XuXijian/credit/2020-08-07.md b/people/XuXijian/credit/2020-08-07.md index 056782a..cca2b2f 100644 --- a/people/XuXijian/credit/2020-08-07.md +++ b/people/XuXijian/credit/2020-08-07.md @@ -4,4 +4,4 @@ Li Yuanchao Xiao Shenwei Wei Jihui Yu Chuan -Zhang Dewang +Zhang Dewan diff --git a/people/XuXijian/credit/2020-08-14.md b/people/XuXijian/credit/2020-08-14.md index 7c60171..e9cfd69 100644 --- a/people/XuXijian/credit/2020-08-14.md +++ b/people/XuXijian/credit/2020-08-14.md @@ -1,4 +1,4 @@ -Shi zhichao +Shi Zhichao Luan Shengde Wang Yong Wu Fengguang -- 2.23.0

1 0

答复: Welcome to the "Compass-ci" mailing list
by Dukaitian (Dukaitian, Intelligent Computing R&D) 23 Sep '20

23 Sep '20

各位老师、开发者好：由于crystal 被其它公司注册了，及后缀CI的通用性，影响到crystal-ci 的名字，因此需要将项目名称 crystal-ci 修改为 compass-ci 后续我们需要通过compass-ci(a)openeuler.org 邮件列表来继续交流，crystal-ci(a)openEuler.org 预计在一个月后进行停用。由于变更给大家带来的麻烦深表歉意，敬请谅解，感谢大家一如既往的关注及支持。祝好 Jimmy_hero -----邮件原件----- 发件人: compass-ci-request(a)openeuler.org [mailto:compass-ci-request@openeuler.org] 发送时间: 2020年9月23日 17:25 收件人: Dukaitian (Dukaitian, Intelligent Computing R&D) <dukaitian(a)huawei.com> 主题: Welcome to the "Compass-ci" mailing list Welcome to the "Compass-ci" mailing list! To post to this list, send your email to: compass-ci(a)openeuler.org You can unsubscribe or make adjustments to your options via email by sending a message to: compass-ci-request(a)openeuler.org with the word 'help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You will need your password to change your options, but for security purposes, this password is not included here. If you have forgotten your password you will need to reset it via the web UI.

1 0