- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests 1/2] hosts: add general vm-*
by Wu Fengguang 09 Oct '20

09 Oct '20

These are VM boxes that don't include HW model or arch in file name or content. Which should be more flexible for matrix combination in general. This means - on job submit, users should provide 'arch' field - multi-qemu should define 'arch' field by default based on host arch - the scheduler should manage queues matching VM/docker testbox's arch Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- hosts/vm-1p1g | 5 +++++ hosts/vm-1p1g-pxe | 5 +++++ hosts/vm-2p1g | 5 +++++ hosts/vm-2p1g-pxe | 5 +++++ hosts/vm-2p4g | 5 +++++ hosts/vm-2p4g-pxe | 5 +++++ hosts/vm-2p8g | 5 +++++ hosts/vm-2p8g-pxe | 5 +++++ 8 files changed, 40 insertions(+) create mode 100644 hosts/vm-1p1g create mode 100644 hosts/vm-1p1g-pxe create mode 100644 hosts/vm-2p1g create mode 100644 hosts/vm-2p1g-pxe create mode 100644 hosts/vm-2p4g create mode 100644 hosts/vm-2p4g-pxe create mode 100644 hosts/vm-2p8g create mode 100644 hosts/vm-2p8g-pxe diff --git a/hosts/vm-1p1g b/hosts/vm-1p1g new file mode 100644 index 00000000..c64cc360 --- /dev/null +++ b/hosts/vm-1p1g @@ -0,0 +1,5 @@ +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 1 +memory: 1G diff --git a/hosts/vm-1p1g-pxe b/hosts/vm-1p1g-pxe new file mode 100644 index 00000000..233f6393 --- /dev/null +++ b/hosts/vm-1p1g-pxe @@ -0,0 +1,5 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 1 +memory: 1G diff --git a/hosts/vm-2p1g b/hosts/vm-2p1g new file mode 100644 index 00000000..c7810336 --- /dev/null +++ b/hosts/vm-2p1g @@ -0,0 +1,5 @@ +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 1G diff --git a/hosts/vm-2p1g-pxe b/hosts/vm-2p1g-pxe new file mode 100644 index 00000000..6129d20e --- /dev/null +++ b/hosts/vm-2p1g-pxe @@ -0,0 +1,5 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 1G diff --git a/hosts/vm-2p4g b/hosts/vm-2p4g new file mode 100644 index 00000000..7e9570da --- /dev/null +++ b/hosts/vm-2p4g @@ -0,0 +1,5 @@ +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 4G diff --git a/hosts/vm-2p4g-pxe b/hosts/vm-2p4g-pxe new file mode 100644 index 00000000..8ba4e58a --- /dev/null +++ b/hosts/vm-2p4g-pxe @@ -0,0 +1,5 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 4G diff --git a/hosts/vm-2p8g b/hosts/vm-2p8g new file mode 100644 index 00000000..e5d5f846 --- /dev/null +++ b/hosts/vm-2p8g @@ -0,0 +1,5 @@ +provider: qemu +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 8G diff --git a/hosts/vm-2p8g-pxe b/hosts/vm-2p8g-pxe new file mode 100644 index 00000000..ae83d6f0 --- /dev/null +++ b/hosts/vm-2p8g-pxe @@ -0,0 +1,5 @@ +provider: qemu +template: pxe +nr_node: 1 +nr_cpu: 2 +memory: 8G -- 2.23.0

1 0

[PATCH v4 compass-ci 7/7] container/mail-robot: mail-robot.md
by Luan Shengde 09 Oct '20

09 Oct '20

guidance file for mail-robot Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..5c42866 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,38 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +monitor received mails + check if the mail is for applying uuid/account + +## steps overview + +1. monitor mailbox dir + - check for new email file + +2. check email + - check email subject + - return if unmatched subject + - call answerback-email + - check email available + - check base_url for the oss_url in upstream-repos list + if not, it will raise "url not in upstream-repos list" error + - check if the email has commit for the url + if not, it will raise "no commit found" error + - check pub_key + - check if there is a pub_key in the email + if not, it will raise 'no pub_key' error + - apply ssh account + - apply account uuid + - write account info to es + - send uuid to user + - mv mail file to drafts + +3. continue the monitor + - after 1 and 2, continue to monitor the mailbox + +4. see the log + - use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

2 1

[PATCH v4 compass-ci 4/7] container/mail-robot: Dockerfile
by Luan Shengde 09 Oct '20

09 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..87c04e7 --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +FROM debian +MAINTAINER luanshd0525(a)163.com +ENV DEBIAN_FRONTEND noninteractive + +COPY conf/sources.list* /etc/apt/ + +RUN apt-get update && \ + apt-get install -y uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport && \ + ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ + echo 'Asia/Shanghai' > /etc/timezone +COPY mail-robot.rb /usr/local/bin/ +COPY answerback-email.rb /usr/local/bin/ +WORKDIR /usr/local/bin/ -- 2.23.0

2 1

[PATCH v4 compass-ci 3/7] container/mail-robot: build file
by Luan Shengde 09 Oct '20

09 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..79af1de --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ + +docker build -t mail-robot:latest . + -- 2.23.0

2 1

[PATCH v4 compass-ci 2/7] container/mail-robot: answerback-email.rb
by Luan Shengde 09 Oct '20

09 Oct '20

get email content for mail-robot.rb check email available check base url in upstream-repos check commit url valid check pub_key return if no pub_key assign account generate uuid write account/uuid to es send uuid to user Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/answerback-email.rb | 201 +++++++++++++++++++++++ 1 file changed, 201 insertions(+) create mode 100755 container/mail-robot/answerback-email.rb diff --git a/container/mail-robot/answerback-email.rb b/container/mail-robot/answerback-email.rb new file mode 100755 index 0000000..6f21358 --- /dev/null +++ b/container/mail-robot/answerback-email.rb @@ -0,0 +1,201 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +=begin + +repo_list: + all repos url list from upstream-repos.git + +API: +call graph: +read_mail_content +send_account_request + email_addr + get email address + email_message_id + get email message_id + check_email_available + check email available + pub_key_value + get pub key + uuid + get jumper server account + generate uuid + mail_message + uuid + submit steps + store account_info + call es_client + send_mail + send mail with mail_message + +the returned data for apply ssh account: +{ + "account" => "guest", + "passwd" => "Use pub_key to login", + "jumper_ip" => "10.10.10.10", + "jumper_port" => "10000" +} + +=end + +require 'json' +require 'mail' +require_relative '/compass-ci/lib/es_client.rb' + +def build_message(to, message_id, account_uuid) + message = <<~EMAIL_MESSAGE + To: #{to} + Message-ID: #{message_id} + Subject: Account Ready + + Dear #{to.split('@')[0]} + + Thank you for joining us. + You can use the following info to submit jobs: + + First add the info to: ~/.config/compass-ci/defaults/{USER}.yaml + + uuid: #{account_uuid} + lab: z9 + + Second: download lkp-tests and dependencies + git clone https://gitee.com_/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source {HOME}/.bashrc && source {HOME}/.bash_profile + + Third: submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + how to write job yaml + submit jobs/netperf.yaml + + Repalce the fields {...} with your local environment variable. + + regards + compass-ci + EMAIL_MESSAGE + + return message +end + +def email_addr(mail_content) + email = mail_content.from[0] + + return email +end + +def url_addr(mail_content) + url_line = mail_content.part[0].body.decoded.split(/\n/).find { |line| line =~ %r{https?://} } + message = 'No commit url found' + + raise message unless url_line + + url = url_line.split[-1] + + return url +end + +def base_url_addr(url) + url_a = url.split('/') + url_a.pop 2 + base_url = url_a.join('/') + + return base_url +end + +def check_email_available(mail_content, email) + url = url_addr(mail_content) + + base_url = base_url_addr(url) + + upstream_dir = '/upstream-repos' + check_url_in_upstream_repos(upstream_dir, base_url) + check_url_commit(url, email) + + return url +end + +def check_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + message = 'The url is not in upstream-repo list' + + raise StandardError, message, base_url if match_out.empty? +end + +def check_url_commit(url, email) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index email + + message = 'No email info found from the url feedback.' + raise StandardError, message, url if email_index.nil? +end + +def email_message_id(mail_content) + message_id = mail_content.message_id + return message_id +end + +def pub_key_value(mail_content) + pub_key = mail_content.part[1].body.decoded.split(/\r|\n/).join if mail_content.part[1].filename == 'id_rsa.pub' + + message = 'No pub_key fount' + + raise message unless pub_key + + return pub_key +end + +def account_info(pub_key) + account_info_str = %x(curl -XGET "#{ENV['JUMPER_IP']}:#{ENV['JUMPER_PORT']}/assign_account" -d "pub_key: #{pub_key}") + account_info = JSON.parse account_info_str + + message = 'No available jumper account returned.' + raise StandardError, message, account_info if account_info['account'].empty? + + return account_info +end + +def setup_account_uuid + account_uuid = %x(uuidgen).chomp + + return account_uuid +end + +def send_account(mail_content) + email = email_addr(mail_content) + message_id = email_message_id(mail_content) + url = check_email_available(mail_content, email) + pub_key = pub_key_value(mail_content) + acct_infos = account_info(pub_key) + account_uuid = setup_account_uuid + + store_account_info(email, account_uuid, acct_infos, url) + + message = build_message(email, message_id, account_uuid) + send_mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp + + %x(curl -XPOST "#{send_mail_server}:#{ENV['SEND_MAIL_PORT']}/send_mail_text" -d "#{message}") +end + +def store_account_info(to, account_uuid, acct_infos, url) + account_uuid_info = { + 'email' => to, + 'name' => to.split('@')[0], + 'uuid' => account_uuid, + 'user' => acct_infos['account'], + 'my_commit_url' => url + } + + es = ESClient.new(index: 'accounts') + es.put_source_by_id(to, account_uuid_info) +end + +def read_mail_content(mail_file) + mail_content = Mail.read(mail_file) + + return mail_content +end -- 2.23.0

2 1

[PATCH compass-ci] sched: refactor get_boot_ipxe to job class
by Cao Xueliang 09 Oct '20

09 Oct '20

According os_mount to add initrds full path and add initrds file name in root append. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 89 +++++++++++++++++++++++++++++++++----- src/lib/sched.cr | 54 ++++------------------- src/scheduler/constants.cr | 4 ++ 3 files changed, 89 insertions(+), 58 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 53a560c..bedfec0 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -72,10 +72,12 @@ class Job lab initrd_pkg initrd_deps + initrds result_root access_key access_key_file lkp_initrd_user + kernel kernel_append_root docker_image ) @@ -123,8 +125,10 @@ class Job set_result_root() set_result_service() set_os_mount() - set_kernel_append_root() set_pp_initrd() + set_initrds() + set_kernel() + set_kernel_append_root() set_lkp_server() end @@ -228,23 +232,83 @@ class Job end end - private def set_kernel_append_root - os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") - lkp_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}/initrd.lkp") - lkp_basename = File.basename(lkp_real_path) + private def vmlinuz + if os_mount == "initramfs" + return "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/vmlinuz")}" + else + return "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/vmlinuz")}" + end + end + + private def set_kernel + self["kernel"] = "kernel #{vmlinuz()}" + end + + private def common_initrds + temp_initrds = [] of String + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz")}" + temp_initrds << "#{SCHED_HTTP_PREFIX}/job_initrd_tmpfs/#{id}/job.cgz" + + return temp_initrds + end + + private def initramfs_initrds + temp_initrds = [] of String + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + temp_initrds << "#{INITRD_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + + temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? + temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? + + return temp_initrds + end + + private def nfs_cifs_initrds + temp_initrds = [] of String + temp_initrds << "#{OS_HTTP_PREFIX}" + + "#{JobHelper.service_path("#{SRV_OS}/#{os_dir}/initrd.lkp")}" + + return temp_initrds + end + + private def set_initrds + temp_initrds = [] of String - current_basename = "" if "#{os_mount}" == "initramfs" - current_real_path = JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current") - current_basename = File.basename(current_real_path) + temp_initrds.concat(initramfs_initrds()) + else + temp_initrds.concat(nfs_cifs_initrds()) + end + + temp_initrds.concat(common_initrds()) + + self["initrds"] = temp_initrds.join(" ") + end + + private def initrd_basename + basenames = "" + + initrds.split(/ /).each do |initrd| + basenames += "initrd=#{File.basename(initrd)} " end + return basenames + end + + private def set_kernel_append_root + os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") + fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} initrd=#{lkp_basename}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrd_basename()}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr initrd=#{lkp_basename}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 initrd=#{current_basename}", - "container" => "" + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrd_basename()}", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrd_basename()}", + "container" => "", } self["kernel_append_root"] = fs2root[os_mount] end @@ -270,6 +334,7 @@ class Job end end end + self["initrd_deps"] = initrd_deps_arr.join(" ") self["initrd_pkg"] = initrd_pkg_arr.join(" ") end diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 309df04..990641c 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -488,61 +488,23 @@ class Sched return returned end - private def get_pp_initrd(job : Job) - initrd_deps = "" - initrd_pkg = "" - if job.os_mount == "initramfs" - initrd_deps += job.initrd_deps.split.join() { |item| "initrd #{item}\n" } - initrd_pkg += job.initrd_pkg.split.join() { |item| "initrd #{item}\n" } + private def get_initrds(job_initrds) + initrds = "" + job_initrds.each do |initrd| + initrds += "initrd #{initrd}\n" end - return initrd_deps, initrd_pkg + return initrds end private def get_boot_ipxe(job : Job) - initrd_lkp_cgz = "lkp-#{job.os_arch}.cgz" - - initrd_deps, initrd_pkg = get_pp_initrd(job) - - initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" - sched_http_prefix = "http://#{SCHED_HOST}:#{SCHED_PORT}" - os_http_prefix = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" - response = "#!ipxe\n\n" - if job.os_mount == "initramfs" - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/current")}\n" - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/run-ipconfig.cgz")}\n" - else - response += "initrd #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/initrd.lkp")}\n" - end - response += "initrd #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/#{initrd_lkp_cgz}")}\n" - response += "initrd #{sched_http_prefix}/job_initrd_tmpfs/#{job.id}/job.cgz\n" - response += initrd_deps - response += initrd_pkg - if job.os_mount == "initramfs" - response += "kernel #{initrd_http_prefix}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" - else - response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" - end + response += get_initrds(job.initrds.split(/ /)) + response += job.kernel response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" response += " #{job.kernel_append_root}" response += add_kernel_console_param(job.os_arch) - if job.os_mount == "initramfs" - response += " initrd=#{initrd_lkp_cgz} initrd=job.cgz" - job.initrd_deps.split.each do |initrd_dep| - response += " initrd=#{File.basename(initrd_dep)}" - end - response += " initrd=#{File.basename(JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/run-ipconfig.cgz"))}\n" - else - response += " initrd=#{initrd_lkp_cgz} initrd=job.cgz\n" - end - response += "boot\n" + response += "\nboot\n" puts %({"job_id": "#{job.id}", "job_state": "boot"}) return response diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 7ce49a1..839c241 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -24,4 +24,8 @@ OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i3 SRV_OS = "/srv/os" SRV_INITRD = "/srv/initrd" +INITRD_HTTP_PREFIX = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" +OS_HTTP_PREFIX = "http://#{OS_HTTP_HOST}:#{OS_HTTP_PORT}" +SCHED_HTTP_PREFIX = "http://#{SCHED_HOST}:#{SCHED_PORT}" + DEMO_JOB = %({"suite":"pixz","testcase":"pixz","category":"benchmark","nr_threads":1,"pixz":null,"job_origin":"jobs/pixz.yaml","testbox":"wfg-e595","arch":"x86_64","tbox_group":"wfg-e595","id":"100","kmsg":null,"boot-time":null,"uptime":null,"iostat":null,"heartbeat":null,"vmstat":null,"numa-numastat":null,"numa-vmstat":null,"numa-meminfo":null,"proc-vmstat":null,"proc-stat":null,"meminfo":null,"slabinfo":null,"interrupts":null,"kconfig":"x86_64-rhel-7.6","compiler":"gcc-7"}) -- 2.23.0

3 6

[PATCH v2 lkp-tests] lib: add transfer_port.sh
by Zhang Yale 09 Oct '20

09 Oct '20

Provide a login port for user, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- lib/transfer_port.sh | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 lib/transfer_port.sh diff --git a/lib/transfer_port.sh b/lib/transfer_port.sh new file mode 100755 index 00000000..61fb67b4 --- /dev/null +++ b/lib/transfer_port.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# uuid + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 + +get_port() +{ + [ -n "$uuid" ] || return + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%10+11333)) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] || continue + echo "$port" + break + done +} + -- 2.23.0

4 8

[PATCH v2 compass-ci] container: add a container auto-submit
by Li Yuanchao 09 Oct '20

09 Oct '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/auto-submit/Dockerfile | 16 +++++++++++++++ container/auto-submit/build | 6 ++++++ container/auto-submit/start | 30 +++++++++++++++++++++++++++++ container/auto-submit/start-depends | 1 + 4 files changed, 53 insertions(+) create mode 100644 container/auto-submit/Dockerfile create mode 100755 container/auto-submit/build create mode 100755 container/auto-submit/start create mode 100755 container/auto-submit/start-depends diff --git a/container/auto-submit/Dockerfile b/container/auto-submit/Dockerfile new file mode 100644 index 0000000..5b44551 --- /dev/null +++ b/container/auto-submit/Dockerfile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk upgrade && \ + apk add --no-cache git && \ + apk add ruby-dev make gcc g++ + +RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc + +RUN umask 002 && \ + gem install bunny json activesupport git rest-client diff --git a/container/auto-submit/build b/container/auto-submit/build new file mode 100755 index 0000000..96383e3 --- /dev/null +++ b/container/auto-submit/build @@ -0,0 +1,6 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t alpine:auto-submit . + diff --git a/container/auto-submit/start b/container/auto-submit/start new file mode 100755 index 0000000..4632ab0 --- /dev/null +++ b/container/auto-submit/start @@ -0,0 +1,30 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +DOCKER_CCI = '/c/compass-ci' +DOCKER_LKP = '/c/lkp-tests' +docker_rm "auto-submit" + +cmd = %W[ + docker run + --restart=always + --name auto-submit + -u 1090:1090 + -d + -e CCI_SRC=#{DOCKER_CCI} + -e LKP_SRC=#{DOCKER_LKP} + -v #{ENV['CCI_SRC']}:#{DOCKER_CCI} + -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} + -v /etc/localtime:/etc/localtime:ro + -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro + -v /srv/git:/srv/git + -w /c/compass-ci/sbin + alpine:auto-submit +] + +cmd += ['sh', '-c', 'umask 002 && ./auto_submit'] +system(*cmd) diff --git a/container/auto-submit/start-depends b/container/auto-submit/start-depends new file mode 100755 index 0000000..5232abf --- /dev/null +++ b/container/auto-submit/start-depends @@ -0,0 +1 @@ +rabbitmq -- 2.23.0

1 0

[PATCH compass-ci] README.md: improve format for README.md
by Lu Kaiyi 09 Oct '20

09 Oct '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- README.md | 66 +++++++++++++++++++++++++++---------------------------- 1 file changed, 33 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index cd02803..644e9ba 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ ### 概念 -Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github,Gitee,Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过Compass-CI，社区开发者可以将开源软件快速引入openEuler社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 +Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上游开源软件（来自 Github, Gitee, Gitlab 等托管平台）的测试服务、登录服务、故障辅助定界服务和基于历史数据的分析服务。通过 Compass-CI，社区开发者可以将开源软件快速引入 openEuler 社区，并补充更多的测试用例，共同构建一个开放、完整的开源软件生态系统。 ### 功能描述 @@ -31,11 +31,11 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 ![Compass-CI架构](doc/pictures/compass-ci-architecture.png) - **接入层** - 成为衔接Compass-CI 服务于开发者的纽带，同时提供内部设备管理的平台，提供web门户查阅测试结果、项目清单、ssh环境登录 + 成为衔接 Compass-CI 服务于开发者的纽带，同时提供内部设备管理的平台，提供 web 门户查阅测试结果、项目清单、ssh 环境登录。 - **服务层** - 提供测试服务、环境登录、结果分析、辅助定界、用户项目注册能力，为本项目的主要功能 + 提供测试服务、环境登录、结果分析、辅助定界、用户项目注册能力，为本项目的主要功能。 - **支撑层** @@ -43,7 +43,7 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 - **数据层** - 为项目提供数据处理服务，包括测试结果格式化及归档 + 为项目提供数据处理服务，包括测试结果格式化及归档。 - **资源层** @@ -54,7 +54,7 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 - **聚合开发者测试用例** - 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 + 开发者向代码托管平台提交代码、测试用例、测试工具时，Compass-CI 自动获取提交的代码开展构建测试，同时获取开发者编写到开源软件包的测试用例进行自动化测试，并反馈测试结果。 - **登录环境随时调测** @@ -66,16 +66,16 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 - **辅助定界** - 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题 Commit 信息。 + 测试过程中，发现有 Bug 时，自动触发 Regression 机制，找出首次引入问题的 Commit 信息。 -## 使用Compass-CI -> 您只需要先注册自己的仓库，当您的仓库有commit提交时，构建测试会自动执行，并且可在我们的网站中查看结果。 +## 使用 Compass-CI +> 您只需要先注册自己的仓库，当您的仓库有 commit 提交时，构建测试会自动执行，并且可在我们的网站中查看结果。 - 注册自己的仓库 - 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库[upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 + 如果您想在 `git push` 的时候, 自动触发测试, 那么需要把您的公开 git url 添加到如下仓库 [upstream-repos](https://gitee.com/wu_fengguang/upstream-repos)。 ```bash git clone https://gitee.com/wu_fengguang/upstream-repos.git less upstream-repos/README.md @@ -83,7 +83,7 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 - `git push` - 更新仓库，自动触发测试 + 更新仓库，自动触发测试。 - 在网页中搜索并查看结果 @@ -93,62 +93,62 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 ## 高级功能 -### 添加OS支持 +### 添加 OS 支持 -制作一个initramfs启动的cgz镜像，当系统内核启动时，直接从打包的cgz镜像中导出rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的I/O性能。 +制作一个 initramfs 启动的 cgz 镜像，当系统内核启动时，直接从打包的 cgz 镜像中导出 rootfs，在内存中展开文件系统，利用磁盘的高速缓存机制让系统直接在内存中读写文件，以提升系统的 I/O 性能。 #### 操作步骤 - 1. 获取对应os版本的rootfs（以openEuler为例） - - 通过docker获取rootfs + 1. 获取对应 os 版本的 rootfs（以 openEuler 为例） + - 通过 docker 获取 rootfs - 1) 下载openEuler官方提供的docker镜像压缩包 + 1) 下载 openEuler 官方提供的 docker 镜像压缩包 ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/docker_img/aarch64/openEuler… ``` - 2) 加载docker镜像 + 2) 加载 docker 镜像 ```bash docker load -i openEuler-docker.aarch64 ``` - 3) 启动openEuler容器 + 3) 启动 openEuler 容器 ```bash docker run -id openeuler-20.03-lts ``` - 4) 拷贝docker的rootfs + 4) 拷贝 docker 的 rootfs ```bash docker cp -a docker run -d openeuler-20.03-lts:/ openEuler-rootfs ``` - - 通过qemu.img(qcow2格式)获取rootfs (以openEuler为例) + - 通过 qemu.img(qcow2格式)获取 rootfs (以 openEuler 为例) - 1) 下载openEuler官方网站提供的qcow2格式镜像 + 1) 下载 openEuler 官方网站提供的 qcow2 格式镜像 ```bash wget https://repo.openeuler.org/openEuler-20.03-LTS/virtual_machine_img/aarch64/… ``` - 2) 使用{compass-ci}/container/qcow2rootfs 制作rootfs + 2) 使用 {compass-ci}/container/qcow2rootfs 制作 rootfs ```bash cd {compass-ci}/container/qcow2rootfs ./run openEuler-20.03-LTS.aarch64.qcow2.xz /tmp/openEuler-rootfs ``` - 2. 定制rootfs - 1. 使用chroot命令切换到rootfs中(此步骤需要root权限) + 2. 定制 rootfs + 1. 使用 chroot 命令切换到 rootfs 中(此步骤需要 root 权限) ```bash chroot openEuler-rootfs ``` 2. 根据个人需要安装并配置服务 - a. 修改root密码 - b. 配置ssh服务 + a. 修改 root 密码 + b. 配置 ssh 服务 c. 检查系统时间 - d. 如果使用docker制作osimage还需要以下操作： + d. 如果使用 docker 制作 osimage 还需要以下操作： > 1. 安装所需版本内核 - > 2. 从centos官方网站下载内核rpm包 - > 3. 使用yum进行安装 - > 4. 删除docker环境变量文件 + > 2. 从 centos 官方网站下载内核 rpm 包 + > 3. 使用 yum 进行安装 + > 4. 删除 docker 环境变量文件 - 3. 退出rootfs，并打包 + 3. 退出 rootfs，并打包 ```bash cd $rootfs find . | coip -o -Hnewc |gzip -9 > $os_name.cgz @@ -167,16 +167,16 @@ Compass-CI 是一个可持续集成的软件平台。为开发者提供针对上 - 解决方法 1）启动内存不足，增加内存可解决。 - 2）内核文件权限不足，给予644权限。 + 2）内核文件权限不足，给予 644 权限。 2. 系统运行缓慢 - 问题现象 - 打包镜像体积过大，会消耗很大内存 + 打包镜像体积过大，会消耗很大内存。 - 解决方法 - 建议用户根据具体需要对rootfs进行裁剪。 + 建议用户根据具体需要对 rootfs 进行裁剪。 ## 呼吁合作 -- 2.23.0

1 0

[PATCH compass-ci] container: add a container auto-submit
by Li Yuanchao 09 Oct '20

09 Oct '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/auto-submit/Dockerfile | 15 +++++++++++++++ container/auto-submit/build | 6 ++++++ container/auto-submit/start | 30 +++++++++++++++++++++++++++++ container/auto-submit/start-depends | 1 + 4 files changed, 52 insertions(+) create mode 100644 container/auto-submit/Dockerfile create mode 100755 container/auto-submit/build create mode 100755 container/auto-submit/start create mode 100755 container/auto-submit/start-depends diff --git a/container/auto-submit/Dockerfile b/container/auto-submit/Dockerfile new file mode 100644 index 0000000..16b3593 --- /dev/null +++ b/container/auto-submit/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + apk upgrade && \ + apk add --no-cache git && \ + apk add ruby-dev make gcc g++ + +RUN umask 002 && \ + gem install bunny json activesupport git rest-client diff --git a/container/auto-submit/build b/container/auto-submit/build new file mode 100755 index 0000000..3ae6b78 --- /dev/null +++ b/container/auto-submit/build @@ -0,0 +1,6 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t alpine311:auto-submit . + diff --git a/container/auto-submit/start b/container/auto-submit/start new file mode 100755 index 0000000..d0a0978 --- /dev/null +++ b/container/auto-submit/start @@ -0,0 +1,30 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +DOCKER_CCI = '/c/compass-ci' +DOCKER_LKP = '/c/lkp-tests' +docker_rm "auto-submit" + +cmd = %W[ + docker run + --restart=always + --name auto-submit + -u 1090:1090 + -d + -e CCI_SRC=#{DOCKER_CCI} + -e LKP_SRC=#{DOCKER_LKP} + -v #{ENV['CCI_SRC']}:#{DOCKER_CCI} + -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} + -v /etc/localtime:/etc/localtime:ro + -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro + -v /srv/git:/srv/git + -w /c/compass-ci/sbin + alpine311:auto-submit +] + +cmd += ['sh', '-c', 'umask 002 && ./auto_submit'] +system(*cmd) diff --git a/container/auto-submit/start-depends b/container/auto-submit/start-depends new file mode 100755 index 0000000..5232abf --- /dev/null +++ b/container/auto-submit/start-depends @@ -0,0 +1 @@ +rabbitmq -- 2.23.0

3 5