October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v6 compass-ci 6/6] container/mail-robot: mail-robot.md
by Luan Shengde 13 Oct '20

13 Oct '20

guidance file for mail-robot Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- doc/mail-robot.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..e248b41 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,40 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +monitor received mails + check if the mail is for applying uuid/account + +## steps overview + +1. monitor mailbox dir + - check for new email file + +2. check email + - check email subject + - return if unmatched subject + - call answerback-email + - check email available + - check base_url for the oss_url in upstream-repos list + if not, it will raise "url not in upstream-repos list" error + - check if the email has commit for the url + - if gitee, git clone the repo and check + if non-gitee, curl url to check + - if not, raise "no commit found" error + - check pub_key + - check if there is a pub_key in the email + if not, it will raise 'no pub_key' error + - apply ssh account + - apply account uuid + - write account info to es + - send uuid to user + - mv mail file to drafts + +3. continue the monitor + - after 1 and 2, continue to monitor the mailbox + +4. service log + - use the following command to get the logs + docker logs -f --tail=100 fluentd |grep mail-robot -- 2.23.0

1 0

[PATCH v6 compass-ci 4/6] container/mail-robot: Dockerfile
by Luan Shengde 13 Oct '20

13 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..57307bf --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +MAINTAINER shdluan(a)163.com + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport && \ + ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ + echo 'Asia/Shanghai' > /etc/timezone + +COPY mail-robot.rb /usr/local/bin/ + +COPY answerback-email.rb /usr/local/bin/ + +WORKDIR /usr/local/bin/ -- 2.23.0

1 0

[PATCH v6 compass-ci 3/6] container/mail-robot: build file
by Luan Shengde 13 Oct '20

13 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/mail-robot/build diff --git a/container/mail-robot/build b/container/mail-robot/build new file mode 100755 index 0000000..c564add --- /dev/null +++ b/container/mail-robot/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t mail-robot:latest . -- 2.23.0

1 0

[PATCH v6 compass-ci 1/6] container/mail-robot: mail-robot.rb
by Luan Shengde 13 Oct '20

13 Oct '20

mail robot for application email monitor mailbox check new added email file in new check new mail's subject return if unmatched subject call answerback-email check to apply uuid/account Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/mail-robot.rb | 43 ++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100755 container/mail-robot/mail-robot.rb diff --git a/container/mail-robot/mail-robot.rb b/container/mail-robot/mail-robot.rb new file mode 100755 index 0000000..393b033 --- /dev/null +++ b/container/mail-robot/mail-robot.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'yaml' +require 'listen' +require 'mail' +require 'fileutils' +require_relative 'answerback-email.rb' + +mail_inbox = '/Maildir/.compass-ci/new/' +mail_drafts = '/Maildir/.compass-ci/cur/' + +def monitor_dir(mail_inbox, mail_drafts) + listener = Listen.to(mail_inbox) do |_modified, added, _removed| + unless added.empty? + added.each do |mail_file| + begin + check_to_send_account(mail_file, mail_drafts) + rescue StandardError => e + puts e.message + puts e.backtrace + end + end + end + end + listener.start + sleep +end + +def check_to_send_account(mail_file, mail_drafts) + mail_content = Mail.read(mail_file) + subject = mail_content.subject + return unless subject =~ /apply account/i + + assign_uuid = AssignUuid.new(mail_content) + assign_uuid.send_account + FileUtils.mv(mail_file, mail_drafts) +end + +monitor_dir(mail_inbox, mail_drafts) -- 2.23.0

1 0

[PATCH v4 compass-ci] feat: build centos-7 initramfs image with one command
by Wang Chenglong 13 Oct '20

13 Oct '20

Use the tool to create a new centos-7 initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/install-list. --- container/osimage/centos-7/README.md | 13 +++++++ container/osimage/centos-7/bin/create-image | 38 +++++++++++++++++++ container/osimage/centos-7/bin/lib | 30 +++++++++++++++ .../osimage/centos-7/bin/packages-to-install | 2 + .../osimage/centos-7/bin/packages-to-remove | 8 ++++ container/osimage/centos-7/build | 29 ++++++++++++++ 6 files changed, 120 insertions(+) create mode 100644 container/osimage/centos-7/README.md create mode 100755 container/osimage/centos-7/bin/create-image create mode 100755 container/osimage/centos-7/bin/lib create mode 100644 container/osimage/centos-7/bin/packages-to-install create mode 100644 container/osimage/centos-7/bin/packages-to-remove create mode 100755 container/osimage/centos-7/build diff --git a/container/osimage/centos-7/README.md b/container/osimage/centos-7/README.md new file mode 100644 index 0000000..d1a0317 --- /dev/null +++ b/container/osimage/centos-7/README.md @@ -0,0 +1,13 @@ +# Use the tool to create a new centos-7 initramfs image. + +Usage: + cd ${CCI_SRC/rootfs/initramfs/centos/aarch64/7} + ./build + +Some configuration items: +./bin/packages-to-install + If you want to pre-install the software, you can write the package names in ./bin/packages-to-install. + +./bin/packages-to-remove + If you want remove some unnecessary files, you can write the names in ./bin/packages-to-remove + diff --git a/container/osimage/centos-7/bin/create-image b/container/osimage/centos-7/bin/create-image new file mode 100755 index 0000000..214204f --- /dev/null +++ b/container/osimage/centos-7/bin/create-image @@ -0,0 +1,38 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Configure rootfs +make_rootfs() +{ + yum repolist + yum -y install --skip-broken $(</root/bin/packages-to-install) + yum clean all + rm -rf $(</root/bin/packages-to-remove) + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime + cd /root/modules + zcat modules-5.8.0.1.cgz | cpio -idm +} + +# Config password +pre_config_rootfs() { + [ -n "$ROOT_NEW_PASSWD" ] && { + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" # Change the password in shadow + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" # Configure ssh service + } +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + cd / + find ./ ! -path "./${image_name}" ! -path "./root/modules/modules-5.8.0.1.cgz" | cpio -o -Hnewc | gzip -9 > /${image_name} + chmod 644 /${image_name} +} + +make_rootfs +pre_config_rootfs +pack_cgz diff --git a/container/osimage/centos-7/bin/lib b/container/osimage/centos-7/bin/lib new file mode 100755 index 0000000..c3c470f --- /dev/null +++ b/container/osimage/centos-7/bin/lib @@ -0,0 +1,30 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(basename $(dirname $(realpath $0)))" +name="$(echo $DIR |awk -F "-" '{print $1}')" +version="$(echo $DIR |awk -F "-" '{print $2}')" +image="${name}:${version}" +image_name="${name}-${version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs.passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs.passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker() +{ + docker pull $image || return + echo "finish downloading image" +} + diff --git a/container/osimage/centos-7/bin/packages-to-install b/container/osimage/centos-7/bin/packages-to-install new file mode 100644 index 0000000..5d31c40 --- /dev/null +++ b/container/osimage/centos-7/bin/packages-to-install @@ -0,0 +1,2 @@ +openssh-server +openssl diff --git a/container/osimage/centos-7/bin/packages-to-remove b/container/osimage/centos-7/bin/packages-to-remove new file mode 100644 index 0000000..ce079b9 --- /dev/null +++ b/container/osimage/centos-7/bin/packages-to-remove @@ -0,0 +1,8 @@ +/.dockerenv +/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo diff --git a/container/osimage/centos-7/build b/container/osimage/centos-7/build new file mode 100755 index 0000000..5e46306 --- /dev/null +++ b/container/osimage/centos-7/build @@ -0,0 +1,29 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos-7/bin/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file "$root_pwd_file" + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/bin/:/root/bin + -v /srv/initrd/modules/:/root/modules:ro + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker + +docker_rm init_docker +${cmd[@]} + +docker cp -a init_docker:/${image_name} ./ +echo "build finished!!!" + -- 2.23.0

3 12

[PATCH compass-ci] sparrow: add docker test
by Liu Yinsi 12 Oct '20

12 Oct '20

1. submit job 2. run docker test Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/6-test/docker | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/sparrow/6-test/docker b/sparrow/6-test/docker index 3d6796d..ab825d3 100755 --- a/sparrow/6-test/docker +++ b/sparrow/6-test/docker @@ -1,7 +1,13 @@ -#!/bin/bash -e +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -## TODO -# 1.submit a job -# 2.run docker.sh +require_relative '../../providers/docker/docker' + +def run_job(hostname) + system "submit -s 'testbox: #{hostname}' #{ENV['CCI_SRC']}/user-client/jobs/iperf-sparrow.yaml" + main "#{hostname}" +end + +run_job "dc-1g" -- 2.23.0

2 4

[PATCH compass-ci] use local docker repository
by Xiao Shenwei 12 Oct '20

12 Oct '20

add DOCKER_REGISTRY_HOST, DOCKER_REGISTRY_PORT field into /etc/compass-ci/defaults/*.yaml, or $HOME/.config/compass-ci/defaults/*.yaml also, if failed from local registry, will pull from docker hub Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index c75fc2c..eea0336 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -11,14 +11,17 @@ require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' +names = Set.new %w[ + SCHED_HOST + SCHED_PORT + DOCKER_REGISTRY_HOST + DOCKER_REGISTRY_PORT +] +DEFAULTS = relevant_defaults(names) + def get_url(hostname) - names = Set.new %w[ - SCHED_HOST - SCHED_PORT - ] - defaults = relevant_defaults(names) - host = defaults['SCHED_HOST'] || '172.17.0.1' - port = defaults['SCHED_PORT'] || 3000 + host = DEFAULTS['SCHED_HOST'] || '172.17.0.1' + port = DEFAULTS['SCHED_PORT'] || 3000 "http://#{host}:#{port}/boot.container/hostname/#{hostname}" end @@ -59,7 +62,7 @@ def clean_dir(path) end def load_initrds(load_path, hash) - clean_dir(load_path) if Dir.exist?(load_path) + clean_dir(load_path) unless Dir.empty?(load_path) arch = RUBY_PLATFORM.split('-')[0] job_url = hash['job'] lkp_url = hash['lkp'] @@ -67,9 +70,18 @@ def load_initrds(load_path, hash) wget_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") end +def pull(image_name) + host = DEFAULTS['DOCKER_REGISTRY_HOST'] || '172.17.0.1' + port = DEFAULTS['DOCKER_REGISTRY_PORT'] || '5001' + flag = system "docker pull #{host}:#{port}/#{image_name}" + unless flag + system "docker pull #{image_name}" + end +end + def run(hostname, load_path, hash) docker_image = hash['docker_image'] - system "docker pull #{docker_image}" + pull docker_image system( { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, ENV['CCI_SRC'] + '/providers/docker/run.sh' -- 2.23.0

2 4

[PATCH compass-ci] fix: scheduler: update job root_result error
by Tong Qunfeng 12 Oct '20

12 Oct '20

1) after job finished, we save update job field back to es. 2) if the job is consumed at "20201010", but finished at "20201011". then the result_root will be update to "/result/$suite/$tbox_group/20201011/$id". 3) but the result is really saved at "/result/$suite/$tbox_group/20201010/$id", 20201010 is the date when it is consumed. no need to set_default for an initialized job (when is is consumed). Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 40 +++++++++++++++++++++++++++++++++++++--- src/lib/sched.cr | 1 + 2 files changed, 38 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 504bc19..be3446e 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -50,10 +50,13 @@ class Job @hash = job_content.as_h # init job with "-1", or use the original job_content["id"] - if "#{id}" == "" - @hash["id"] = JSON::Any.new("-1") + id == "-1" if "#{id}" == "" + + if initialized? + return if @hash["id"] == "#{id}" end + @hash["id"] = JSON::Any.new("#{id}") check_required_keys() set_defaults() end @@ -165,7 +168,7 @@ class Job self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end - private def set_result_root + def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group date = Time.local.to_s("%F") self["result_root"] = "/result/#{suite}/#{tbox_group}/#{date}/#{id}" @@ -234,6 +237,37 @@ class Job end end + private def initialized? + initialized_keys = [] of String + + REQUIRED_KEYS.each do |key| + initialized_keys << key.to_s + end + + METHOD_KEYS.each do |key| + initialized_keys << key.to_s + end + + INIT_FIELD.each do |key, _value| + initialized_keys << key.to_s + end + + initialized_keys += ["result_service", + "LKP_SERVER", + "LKP_CGI_PORT", + "SCHED_HOST", + "SCHED_PORT"] + + initialized_keys.each do |key| + if @hash.has_key?(key) == false + return false + end + end + + return false if "#{@hash["id"]}" == "" + return true + end + private def vmlinuz if os_mount == "initramfs" return "#{INITRD_HTTP_PREFIX}" + diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 23332df..2fbf05d 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -451,6 +451,7 @@ class Sched if job job.update({"testbox" => testbox}) + job.set_result_root @redis.set_job(job) end return job -- 2.23.0

1 0

[PATCH compass-ci] container/remote-git: disable encode
by Lin Jiaxin 12 Oct '20

12 Oct '20

encode change binary file's md5sums. error: ==> Validating source files with md5sums... VUE.jar ... Passed vue ... Passed vue.png ... FAILED vue.desktop ... Passed ==> ERROR: One or more files did not pass the validity check! md5sum vue.png 7253b39732630ba58186d7b79263b2a3 vue.png correct: ==> Validating source files with md5sums... VUE.jar ... Passed vue ... Passed vue.png ... Passed vue.desktop ... Passed md5sum vue.png 9369da34c69baefc77a7baeaceb4e476 vue.png Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- container/remote-git/remote-git.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/remote-git/remote-git.rb b/container/remote-git/remote-git.rb index 085f4db..44a8e18 100755 --- a/container/remote-git/remote-git.rb +++ b/container/remote-git/remote-git.rb @@ -41,7 +41,7 @@ post '/git_command' do # execute git command Dir.chdir(repo_path) _stdin, stdout, _stderr, wait_thr = Open3.popen3(*data['git_command']) - out = stdout.read.force_encoding('ISO-8859-1').encode('UTF-8') + out = stdout.read exit_code = wait_thr.value.to_i [200, headers.update({ 'errcode' => '0', 'exit_code' => exit_code.to_s }), out] -- 2.23.0

1 0

[PATCH v5 compass-ci 4/6] container/mail-robot: Dockerfile
by Luan Shengde 12 Oct '20

12 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/Dockerfile | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/mail-robot/Dockerfile diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile new file mode 100644 index 0000000..0eddbc4 --- /dev/null +++ b/container/mail-robot/Dockerfile @@ -0,0 +1,18 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian +MAINTAINER shdluan(a)163.com +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y uuid-runtime curl ruby-listen ruby-json ruby-mail && \ + gem install fileutils elasticsearch activesupport && \ + ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ + echo 'Asia/Shanghai' > /etc/timezone + +COPY mail-robot.rb /usr/local/bin/ + +COPY answerback-email.rb /usr/local/bin/ + +WORKDIR /usr/local/bin/ -- 2.23.0

2 1