- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 2/5] container/send-internet-mail: send-internet-mail.rb
by Luan Shengde 20 Oct '20

20 Oct '20

set up smtp config for smtp.qq.com send mail with recieved email data Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- .../send-internet-mail/send-internet-mail.rb | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100755 container/send-internet-mail/send-internet-mail.rb diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb new file mode 100755 index 0000000..d8c8d64 --- /dev/null +++ b/container/send-internet-mail/send-internet-mail.rb @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'mail' + +# setup smtp config +smtp = { + address: 'smtp.qq.com', + port: 25, + domain: 'qq.com', + user_name: ENV['EMAIL_ADDRESS'], + password: ENV['EMAIL_PASSWORD'], + openssl_verify_mode: 'none', + enable_starttls_auto: true +} + +Mail.defaults { delivery_method :smtp, smtp } + +# send mail +def send_mail(mail_info) + mail = Mail.new do + references mail_info['references'] + from ENV['EMAIL_ADDRESS'] + subject mail_info['subject'] + to mail_info['to'] + body mail_info['body'] + end + mail.deliver! +end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: merge overrides in advance
by Wei Jihui 20 Oct '20

20 Oct '20

[why] we have a new job yaml like: override: nr_task: - 100% - 50% << : jobs/unixbench.yaml overrides args:[100%, 50%] shoule be add job at the first time, so it will split to atomic job, nr_task:[nr_cpu*100%, nr_cpu*50%] job has overrides, so del revise_hash(job, deepcopy(@overrides), true), otherwise it will override job with nr_task:[100%, 50%], expand_params can not deal this list directly. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 2fd4567e..bca0a100 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -249,6 +249,7 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end + hash.merge!(@overrides) @jobs << hash end rescue StandardError => e @@ -404,7 +405,6 @@ class Job i = include_files job = deepcopy(@job) revise_hash(job, deepcopy(@job2), true) - revise_hash(job, deepcopy(@overrides), true) job['___'] = nil @jobx = job if @job.include?(:expand_params) -- 2.23.0

2 1

[PATCH lkp-tests] submit: mark .yaml when there is no =
by Wu Fengguang 20 Oct '20

20 Oct '20

This feels more symmetric logic. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 2475466f..a749ea20 100755 --- a/sbin/submit +++ b/sbin/submit @@ -60,7 +60,6 @@ options.parse!(ARGV) seen_job_yaml = false ARGV.delete_if do |arg| - seen_job_yaml = true if arg =~ /\.yaml$/ if arg.index '=' if seen_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') @@ -69,6 +68,7 @@ ARGV.delete_if do |arg| end true else + seen_job_yaml = true if arg =~ /\.yaml$/ false end end -- 2.23.0

1 0

[PATCH lkp-tests] submit: rename find_job_yaml to seen_job_yaml
by Wu Fengguang 20 Oct '20

20 Oct '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/submit | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sbin/submit b/sbin/submit index 979e3dca..2475466f 100755 --- a/sbin/submit +++ b/sbin/submit @@ -58,11 +58,11 @@ end options.parse!(ARGV) -find_job_yaml = false +seen_job_yaml = false ARGV.delete_if do |arg| - find_job_yaml = true if arg =~ /\.yaml$/ + seen_job_yaml = true if arg =~ /\.yaml$/ if arg.index '=' - if find_job_yaml + if seen_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') else opt_cmdline_defaults.merge! YAML.load arg.sub(/=/, ': ') -- 2.23.0

1 0

[PATCH v4 lkp-tests] tests: add get_ipmi_ip function to host-info
by Zhang Yu 20 Oct '20

20 Oct '20

output: ipmi_ip: 9.3.4.1 Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/host-info b/tests/host-info index a57855dc..5a4f590c 100755 --- a/tests/host-info +++ b/tests/host-info @@ -80,6 +80,24 @@ get_partitions() fi } +get_ipmi_ip() +{ + # ipmitool lan print + # Set in Progress : Set Complete + # IP Address Source : Static Address + # IP Address : 9.3.4.1 + # Subnet Mask : 255.255.0.0 + # MAC Address : e0:00:84:2b:51:7b + # SNMP Community String : TrapAdmin12#$ + # IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10 + # Default Gateway IP : 9.3.0.1 + # 802.1q VLAN ID : Disabled + # RMCP+ Cipher Suites : 0,1,2,3,17 + # Cipher Suite Priv Max : XuuaXXXXXXXXXXX + imp_ip=$(ipmitool lan print |awk 'NR==3{print $4}') + [ $? -eq 0 ] && echo "ipmi_ip:" $imp_ip +} + main() { get_node @@ -87,6 +105,7 @@ main() get_partitions get_mac get_model_name + get_ipmi_ip } main -- 2.23.0

4 5

[PATCH compass-ci] code-spec: add reuse rule
by Wu Fengguang 20 Oct '20

20 Oct '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index 52af5e6..37c86b6 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -1,6 +1,9 @@ code注意事项 ============ +- 在开始设计、编码之前，请善用搜索，善于复用现成code。 + 大部分基本问题在网上有现成的答案，往往还是最优解。 + - 代码应当放进函数里不要在脚本顶层直接写逻辑 - 代码应当放进lib/里脚本本身最小化, 仅解析命令行参数, 调用lib/入口函数 -- 2.23.0

1 0

[PATCH compass-ci] /src/lib/web_backend.rb: support multi-fields search in "/get_jobs"
by Zhang Yuhang 20 Oct '20

20 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 49 ++++++++++++++++++++++++------------------ 1 file changed, 28 insertions(+), 21 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 48cfadb..c8b6b66 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -47,7 +47,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end @@ -245,26 +245,36 @@ def es_search(must, size, from) } return {}, total if wrong_size?(size, from) + p query return es_query(query)['hits']['hits'], total end -def get_job(result) - job = {} - ALL_FIELDS.each do |f| - job[f] = result[f] +def get_jobs_result(result) + jobs = [] + result.each do |r| + job = {} + ALL_FIELDS.each do |f| + job[f] = r['_source'][f] + end + jobs << job end - job + jobs end -def search_job(git_repo, page_size, page_num) +def search_job(fields, page_size, page_num) must = [] - must << { regexp: { upstream_repo: ".*#{git_repo}.*" } } if git_repo - jobs = [] - result, total = es_search(must, page_size, page_num * page_size) - result.each do |r| - jobs << get_job(r['_source']) + FIELDS.each do |field| + value = space_to_nil(fields[field]) + next unless value + + must << if field == 'upstream_repo' + { regexp: { field => ".*#{value}.*" } } + else + { term: { field => value } } + end end - return jobs, total + result, total = es_search(must, page_size, page_num * page_size) + return get_jobs_result(result), total end def get_banner(git_repo, branches) @@ -291,17 +301,14 @@ def get_optimize_jobs_braches(jobs) end def get_jobs_body(params) - git_repo = space_to_nil(params[:upstream_repo]) - page_size = get_positive_number(params[:page_size], 20) - page_num = get_positive_number(params[:page_num], 1) - 1 - jobs, total = search_job(git_repo, page_size, page_num) + page_size = get_positive_number(params.delete(:page_size), 20) + page_num = get_positive_number(params.delete(:page_num), 1) - 1 + jobs, total = search_job(params, page_size, page_num) jobs, branches = get_optimize_jobs_braches(jobs) { total: total, - filter: { - upstream_repo: git_repo - }, - banner: get_banner(git_repo, branches), + filter: params, + banner: get_banner(params[:upstream_repo], branches), jobs: jobs, fields: FIELDS }.to_json -- 2.23.0

2 6

[PATCH v2 compass-ci] service/scheduler: add ssh port to job content
by Wu Zhende 20 Oct '20

20 Oct '20

[Why] Network users need to use sshr to login testbox. So testbox should set sshr. And testbox get sshr info from job content. Scheduler needs to add ssh info to job content Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/start | 4 ++++ src/lib/job.cr | 6 ++++++ 2 files changed, 10 insertions(+) diff --git a/container/scheduler/start b/container/scheduler/start index 81e7139..1183cde 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -24,10 +24,14 @@ names = Set.new %w[ GIT_SERVER FLUENTD_SERVER_HOST FLUENTD_SERVER_PORT + SSH_PORT_BASE + SSH_PORT_LEN lab ] defaults = relevant_defaults(names) +defaults["SSH_PORT_BASE"] = 50000 unless defaults["SSH_PORT_BASE"] +defaults["SSH_PORT_LEN"]= 2000 unless defaults["SSH_PORT_LEN"] env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..bd20828 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -139,6 +139,7 @@ class Job set_kernel_append_root() set_kernel_params() set_lkp_server() + set_ssh_port() if self["uuid"]? end private def append_init_field @@ -168,6 +169,11 @@ class Job end end + private def set_ssh_port + self["ssh_port_base"] = ENV["SSH_PORT_BASE"] + self["ssh_port_len"] = ENV["SSH_PORT_LEN"] + end + private def set_os_dir self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end -- 2.23.0

2 1

[PATCH compass-ci] code-spec: add rules
by Wu Fengguang 20 Oct '20

20 Oct '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/doc/code-spec.md b/doc/code-spec.md index d405cdd..52af5e6 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -2,12 +2,18 @@ code注意事项 ============ - 代码应当放进函数里不要在脚本顶层直接写逻辑 + - 代码应当放进lib/里脚本本身最小化, 仅解析命令行参数, 调用lib/入口函数 + - 如果出现了全局变量一般意味着需要写一个 class/module 来放这个状态量了 + - 一个函数, 不要超过10行代码, 该原则适用于所有ruby/crystal/shell各类程序. + - 起准确和有意义的变量/函数名字, 名字要让人看出赋予他们的确切意义 + - 函数要有分层结构. 每个函数须有明确的定位: 业务层逻辑, 还是底层支持性routine. 一个函数不能同时做两个层次的事. 同一函数内避免混杂不同类型的事务. + - 如果注释仅仅是代码的简单重复, 不提供额外信息量, 不要写. 重构友好编码 @@ -31,6 +37,7 @@ PATCH注意事项 ============= - PATCH title 应当简要说明做了什么 (what) + - PATCH changelog / code comment 应当解释读者无法从代码简单推断的 - 背景 - 目的 @@ -39,9 +46,18 @@ PATCH注意事项 - 实际例子 - 改进数字 - bug场景, 出错消息原文 + - PATCH 应当加上作者签名 (Signed-off-by: Name <email>) + - PATCH 最好不超过100行, 一个PATCH只做一件事, 避免混杂多个逻辑变更 +- changelog/comment里的话，需理顺逻辑关系，让人一看就知道在讲什么， + 比如问题、原因、条件、结果、预期、假设、场景、目的、副作用、权衡， + 哪些是因果关系，哪些是可能，哪些是肯定。 + +- 论述问题的时候，引用必要的代码或output，帮助明确逻辑流/数据流。 + 引文加TAB缩进显示。 + PATCH参考材料 ============= -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] jobs: add test-tools.yaml
by Li Ping 20 Oct '20

20 Oct '20

Signed-off-by: Li Ping <15396232681(a)163.com> --- jobs/test-tools.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 jobs/test-tools.yaml diff --git a/jobs/test-tools.yaml b/jobs/test-tools.yaml new file mode 100644 index 00000000..1f711dcf --- /dev/null +++ b/jobs/test-tools.yaml @@ -0,0 +1,6 @@ +suite: test-tools +category: functional + +test-tools: + testsuite: + case: -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] delete the field: testcase
by Li Ping 20 Oct '20

20 Oct '20

[why] there is no need to add the field: testcase in the $job.yaml Signed-off-by: Li Ping <15396232681(a)163.com> --- bin/lkp-setup-rootfs | 2 +- lib/bootstrap.sh | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index 09f7e7eb..05a3325c 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -51,7 +51,7 @@ job_does_not_need_reboot() } grep -q "^do_not_reboot_for_same_kernel: 1$" $job && - is_same_kernel_and_rootfs && is_same_testcase && is_same_bp_memmap && { + is_same_kernel_and_rootfs && is_same_suite && is_same_bp_memmap && { echo "LKP: [do_not_reboot_for_same_kernel] start a new job without rebooting" return 0 } diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 2506c021..a7776d68 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -631,12 +631,12 @@ is_same_kernel_and_rootfs() return 1 } -is_same_testcase() +is_same_suite() { - local current_testcase=$testcase - local next_testcase=$(awk '/^testcase: /{print $2}' $job | tr -d '"') + local current_suite=$suite + local next_suite=$(awk '/^suite: /{print $2}' $job | tr -d '"') - [ "$current_testcase" = "$next_testcase" ] + [ "$current_suite" = "$next_suite" ] } is_same_bp_memmap() -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: submit with yaml file without directory
by Wei Jihui 20 Oct '20

20 Oct '20

if get string named with *.yaml from ARGV, treat it as job yaml. even it is without directory. and find_jobfile check whether the job yaml exists. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 831bfb5a..979e3dca 100755 --- a/sbin/submit +++ b/sbin/submit @@ -60,7 +60,7 @@ options.parse!(ARGV) find_job_yaml = false ARGV.delete_if do |arg| - find_job_yaml = true if File.file?(arg) + find_job_yaml = true if arg =~ /\.yaml$/ if arg.index '=' if find_job_yaml opt_set_key_value.merge! YAML.load arg.sub(/=/, ': ') -- 2.23.0

3 2

[PATCH v2 compass-ci] add exception handing for docker.rb
by Xiao Shenwei 20 Oct '20

20 Oct '20

[why] when run multi-docker job on physical machine, run 50 client to request job, if server down or error like Connection refused. these client willexit. when the server recovers, need to run these client again. [how] add exception handing to ensure all client does not exit Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index c241d3c..3600025 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -88,6 +88,19 @@ def main(hostname) run(hostname, load_path, hash) end +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.backtrace + sleep 25 + ensure + sleep 5 + end + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -99,10 +112,7 @@ def multi_docker(hostname, nr_container) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop do - main("#{hostname}-#{i}") - sleep 5 - end + loop_main("#{hostname}-#{i}") end pids << pid end -- 2.23.0

2 2

[PATCH v3 compass-ci] feat: build centos initramfs image with one command
by Wang Chenglong 20 Oct '20

20 Oct '20

Use the tool to create a new centos-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/packages-to-install. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/centos/README.md | 16 ++++++++ container/osimage/centos/create-image | 43 ++++++++++++++++++++ container/osimage/centos/lib | 41 +++++++++++++++++++ container/osimage/centos/packages-to-install | 9 ++++ container/osimage/centos/packages-to-remove | 11 +++++ container/osimage/centos/run | 27 ++++++++++++ 6 files changed, 147 insertions(+) create mode 100644 container/osimage/centos/README.md create mode 100755 container/osimage/centos/create-image create mode 100755 container/osimage/centos/lib create mode 100644 container/osimage/centos/packages-to-install create mode 100644 container/osimage/centos/packages-to-remove create mode 100755 container/osimage/centos/run diff --git a/container/osimage/centos/README.md b/container/osimage/centos/README.md new file mode 100644 index 0000000..b5c7b07 --- /dev/null +++ b/container/osimage/centos/README.md @@ -0,0 +1,16 @@ ++# Use the tool to create a new centos-${version} initramfs image. ++ ++Usage: ++ cd ${CCI_SRC}/rootfs/initramfs/centos/aarch64/${version}/ ++ ./build ++ ++Some configuration items: ++./bin/packages-to-install ++ If you want to pre-install the software, you can write the package names in ./bin/packages-to-install. ++ ++./bin/packages-to-remove ++ If you want remove some unnecessary files, you can write the names in ./bin/packages-to-remove. ++ ++$HOME/.config/compass-ci/rootfs-passwd ++ Set the password for the image into this file. + diff --git a/container/osimage/centos/create-image b/container/osimage/centos/create-image new file mode 100755 index 0000000..a3495b2 --- /dev/null +++ b/container/osimage/centos/create-image @@ -0,0 +1,43 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Install functions +install_packages() +{ + echo "install" + yum repolist + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +# Setup functions +setup_rootfs() +{ + echo "setup" + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + + +# Config password +setup_login() { + [ -n "$ROOT_NEW_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${image_name}" |grep -vf /tmp/packages-to-remove | cpio -o -Hnewc | gzip -9 > /tmp/${image_name} + chmod 644 /tmp/${image_name} +} + +install_packages +setup_rootfs +setup_login +pack_cgz diff --git a/container/osimage/centos/lib b/container/osimage/centos/lib new file mode 100755 index 0000000..53aae2a --- /dev/null +++ b/container/osimage/centos/lib @@ -0,0 +1,41 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${os_name}:${os_version}" +image_name="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs-passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker_image() +{ + docker pull $image + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[INFO] Pull docker image failed." + exit + fi +} + +# cp image package to host +cp_image() +{ + docker cp -a init_docker:/tmp/${image_name} $HOME/ + echo "result: $(ls $HOME/${image_name})" +} diff --git a/container/osimage/centos/packages-to-install b/container/osimage/centos/packages-to-install new file mode 100644 index 0000000..a09bc28 --- /dev/null +++ b/container/osimage/centos/packages-to-install @@ -0,0 +1,9 @@ +openssh-server +openssl +cifs-utils +hostname +curl +iproute +iputils +which +time diff --git a/container/osimage/centos/packages-to-remove b/container/osimage/centos/packages-to-remove new file mode 100644 index 0000000..07a4a23 --- /dev/null +++ b/container/osimage/centos/packages-to-remove @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/packages-to-remove +/tmp/packages-to-install diff --git a/container/osimage/centos/run b/container/osimage/centos/run new file mode 100755 index 0000000..699bbc6 --- /dev/null +++ b/container/osimage/centos/run @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/packages-to-remove:/tmp/packages-to-remove + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker_image +${cmd[@]} +cp_image +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

2 2

[PATCH v4 compass-ci] feat: build centos initramfs image with one command
by Wang Chenglong 20 Oct '20

20 Oct '20

Use the tool to create a new centos-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/packages-to-install. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/centos/README.md | 16 ++++++++ container/osimage/centos/create-image | 43 ++++++++++++++++++++ container/osimage/centos/lib | 41 +++++++++++++++++++ container/osimage/centos/packages-to-install | 9 ++++ container/osimage/centos/packages-to-remove | 11 +++++ container/osimage/centos/run | 27 ++++++++++++ 6 files changed, 147 insertions(+) create mode 100644 container/osimage/centos/README.md create mode 100755 container/osimage/centos/create-image create mode 100755 container/osimage/centos/lib create mode 100644 container/osimage/centos/packages-to-install create mode 100644 container/osimage/centos/packages-to-remove create mode 100755 container/osimage/centos/run diff --git a/container/osimage/centos/README.md b/container/osimage/centos/README.md new file mode 100644 index 0000000..b5c7b07 --- /dev/null +++ b/container/osimage/centos/README.md @@ -0,0 +1,16 @@ ++# Use the tool to create a new centos-${version} initramfs image. ++ ++Usage: ++ cd ${CCI_SRC}/rootfs/initramfs/centos/aarch64/${version}/ ++ ./build ++ ++Some configuration items: ++./bin/packages-to-install ++ If you want to pre-install the software, you can write the package names in ./bin/packages-to-install. ++ ++./bin/packages-to-remove ++ If you want remove some unnecessary files, you can write the names in ./bin/packages-to-remove. ++ ++$HOME/.config/compass-ci/rootfs-passwd ++ Set the password for the image into this file. + diff --git a/container/osimage/centos/create-image b/container/osimage/centos/create-image new file mode 100755 index 0000000..fac8ddc --- /dev/null +++ b/container/osimage/centos/create-image @@ -0,0 +1,43 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Install functions +install_packages() +{ + echo "install" + yum repolist + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +# Setup functions +setup_rootfs() +{ + echo "setup" + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + + +# Config password +setup_login() { + [ -n "$ROOT_NEW_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${image_name}" | grep -vf /tmp/packages-to-remove | cpio -o -Hnewc | gzip -9 > /tmp/${image_name} + chmod 644 /tmp/${image_name} +} + +install_packages +setup_rootfs +setup_login +pack_cgz diff --git a/container/osimage/centos/lib b/container/osimage/centos/lib new file mode 100755 index 0000000..53aae2a --- /dev/null +++ b/container/osimage/centos/lib @@ -0,0 +1,41 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${os_name}:${os_version}" +image_name="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs-passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker_image() +{ + docker pull $image + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[INFO] Pull docker image failed." + exit + fi +} + +# cp image package to host +cp_image() +{ + docker cp -a init_docker:/tmp/${image_name} $HOME/ + echo "result: $(ls $HOME/${image_name})" +} diff --git a/container/osimage/centos/packages-to-install b/container/osimage/centos/packages-to-install new file mode 100644 index 0000000..a09bc28 --- /dev/null +++ b/container/osimage/centos/packages-to-install @@ -0,0 +1,9 @@ +openssh-server +openssl +cifs-utils +hostname +curl +iproute +iputils +which +time diff --git a/container/osimage/centos/packages-to-remove b/container/osimage/centos/packages-to-remove new file mode 100644 index 0000000..07a4a23 --- /dev/null +++ b/container/osimage/centos/packages-to-remove @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/packages-to-remove +/tmp/packages-to-install diff --git a/container/osimage/centos/run b/container/osimage/centos/run new file mode 100755 index 0000000..699bbc6 --- /dev/null +++ b/container/osimage/centos/run @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/packages-to-remove:/tmp/packages-to-remove + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker_image +${cmd[@]} +cp_image +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

3 6

[PATCH v3 compass-ci] src/lib: find depends_initrd for multitask
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the code need multi-qemu, not multi-qemu-x, and scheduler can not find the depends_initrd multi-qemu-x.cgz. Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..93bc712 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,9 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + if program =~ /(.*)-\d+$/ + program = $1 + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +415,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

1 0

[PATCH compass-ci] src/lib: find depends_initrd for multitask
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the code need multi-qemu, not multi-qemu-x, and scheduler can not find the depends_initrd multi-qemu-x.cgz. Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..a0d7263 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,9 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + if program =~ /(.*)(\-\d{1,}$)/ + program = $1 + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +415,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq!.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq!.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

1 0

[PATCH compass-ci] code-spec: add some rules
by Wu Fengguang 20 Oct '20

20 Oct '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- doc/code-spec.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/doc/code-spec.md b/doc/code-spec.md index a859239..d405cdd 100644 --- a/doc/code-spec.md +++ b/doc/code-spec.md @@ -4,8 +4,11 @@ code注意事项 - 代码应当放进函数里不要在脚本顶层直接写逻辑 - 代码应当放进lib/里脚本本身最小化, 仅解析命令行参数, 调用lib/入口函数 - 如果出现了全局变量一般意味着需要写一个 class/module 来放这个状态量了 -- 一个函数, 不要超过10行代码 +- 一个函数, 不要超过10行代码, 该原则适用于所有ruby/crystal/shell各类程序. - 起准确和有意义的变量/函数名字, 名字要让人看出赋予他们的确切意义 +- 函数要有分层结构. 每个函数须有明确的定位: 业务层逻辑, 还是底层支持性routine. + 一个函数不能同时做两个层次的事. 同一函数内避免混杂不同类型的事务. +- 如果注释仅仅是代码的简单重复, 不提供额外信息量, 不要写. 重构友好编码 ============ -- 2.23.0

1 0

[PATCH v2 compass-ci] src/lib: for multitask like multi-qemu
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the code need multi-qemu, not multi-qemu-x. Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..a48f233 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,9 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + if program =~ /(.*)(\-\d{1,}$)/ + program_num = $1 + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +415,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq!.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq!.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

2 2

[PATCH v3 compass-ci] fix: sched: set 2 fields to job: direct_macs, direct_ips
by Ren Wen 20 Oct '20

20 Oct '20

[Why] nodes need communicate with each other when running cluster jobs. after adding 2 fileds, 'direct_ips' and 'direct_macs', can bind ips to macs. 'macs' get from cluster file ('$LKP_SRC/cluster/*'), 'ips' set by scheduler default: 192.168.<211..254>.<host_id> Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a18bdee..1d5b5c4 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -198,6 +198,10 @@ class Sched # collect all job ids job_ids = [] of String + # used to set ips + ip_prefix = "192.168." + host_id = 1 + # steps for each host cluster_config.each do |host, config| tbox_group = host.to_s @@ -219,7 +223,18 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + + # host macs and host ips + direct_macs = config["macs"].as_a + direct_ips = [] of String + net_id = "#{ip_prefix}#{254 - job_id.hash%44}" # 192.168.<211..254> + direct_macs.size.times do + raise "more than 255 macs" if host_id > 255 + direct_ips << "#{net_id}.#{host_id}" + host_id += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

2 2

[PATCH compass-ci] /src/lib/web_backend.rb: fix "/compare_candidates" tbox_group regex bug
by Zhang Yuhang 20 Oct '20

20 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 48cfadb..fda01ca 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -47,7 +47,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end -- 2.23.0

2 2

[PATCH lab-z9] hosts: add new hostfile
by Zhang Yu 20 Oct '20

20 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a101 | 22 ++++++++++++++++++++++ hosts/taishan200-2280-2s48p-256g--a102 | 24 ++++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a101 create mode 100644 hosts/taishan200-2280-2s48p-256g--a102 diff --git a/hosts/taishan200-2280-2s48p-256g--a101 b/hosts/taishan200-2280-2s48p-256g--a101 new file mode 100644 index 0000000..1226913 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a101 @@ -0,0 +1,22 @@ +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 3 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b89192e1 + - /dev/disk/by-id/scsi-350000399b88bf66d + - /dev/disk/by-id/scsi-350000399b88bef85 +ssd_partitions: + - /dev/disk/by-id/ata-INTEL_SSDSC2KB480G8_BTYF94730FF4480BGN +mac_addr: + - 44:67:47:d7:6c:9c + - 44:67:47:d7:6c:9d + - 44:67:47:d7:6c:9e + - 44:67:47:d7:6c:9f + - 44:67:47:d7:6c:a0 + - 44:67:47:d7:6c:a1 + - 44:67:47:d7:6c:a2 + - 44:67:47:d7:6c:a3 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.6 diff --git a/hosts/taishan200-2280-2s48p-256g--a102 b/hosts/taishan200-2280-2s48p-256g--a102 new file mode 100644 index 0000000..267a962 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a102 @@ -0,0 +1,24 @@ +nr_node: 4 +nr_cpu: 96 +memory: 512G +nr_hdd_partitions: 2 +nr_ssd_partitions: 3 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88bfbc9 + - /dev/disk/by-id/scsi-350000399b8919395 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH1T9HMLT-00005_S455NY0MB55506 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH1T9HMLT-00005_S455NY0MB55187 + - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000335 + - nvme-nvme.19e5-30333242454a31304743303030333335-48574533365034333031324d3030304e-00000001-part* +mac_addr: + - 44:67:47:c9:ea:30 + - 44:67:47:c9:ea:31 + - 44:67:47:c9:ea:32 + - 44:67:47:c9:ea:33 + - 44:67:47:c9:ea:34 + - 44:67:47:c9:ea:35 + - 44:67:47:c9:ea:36 + - 44:67:47:c9:ea:37 +model_name: Kunpeng-920 +ipmi_ip: 9.3.10.7 -- 2.23.0

1 0

[PATCH compass-ci] src/lib: for multitask like multi-qemu
by Li Ping 20 Oct '20

20 Oct '20

[why] when the job.yaml like multi-qemu.yaml: multi-qemu-0: multi-qemu-1: the scheduler can not find the program multi-qemu-0 and the program multi-qemu-1 Signed-off-by: Li Ping <15396232681(a)163.com> --- src/lib/job.cr | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1e44d15..e563ddf 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -399,6 +399,10 @@ class Job if @hash["pp"]? program_params = @hash["pp"].as_h program_params.keys.each do |program| + program_num = program.match(/(.*)(\-\d{1,}$)/) + if program_num != nil + program = program_num.not_nil![1] + end deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" @@ -412,8 +416,8 @@ class Job end end - self["initrd_deps"] = initrd_deps_arr.join(" ") - self["initrd_pkg"] = initrd_pkg_arr.join(" ") + self["initrd_deps"] = initrd_deps_arr.uniq!.join(" ") + self["initrd_pkg"] = initrd_pkg_arr.uniq!.join(" ") end def update_tbox_group(tbox_group) -- 2.23.0

3 3

[PATCH v2 compass-ci] fix: sched: set 2 fields to job: direct_macs, direct_ips
by Ren Wen 20 Oct '20

20 Oct '20

[Why] nodes need communicate with each other when running cluster jobs. after adding 2 fileds, 'direct_ips' and 'direct_macs', can bind ips to macs. 'macs' get from cluster file ('$LKP_SRC/cluster/*'), 'ips' set by scheduler default. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a18bdee..82d3610 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -198,6 +198,10 @@ class Sched # collect all job ids job_ids = [] of String + # used to set ips + ip_prefix = "192.169." + host_id = 1 + # steps for each host cluster_config.each do |host, config| tbox_group = host.to_s @@ -219,7 +223,17 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + + # host macs and host ips + direct_macs = config["macs"].as_a + direct_ips = [] of String + net_id = "#{ip_prefix}#{job_id.hash%255}" # 192.169.<0..254> + direct_macs.size.times do + direct_ips << "#{net_id}.#{host_id}" + host_id += 1 + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

1 1

[PATCH v7 compass-ci 2/6] container/mail-robot: answerback-email.rb
by Luan Shengde 20 Oct '20

20 Oct '20

AssignUuid: read email content check email available check if base url in upstream-repos check if commit url valid check pub_key assign account generate uuid write account/uuid to es send uuid to user Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/answerback-email.rb | 229 +++++++++++++++++++++++ 1 file changed, 229 insertions(+) create mode 100755 container/mail-robot/answerback-email.rb diff --git a/container/mail-robot/answerback-email.rb b/container/mail-robot/answerback-email.rb new file mode 100755 index 0000000..66d13e4 --- /dev/null +++ b/container/mail-robot/answerback-email.rb @@ -0,0 +1,229 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '/c/compass-ci/lib/es_client.rb' + +# Assign uuid/account +class AssignUuid + def initialize(mail_content) + @mail_content = mail_content + @send_mail_server = %x(/sbin/ip route |awk '/default/ {print $3}').chomp + end + + def build_message(sum_infos) + message = <<~EMAIL_MESSAGE + To: #{sum_infos['my_email']} + Subject: Account Ready + + Dear #{sum_infos['my_name']} + + Thank you for joining us. + You can use the following info to submit jobs: + + First: add the info to ~/.config/compass-ci/defaults/{USER}.yaml + + uuid: #{sum_infos['my_uuid']} + lab: z9 + + Second: download lkp-tests and dependencies + git clone https://gitee.com_/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source {HOME}/.bashrc && source {HOME}/.bash_profile + + Third: submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + how to write job yaml + submit jobs/netperf.yaml + + lastly: add your own information and uuid to ~/.config/compass-ci/default/{USER}.yml + my_name: {your name} + my_email: {your email address} + my_uuid: {applied uuid} + + Repalce the fields {...} with your local environment variable. + + regards + compass-ci + EMAIL_MESSAGE + + return message + end + + def build_err_message(email, message) + message = <<~EMAIL_MESSAGE + To: #{email} + Subject: apply account failed + + Dear #{email.split('@')[0]} + + Your application for account failed with following errors + #{message} + + regards + compass-ci + EMAIL_MESSAGE + end + + def send_mail(email_msg) + %x(curl -XPOST "#{@send_mail_server}:#{ENV['S_M_PORT']}/send_mail_text" -d "#{email_msg}") + end + + def url_addr(email) + mail_body = @mail_content.part[0].body.decoded + mail_content = mail_body.gsub(/=/, '').split(/\r|\n|=/).join + url = %x(echo '#{mail_content}' |grep -o -E "https?://[^/]*/[^/]*/[^/]*/[^/]*/[0-9a-zA-Z]{40}").split(/\n/)[0] + + if url.nil? + message = 'No commit url found' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise message + end + + return url + end + + def check_email_available(email) + url = url_addr(email).chomp + + base_url = url.gsub(%r{/[^/]*/[0-9a-zA-Z]{40}$}, '') + + upstream_dir = '/c/upstream-repos' + check_url_in_upstream_repos(upstream_dir, base_url, email) + check_url_commit(url, base_url, email) + + return url + end + + def check_url_in_upstream_repos(upstream_dir, base_url, email) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + if match_out.empty? + message = 'The url is not in upstream-repo list' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise StandardError, message, base_url + end + end + + def check_url_commit(url, base_url, email) + hub_name = url.split('/')[2] + + # it requires authentication when execute curl to get the commit infomation + # clone the repo and then validate the commit for the email address + check_gitee_commit(url, base_url, email) if hub_name.eql? 'gitee.com' + check_non_gitee_commit(url, email) unless hub_name.eql? 'gitee.com' + end + + def check_non_gitee_commit(url, email) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index email + + if email_index.nil? + message = 'Your commit url is not a valid commit url.' + email_msg = build_err_message(email, message) + send_mail(email_msg) + raise StandardError, message, url + end + end + + def check_gitee_commit(url, base_url, email) + repo_info = { + 'repos_dir' => '/repos_dir', + 'repo_dir' => url.split('/')[-3], + 'repo_url' => [base_url, 'git'].join('.'), + 'repo_path' => File.join('/repos_dir', url.split('/')[-3]), + 'commit_id' => url.split('/')[-1] + } + check_gitee_repo_commit(url, email, repo_info) + end + + def check_gitee_repo_commit(url, email, repo_info) + Dir.chdir repo_info['repos_dir'] + %x(/usr/bin/git clone #{repo_info['repo_url']}) + email_index = %x(/usr/bin/git -C #{repo_info['repo_dir']} show #{repo_info['commit_id']}).index email + + FileUtils.rm_rf repo_info['repo_dir'] + + if email_index.nil? + message = 'Your commit url is not a valid commit url.' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise StandardError, message, url + end + end + + def pub_key_value(email) + pub_key = @mail_content.part[1].body.decoded.split(/\r|\n/).join if @mail_content.part[1].filename == 'id_rsa.pub' + + unless pub_key + message = 'No pub_key found' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise message + end + + return pub_key + end + + def account_info(pub_key, email) + account_info_str = %x(curl -XGET "#{ENV['J_HOST']}:#{ENV['J_PORT']}/assign_account" -d "pub_key: #{pub_key}") + account_info = JSON.parse account_info_str + + if account_info['account'].empty? + message = 'No more available jumper account.' + email_msg = build_err_message(email, message) + send_mail(email_msg) + + raise StandardError, message, account_info if account_info['account'].empty? + end + + return account_info + end + + def build_sum_infos + email = @mail_content.from[0] + my_name = @mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '') + commit_url = check_email_available(email) + pub_key = pub_key_value(email) + acct_infos = account_info(pub_key, email) + account_uuid = %x(uuidgen).chomp + sum_infos = { + 'my_email' => email, + 'my_name' => my_name, + 'my_commit_url' => commit_url, + 'my_login_name' => acct_infos['account'], + 'my_uuid' => account_uuid, + } + store_account_info(sum_infos) + sum_infos['acct_infos'] = acct_infos + sum_infos + end + + def send_uuid(sum_infos) + email_msg = build_message(sum_infos) + + send_mail(email_msg) + end + + def send_account + sum_infos = build_sum_infos + + send_uuid(sum_infos) + end + + def store_account_info(sum_infos) + es = ESClient.new(index: 'accounts') + es.put_source_by_id(sum_infos['my_email'], sum_infos) + end +end -- 2.23.0

2 4