October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/5] container/send-mail: send-mail.rb
by Luan Shengde 27 Oct '20

27 Oct '20

setup smtp config for email server invoke mail-post to hand the send mail request Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/send-mail.rb | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index f459e78..436736c 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -4,25 +4,14 @@ # frozen_string_literal: true require 'mail' +require 'sinatra' +require "#{ENV['CCI_SRC']}/lib/mail-post" -mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp +mail_server = %x(/sbin/ip route |awk '/default/ {print $3}').chomp -# setup smtp config smtp = { address: mail_server, enable_starttls_auto: false } Mail.defaults { delivery_method :smtp, smtp } - -# send mail -def send_mail(mail_info) - mail = Mail.new do - references mail_info['references'] - from mail_info['from'] - subject mail_info['subject'] - to mail_info['to'] - body mail_info['body'] - end - mail.deliver! -end -- 2.23.0

1 0

[PATCH lkp-tests] jobs: add cci-makepkg.yaml
by Zhang Yale 27 Oct '20

27 Oct '20

[Why] Now we run cci-makepkg to make packages. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- jobs/cci-makepkg.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/cci-makepkg.yaml diff --git a/jobs/cci-makepkg.yaml b/jobs/cci-makepkg.yaml new file mode 100644 index 00000000..5d437969 --- /dev/null +++ b/jobs/cci-makepkg.yaml @@ -0,0 +1,5 @@ +suite: cci-makepkg +category: functional + +cci-makepkg: + benchmark: -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs: add cci-depends.yaml
by Zhang Yale 27 Oct '20

27 Oct '20

[Why] Now we run cci-depends to make pkg depends. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- jobs/cci-depends.yaml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 jobs/cci-depends.yaml diff --git a/jobs/cci-depends.yaml b/jobs/cci-depends.yaml new file mode 100644 index 00000000..0867068d --- /dev/null +++ b/jobs/cci-depends.yaml @@ -0,0 +1,5 @@ +suite: cci-depends +category: functional + +cci-depends: + benchmark: -- 2.23.0

1 0

[PATCH compass-ci] /lib/es_query.rb: add range query
by Zhang Yuhang 27 Oct '20

27 Oct '20

[why] Some queries need to limit the range of fields. Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- lib/es_query.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index aaaadc2..3cf9852 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -42,6 +42,8 @@ def build_mutli_field_subquery_body(items) if value.is_a?(Array) inner_query = build_multi_field_or_query_body(key, value) query_fields.push({ bool: { should: inner_query } }) + elsif key.to_s == 'range' + query_fields.push({ range: value }) else query_fields.push({ term: { key => value } }) end -- 2.23.0

3 6

[PATCH v4 lkp-tests 2/2] tests/iozone: add specified parameter for iozone tool
by Lu Kaiyi 27 Oct '20

27 Oct '20

[why] need to test write, rewrite, read, reread, random write, random read for 4g size and different block_size. [How] add judement for variable $block_size to execute iozone command with specificed parameter. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/iozone b/tests/iozone index 3095ce80..c5f29d74 100755 --- a/tests/iozone +++ b/tests/iozone @@ -1,4 +1,5 @@ #!/bin/sh +# - block_size ## IOzone is a filesystem benchmark tool. The benchmark generates ## and measures a variety of file operations. @@ -7,7 +8,11 @@ for mnt in $mount_points do - cmd="iozone -a -f $mnt/iozonetest" + if [ -n "$block_size" ]; then + cmd="iozone -r $block_size -s 4g -i 0 -i 1 -i 2 -f $mnt/iozonetest" + else + cmd="iozone -a -f $mnt/iozonetest" + fi log_echo "$cmd &" $cmd > $TMP/iozone-$i & i=$((i+1)) -- 2.23.0

1 0

[PATCH v4 lkp-tests 1/2] jobs/iozone.yaml: add specified block_size parameter
by Lu Kaiyi 27 Oct '20

27 Oct '20

[why] need to test different block_size value Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone.yaml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/jobs/iozone.yaml b/jobs/iozone.yaml index f281186b..1e2239e5 100644 --- a/jobs/iozone.yaml +++ b/jobs/iozone.yaml @@ -2,6 +2,13 @@ suite: iozone testcase: iozone category: benchmark +block_size: +- 64k +- 256k +- 1m +- 4m +- 16m + disk: 2HDD fs: -- 2.23.0

1 0

[PATCH v7 compass-ci 5/6] container/send-internet-mail: start
by Luan Shengde 26 Oct '20

26 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/start | 39 ++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100755 container/send-internet-mail/start diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start new file mode 100755 index 0000000..483f01a --- /dev/null +++ b/container/send-internet-mail/start @@ -0,0 +1,39 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'send-internet-mail' + +names = Set.new %w[ + ROBOT_EMAIL_ADDRESS + ROBOT_EMAIL_PASSWORD + SEND_MAIL_PORT +] + +defaults = relevant_defaults(names) + +defaults['CCI_SRC'] = '/c/compass-ci' + +env = docker_env(defaults) + +cmd = %W[ + docker run + --restart=always + --name=send-internet-mail + -d +] + env + %W[ + -p #{defaults['SEND_MAIL_PORT']}:#{defaults['SEND_MAIL_PORT']} + -v #{ENV['CCI_SRC']}:#{defaults['CCI_SRC']} + -v /etc/localtime:/etc/localtime:ro + -w #{defaults['CCI_SRC']}/container/send-internet-mail + --log-driver json-file + send-internet-mail +] + +cmd += ['ruby', 'send-internet-mail.rb'] + +system(*cmd) -- 2.23.0

2 1

[PATCH compass-ci] sparrow: ignore os-nfs container
by Liu Yinsi 26 Oct '20

26 Oct '20

os-nfs service no need to deploy. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 26778a4..b3e5374 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -30,7 +30,8 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd $container - [ "$container_name" == 'sub-fluentd' ] && exit + [ "$container_name" == 'sub-fluentd' ] || \ + [ "$container_name" == 'os-nfs' ] && exit if [ -z "$start_only" ]; then [ -x build ] && ./build [ -x install ] && ./install -- 2.23.0

1 0

[PATCH compass-ci] container: docker run delete fluentd configration
by Wu Zhende 26 Oct '20

26 Oct '20

[Why] --log-driver has been configured in the global Docker. By default, each docker uses fluentd to collect logs and does not need to be configured separately. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/extract-stats/start | 7 ------- container/scheduler-https/start | 6 ------ container/scheduler/start | 6 ------ container/web-backend-nginx/start | 5 ----- container/web-backend/start | 5 ----- 5 files changed, 29 deletions(-) diff --git a/container/extract-stats/start b/container/extract-stats/start index e5ee777..74cf917 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -12,8 +12,6 @@ names = Set.new %w[ REDIS_PORT ES_HOST ES_PORT - FLUENTD_HOST - FLUENTD_PORT MAIL_HOST MAIL_PORT LAB @@ -24,8 +22,6 @@ env = docker_env(defaults) DEFAULT_LKP = '/c/lkp-tests' DEFAULT_CCI = '/c/compass-ci' -FLUENTD_HOST = defaults['FLUENTD_HOST'] || '172.17.0.1' -FLUENTD_PORT = defaults['FLUENTD_PORT'] || '24224' SRV_HTTP_HOST = defaults['SRV_HTTP_HOST'] || Open3.capture3('curl ip.sb')[0].chomp SRV_HTTP_PORT = defaults['SRV_HTTP_PORT'] || '11300' docker_rm "extract-stats" @@ -44,11 +40,8 @@ cmd = %w[ -v /etc/localtime:/etc/localtime:ro -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v /srv/result:/result - --log-driver=fluentd - --log-opt fluentd-address=#{FLUENTD_HOST}:#{FLUENTD_PORT} --log-opt mode=non-blocking --log-opt max-buffer-size=4m - --log-opt tag=extract-stats extract-stats ] diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 6b06769..38c4892 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -19,8 +19,6 @@ names = Set.new %w[ OS_HTTP_PORT INITRD_HTTP_HOST INITRD_HTTP_PORT - FLUENTD_HOST - FLUENTD_PORT TASKQUEUE_PORT TASKQUEUE_HOST ] @@ -34,8 +32,6 @@ SCHED_PORT = defaults['SCHED_PORT'] || '3000' SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] || 11306 SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] || 'api.compass-ci.openeuler.org' -FLUENTD_HOST = defaults['FLUENTD_HOST'] || '172.17.0.1' -FLUENTD_PORT = defaults['FLUENTD_PORT'] || '24224' docker_rm "s001-alpine-#{SCHED_HTTPS_PORT}" @@ -59,8 +55,6 @@ cmd = %W[ -v /srv/initrd:/srv/initrd:ro -v /srv/os:/srv/os:ro -v /etc/compass-ci/ca:/ca - --log-driver=fluentd - --log-opt fluentd-address=#{FLUENTD_HOST}:#{FLUENTD_PORT} --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_HTTPS_PORT} diff --git a/container/scheduler/start b/container/scheduler/start index 305916e..e0d70a8 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -17,8 +17,6 @@ names = Set.new %w[ OS_HTTP_PORT INITRD_HTTP_HOST INITRD_HTTP_PORT - FLUENTD_HOST - FLUENTD_PORT TASKQUEUE_PORT TASKQUEUE_HOST GIT_SERVER @@ -42,8 +40,6 @@ DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" LAB = defaults['lab'] || 'z9' SCHED_PORT = defaults['SCHED_PORT'] || '3000' SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' -FLUENTD_HOST = defaults['FLUENTD_HOST'] || '172.17.0.1' -FLUENTD_PORT = defaults['FLUENTD_PORT'] || '24224' FLUENTD_SERVER_PORT = defaults['FLUENTD_SERVER_PORT'] || '24224' docker_rm "s001-alpine-#{SCHED_PORT}" @@ -71,8 +67,6 @@ cmd = %W[ -v /srv/result:/srv/result -v /srv/initrd:/srv/initrd:ro -v /srv/os:/srv/os:ro - --log-driver=fluentd - --log-opt fluentd-address=#{FLUENTD_HOST}:#{FLUENTD_PORT} --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_PORT} diff --git a/container/web-backend-nginx/start b/container/web-backend-nginx/start index a28101d..4da4c8b 100755 --- a/container/web-backend-nginx/start +++ b/container/web-backend-nginx/start @@ -3,8 +3,6 @@ [[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci -[[ $FLUENTD_HOST ]] || FLUENTD_HOST=localhost -[[ $FLUENTD_PORT ]] || FLUENTD_PORT=24224 . $CCI_SRC/container/defconfig.sh docker_rm web-backend-nginx @@ -17,11 +15,8 @@ cmd=( -v /etc/ssl/certs:/opt/cert - --log-driver=fluentd - --log-opt fluentd-address=$FLUENTD_HOST:$FLUENTD_PORT --log-opt mode=non-blocking --log-opt max-buffer-size=4m - --log-opt tag=web-backend-nginx web-backend-nginx ) diff --git a/container/web-backend/start b/container/web-backend/start index 5fb493c..941fb11 100755 --- a/container/web-backend/start +++ b/container/web-backend/start @@ -5,8 +5,6 @@ [[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci [[ $LKP_SRC ]] || LKP_SRC=/c/lkp-tests [[ $UPSTREAM_REPOS_PATH ]] || UPSTREAM_REPOS_PATH=/c/upstream-repos -[[ $FLUENTD_HOST ]] || FLUENTD_HOST=localhost -[[ $FLUENTD_PORT ]] || FLUENTD_PORT=24224 . $CCI_SRC/container/defconfig.sh docker_rm web-backend @@ -27,11 +25,8 @@ cmd=( -v $LKP_SRC:/c/lkp-tests -v $UPSTREAM_REPOS_PATH:/c/upstream-repos - --log-driver=fluentd - --log-opt fluentd-address=$FLUENTD_HOST:$FLUENTD_PORT --log-opt mode=non-blocking --log-opt max-buffer-size=4m - --log-opt tag=web-backend debian:web-backend ) -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow: fix containers auto resume failed after reboot
by Liu Yinsi 26 Oct '20

26 Oct '20

call 4-docker/buildall to fix container auto resume depends issues. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 26 +++++++++++++++++--------- sparrow/7-systemd/cci-network | 9 +++++---- 2 files changed, 22 insertions(+), 13 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 230c6ac..26778a4 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -3,6 +3,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. CONTAINER_PATH="$CCI_SRC/container" +export start_only=$1 build_depends() { @@ -30,20 +31,27 @@ do_one() ( cd $container [ "$container_name" == 'sub-fluentd' ] && exit - [ -x build ] && ./build - [ -x install ] && ./install + if [ -z "$start_only" ]; then + [ -x build ] && ./build + [ -x install ] && ./install + fi [ -x first-run ] && ./first-run [ -x start ] && ./start ) lockfile-remove --lock-name "$container_name".lock } -tmpdir=$(mktemp -d) +buildall() +{ + tmpdir=$(mktemp -d) -for dir in $CONTAINER_PATH/*/ -do - build_depends $dir -done + for dir in $CONTAINER_PATH/*/ + do + build_depends $dir + done + + wait + rm -fr $tmpdir +} -wait -rm -fr $tmpdir +buildall diff --git a/sparrow/7-systemd/cci-network b/sparrow/7-systemd/cci-network index 4aadcdc..0d21a2d 100755 --- a/sparrow/7-systemd/cci-network +++ b/sparrow/7-systemd/cci-network @@ -4,13 +4,14 @@ #this file used in our own systemd unit file: cci-network.service. #contents as follows will be automatically executed after system reboot. -[[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci +. $CCI_SRC/container/defconfig.sh + +# export ENV +. /etc/profile.d/compass.sh $CCI_SRC/sparrow/2-network/br0 $CCI_SRC/sparrow/2-network/iptables $CCI_SRC/sparrow/2-network/nfs $CCI_SRC/sparrow/2-network/cifs -# --restart=always option is not absolutely reliable -containers=($(docker ps -a |grep -v NAMES |awk '{print $NF}')) -[ -n "$containers" ] && docker start "${containers[@]}" >/dev/null 2>&1 +$CCI_SRC/sparrow/4-docker/buildall start_only -- 2.23.0

1 0