- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] container: docker run delete fluentd configration
by Wu Zhende 28 Oct '20

28 Oct '20

[Why] --log-driver has been configured in the global Docker. By default, each docker uses fluentd to collect logs and does not need to be configured separately. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/extract-stats/start | 7 ------- container/scheduler-https/start | 6 ------ container/scheduler/start | 6 ------ container/web-backend-nginx/start | 5 ----- container/web-backend/start | 5 ----- 5 files changed, 29 deletions(-) diff --git a/container/extract-stats/start b/container/extract-stats/start index e5ee777..74cf917 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -12,8 +12,6 @@ names = Set.new %w[ REDIS_PORT ES_HOST ES_PORT - FLUENTD_HOST - FLUENTD_PORT MAIL_HOST MAIL_PORT LAB @@ -24,8 +22,6 @@ env = docker_env(defaults) DEFAULT_LKP = '/c/lkp-tests' DEFAULT_CCI = '/c/compass-ci' -FLUENTD_HOST = defaults['FLUENTD_HOST'] || '172.17.0.1' -FLUENTD_PORT = defaults['FLUENTD_PORT'] || '24224' SRV_HTTP_HOST = defaults['SRV_HTTP_HOST'] || Open3.capture3('curl ip.sb')[0].chomp SRV_HTTP_PORT = defaults['SRV_HTTP_PORT'] || '11300' docker_rm "extract-stats" @@ -44,11 +40,8 @@ cmd = %w[ -v /etc/localtime:/etc/localtime:ro -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} -v /srv/result:/result - --log-driver=fluentd - --log-opt fluentd-address=#{FLUENTD_HOST}:#{FLUENTD_PORT} --log-opt mode=non-blocking --log-opt max-buffer-size=4m - --log-opt tag=extract-stats extract-stats ] diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 6b06769..38c4892 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -19,8 +19,6 @@ names = Set.new %w[ OS_HTTP_PORT INITRD_HTTP_HOST INITRD_HTTP_PORT - FLUENTD_HOST - FLUENTD_PORT TASKQUEUE_PORT TASKQUEUE_HOST ] @@ -34,8 +32,6 @@ SCHED_PORT = defaults['SCHED_PORT'] || '3000' SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_HTTPS_PORT = defaults['SCHED_HTTPS_PORT'] || 11306 SCHED_HTTPS_HOST = defaults['SCHED_HTTPS_HOST'] || 'api.compass-ci.openeuler.org' -FLUENTD_HOST = defaults['FLUENTD_HOST'] || '172.17.0.1' -FLUENTD_PORT = defaults['FLUENTD_PORT'] || '24224' docker_rm "s001-alpine-#{SCHED_HTTPS_PORT}" @@ -59,8 +55,6 @@ cmd = %W[ -v /srv/initrd:/srv/initrd:ro -v /srv/os:/srv/os:ro -v /etc/compass-ci/ca:/ca - --log-driver=fluentd - --log-opt fluentd-address=#{FLUENTD_HOST}:#{FLUENTD_PORT} --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_HTTPS_PORT} diff --git a/container/scheduler/start b/container/scheduler/start index 6a2a4b0..8ca12f0 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -17,8 +17,6 @@ names = Set.new %w[ OS_HTTP_PORT INITRD_HTTP_HOST INITRD_HTTP_PORT - FLUENTD_HOST - FLUENTD_PORT TASKQUEUE_PORT TASKQUEUE_HOST GIT_SERVER @@ -42,8 +40,6 @@ DEFAULT_CCI = "#{CCI_REPOS}/compass-ci" LAB = defaults['lab'] || 'z9' SCHED_PORT = defaults['SCHED_PORT'] || '3000' SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' -FLUENTD_HOST = defaults['FLUENTD_HOST'] || '172.17.0.1' -FLUENTD_PORT = defaults['FLUENTD_PORT'] || '24224' FLUENTD_SERVER_PORT = defaults['FLUENTD_SERVER_PORT'] || '24224' DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' @@ -79,8 +75,6 @@ cmd = %W[ -v /srv/result:/srv/result -v /srv/initrd:/srv/initrd:ro -v /srv/os:/srv/os:ro - --log-driver=fluentd - --log-opt fluentd-address=#{FLUENTD_HOST}:#{FLUENTD_PORT} --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt tag=scheduler-#{SCHED_PORT} diff --git a/container/web-backend-nginx/start b/container/web-backend-nginx/start index a28101d..4da4c8b 100755 --- a/container/web-backend-nginx/start +++ b/container/web-backend-nginx/start @@ -3,8 +3,6 @@ [[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci -[[ $FLUENTD_HOST ]] || FLUENTD_HOST=localhost -[[ $FLUENTD_PORT ]] || FLUENTD_PORT=24224 . $CCI_SRC/container/defconfig.sh docker_rm web-backend-nginx @@ -17,11 +15,8 @@ cmd=( -v /etc/ssl/certs:/opt/cert - --log-driver=fluentd - --log-opt fluentd-address=$FLUENTD_HOST:$FLUENTD_PORT --log-opt mode=non-blocking --log-opt max-buffer-size=4m - --log-opt tag=web-backend-nginx web-backend-nginx ) diff --git a/container/web-backend/start b/container/web-backend/start index 5fb493c..941fb11 100755 --- a/container/web-backend/start +++ b/container/web-backend/start @@ -5,8 +5,6 @@ [[ $CCI_SRC ]] || CCI_SRC=/c/compass-ci [[ $LKP_SRC ]] || LKP_SRC=/c/lkp-tests [[ $UPSTREAM_REPOS_PATH ]] || UPSTREAM_REPOS_PATH=/c/upstream-repos -[[ $FLUENTD_HOST ]] || FLUENTD_HOST=localhost -[[ $FLUENTD_PORT ]] || FLUENTD_PORT=24224 . $CCI_SRC/container/defconfig.sh docker_rm web-backend @@ -27,11 +25,8 @@ cmd=( -v $LKP_SRC:/c/lkp-tests -v $UPSTREAM_REPOS_PATH:/c/upstream-repos - --log-driver=fluentd - --log-opt fluentd-address=$FLUENTD_HOST:$FLUENTD_PORT --log-opt mode=non-blocking --log-opt max-buffer-size=4m - --log-opt tag=web-backend debian:web-backend ) -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/monitor.rb:stop optimize
by Wu Zhende 28 Oct '20

28 Oct '20

1.add stop reason 2.close ws before stop EM [Why] Before stop EM, should close ws and give reason first. When the client disconnects from the ws, the reason is not printed. Like this: connect to ws://localhost:11310/filter ............. connection closed: Add stop reason, and it will be like: connect to ws://localhost:11310/filter .......... connection closed: timeout or connection closed: normal Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 9ce1d475..c20794e8 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -27,6 +27,7 @@ class Monitor load_default @result = [] @stop_query = {} + @reason = nil end def load_default @@ -87,16 +88,12 @@ class Monitor ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" end - def stop_em(web_socket) - web_socket.close - EM.stop - end - - def stop(data, web_socket) + def stop(data, web_socket, code = 1000, reason = 'normal') @stop_query.each do |key, value| return false unless data[key] == value end - stop_em(web_socket) + @reason = reason + web_socket.close(code, reason) end def run(timeout = nil) @@ -111,7 +108,8 @@ class Monitor if timeout && timeout != 0 EM.add_timer(timeout) do @exit_status_code = 1 - stop_em(ws) + @reason = 'timeout' + ws.close(1000, @reason) end end @@ -131,7 +129,9 @@ class Monitor end ws.on :close do |event| - puts "connection closed: #{event.reason}" + reason = event.reason || @reason + puts "connection closed: #{reason}" + EM.stop end end return @exit_status_code -- 2.23.0

1 0

[PATCH v2 compass-ci 5/5] container/send-mail: delete old useless files
by Luan Shengde 28 Oct '20

28 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/conf/sources.list | 9 ----- container/send-mail/mail-post.rb | 48 --------------------------- 2 files changed, 57 deletions(-) delete mode 100644 container/send-mail/conf/sources.list delete mode 100755 container/send-mail/mail-post.rb diff --git a/container/send-mail/conf/sources.list b/container/send-mail/conf/sources.list deleted file mode 100644 index 8179e01..0000000 --- a/container/send-mail/conf/sources.list +++ /dev/null @@ -1,9 +0,0 @@ -deb [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib -deb [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib -deb [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib -deb [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib - -deb-src [trusted=yes] http://mirrors.163.com/debian/ buster main non-free contrib -deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-updates main non-free contrib -deb-src [trusted=yes] http://mirrors.163.com/debian/ buster-backports main non-free contrib -deb-src [trusted=yes] http://mirrors.163.com/debian-security/ buster/updates main non-free contrib diff --git a/container/send-mail/mail-post.rb b/container/send-mail/mail-post.rb deleted file mode 100755 index 4eecf6e..0000000 --- a/container/send-mail/mail-post.rb +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env ruby -# SPDX-License-Identifier: MulanPSL-2.0+ -# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# frozen_string_literal: true - -require 'sinatra' -require 'json' -require 'yaml' -require 'open3' -require_relative 'send-mail.rb' - -set :bind, '0.0.0.0' -set :port, 11311 - -post '/send_mail_yaml' do - data = YAML.safe_load request.body.read - raise TypeError, data, 'request data type error' unless data.class.eql? Hash - - mail_info = { - 'references' => data['references'] || '', - 'from' => data['from'] || 'team(a)crystal.ci', - 'subject' => data['subject'], - 'to' => data['to'], - 'body' => data['body'] - } - check_send_mail(mail_info) -end - -post '/send_mail_text' do - data = Mail.read_from_string(request.body.read) - - mail_info = { - 'references' => data.references || '', - 'from' => data.from || 'team(a)crystal.ci', - 'subject' => data.subject, - 'to' => data.to, - 'body' => data.body.decoded - } - check_send_mail(mail_info) -end - -def check_send_mail(mail_info) - raise TypeError, data, 'empty subject.' if mail_info['subject'].empty? - raise TypeError, data, 'empty email address.' if mail_info['to'].empty? - raise TypeError, data, 'empty email content.' if mail_info['body'].empty? - - send_mail(mail_info) -end -- 2.23.0

1 0

[PATCH v2 compass-ci 4/5] container/send-mail: start
by Luan Shengde 28 Oct '20

28 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/start | 92 +++++++++++++++------------------------ 1 file changed, 34 insertions(+), 58 deletions(-) diff --git a/container/send-mail/start b/container/send-mail/start index 80be86a..95d797f 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -1,60 +1,36 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. - -. $CCI_SRC/container/defconfig.sh - -docker_rm send-mail - -cmd=( - docker run - --restart=always - --name=send-mail - -u nobody - -d - -p 11311:11311 - -v /etc/localtime:/etc/localtime:ro - send-mail -) - -"${cmd[@]}" - -cat<<EOF -port: 11311 ---- -data='{ -"from": "email_from_addr" # default: team(a)crystal.ci -"subject": "email subject", -"to": "email_to_addr", -"body": "email message" -}' - -or - -data=" -from: email_from_addr # default: team(a)crystal.ci -subject: email subject -to: email_to_addr -body: email message" - -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_yaml' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_yaml' -d "$data" - ---- -data=" -From: email_from_addr # default: team(a)crystal.ci -To: email_to_addr -Subject: email_subject - -mail_msg_line1 -mail_msg_line2 -... -" - -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_text' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_text' -d "$data" -EOF +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig' + +docker_rm 'send-mail' + +names = Set.new %w[ + LOCAL_SEND_MAIL_PORT + LOCAL_ROBOT_EMAIL_ADDRESS +] + +defaults = relevant_defaults(names) + +cmd = %W[ + docker run + --restart=always + --name=send-mail + -u nobody + -d + -e CCI_SRC=/c/compass-ci + -e SEND_MAIL_PORT=#{defaults['LOCAL_SEND_MAIL_PORT']} + -e ROBOT_EMAIL_ADDRESS=#{defaults['LOCAL_ROBOT_EMAIL_ADDRESS']} + -p #{defaults['LOCAL_SEND_MAIL_PORT']}:#{defaults['LOCAL_SEND_MAIL_PORT']} + -v #{ENV['CCI_SRC']}:/c/compass-ci + -v /etc/localtime:/etc/localtime:ro + -w /c/compass-ci/container/send-mail + send-mail +] + +cmd += ['ruby', 'send-mail.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v2 compass-ci 3/5] container/send-mail: Dockerfile
by Luan Shengde 28 Oct '20

28 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/Dockerfile | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index 36b145c..d151bda 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -2,18 +2,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. FROM debian -MAINTAINER luanshd0525(a)163.com +MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive -COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra && \ - ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ - echo 'Asia/Shanghai' > /etc/timezone - -EXPOSE 25 - -COPY mail-post.rb /usr/local/bin/ -COPY send-mail.rb /usr/local/bin/ - -CMD ["/usr/local/bin/mail-post.rb"] + apt-get install -y ruby-mail ruby-json ruby-sinatra -- 2.23.0

1 0

[PATCH v2 compass-ci 2/5] container/send-mail: mail-post.rb
by Luan Shengde 28 Oct '20

28 Oct '20

fix invoke function name error Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- lib/mail-post.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index 5832bfb..0b1f94a 100755 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -22,7 +22,7 @@ post '/send_mail_yaml' do 'body' => data['body'] } - check_send_mail(mail_info) + send_mail(mail_info) end post '/send_mail_text' do -- 2.23.0

1 0

[PATCH v2 compass-ci 1/5] container/send-mail: send-mail.rb
by Luan Shengde 28 Oct '20

28 Oct '20

setup smtp config for email server invoke mail-post to hand the send mail request Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/send-mail.rb | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index f459e78..436736c 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -4,25 +4,14 @@ # frozen_string_literal: true require 'mail' +require 'sinatra' +require "#{ENV['CCI_SRC']}/lib/mail-post" -mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp +mail_server = %x(/sbin/ip route |awk '/default/ {print $3}').chomp -# setup smtp config smtp = { address: mail_server, enable_starttls_auto: false } Mail.defaults { delivery_method :smtp, smtp } - -# send mail -def send_mail(mail_info) - mail = Mail.new do - references mail_info['references'] - from mail_info['from'] - subject mail_info['subject'] - to mail_info['to'] - body mail_info['body'] - end - mail.deliver! -end -- 2.23.0

1 0

[PATCH compass-ci] LICENSE: add document license
by Bai Jing 27 Oct '20

27 Oct '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- LICENSES/CCBY-4.0 | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 LICENSES/CCBY-4.0 diff --git a/LICENSES/CCBY-4.0 b/LICENSES/CCBY-4.0 new file mode 100644 index 0000000..112f1cd --- /dev/null +++ b/LICENSES/CCBY-4.0 @@ -0,0 +1,2 @@ +Copyright (c) 2020 Huawei Technologies Co., Ltd. +Your copying, use, modification, and distribution of this document is subject to the Creative Commons Attribution-ShareAlike 4.0 International Public License ("CC BY-SA 4.0"). For ease of understanding, you can visit https://creativecommons.org/licenses/by-sa/4.0/to learn about the outline of CC BY-SA 4.0(but not a substitute).The complete agreement of CC BY-SA 4.0 can be obtained from the following website: https://creativecommons.org/licenses/by-sa/4.0/legalcode. -- 2.23.0

2 1

[PATCH compass-ci] sparrow: 3-code/dev-env: add ENV TASKQUEUE_HOST
by Liu Yinsi 27 Oct '20

27 Oct '20

TASKQUEUE_HOST used by container service taskqueue, taskqueue port is default 3060. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 11c7fb0..44f7a93 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -17,6 +17,7 @@ mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port +TASKQUEUE_HOST: $server_ip OS_HTTP_HOST: $os_http_host INITRD_HTTP_HOST: $initrd_http_host INITRD_HTTP_PORT: $initrd_http_port -- 2.23.0

3 4

[PATCH v2 lkp-tests 1/2] jobs/block_size.yaml: add block_size.yaml
by Lu Kaiyi 27 Oct '20

27 Oct '20

[why] need test different block_size value Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/block_size.yaml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 jobs/block_size.yaml diff --git a/jobs/block_size.yaml b/jobs/block_size.yaml new file mode 100644 index 00000000..1e2239e5 --- /dev/null +++ b/jobs/block_size.yaml @@ -0,0 +1,21 @@ +suite: iozone +testcase: iozone +category: benchmark + +block_size: +- 64k +- 256k +- 1m +- 4m +- 16m + +disk: 2HDD + +fs: +- xfs +- ext4 +- btrfs +iosched: +- kyber + +iozone: -- 2.23.0

2 1