- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/openresty-proxy-cache: add nginx.conf
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/nginx.conf | 85 ++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 container/openresty-proxy-cache/nginx.conf diff --git a/container/openresty-proxy-cache/nginx.conf b/container/openresty-proxy-cache/nginx.conf new file mode 100755 index 0000000..79aa10e --- /dev/null +++ b/container/openresty-proxy-cache/nginx.conf @@ -0,0 +1,85 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user nobody; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + proxy_cache_path /nginx_cache levels=1:2 keys_zone=my_cache:10m max_size=10g + inactive=6000m use_temp_path=off; + + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for" ' + '"$upstream_cache_status" $body_bytes_sent'; + access_log /tmp/access.log main; + error_log /tmp/error.log; + + resolver 8.8.8.8; + client_max_body_size 700m; + server_tokens off; + + proxy_cache my_cache; + proxy_ignore_headers Set-Cookie Cache-Control Expires; + proxy_cache_valid 200 1440m; + proxy_cache_valid 301 302 5m; + proxy_set_header X-Real-Ip $remote_addr; + proxy_set_header X-Forwarded-For $remote_addr; + add_header X-Cache $upstream_cache_status; + add_header X-Via $server_addr; + + server { + listen 443 ssl; + ssl_certificate /cert/ca.crt; + ssl_certificate_key /cert/ca.key; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + proxy_ssl_server_name on; + set_by_lua_block $cur_ups { + uri = ngx.var.request_uri + host = ngx.var.http_host + + path_file = "/srv/initrd/tar/"..uri + local f = io.open(path_file) + if f then + return "/127.0.0.1:443/tar/"..uri + end + return uri + } + proxy_pass $scheme:/$cur_ups; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } + server { + listen 80; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + + content_by_lua_file lua_script/forward.lua; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } +} -- 2.23.0

1 0

[PATCH compass-ci] container/openresty-proxy-cache: add start script
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/start | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 container/openresty-proxy-cache/start diff --git a/container/openresty-proxy-cache/start b/container/openresty-proxy-cache/start new file mode 100755 index 0000000..b946426 --- /dev/null +++ b/container/openresty-proxy-cache/start @@ -0,0 +1,23 @@ +#!/bin/bash +# Refer to: https://hub.docker.com/r/openresty/openresty +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# SPDX-License-Identifier: MulanPSL-2.0+ + +. $CCI_SRC/container/defconfig.sh + +docker_rm openresty-proxy-cache + +cmd=( + docker run -d + --name openresty-proxy-cache + --restart=always + -p 11333:80 + -p 11334:443 + -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v $PWD/lua_script/forward.lua:/usr/local/openresty/nginx/lua_script/forward.lua + -v /srv/initrd/tar:/srv/initrd/tar:ro + -v /etc/localtime:/etc/localtime:ro + openresty:proxy_cache +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH v5 compass-ci 7/7] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 08 Jan '21

08 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v5 compass-ci 6/7] lib/mail-post: add email checks
by Luan Shengde 08 Jan '21

08 Jan '21

add check email count only email addresses that email count within the limited value can be sent emails add check email mapping if has set email mapping for the entered address. the address will be converted to the mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..803a134 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +97,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v5 compass-ci 5/7] send-internet-mail: email rate limit
by Luan Shengde 08 Jan '21

08 Jan '21

add email_in_limit to store email address that send count less than limit value add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..c7e3954 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit. +# email_in_limit is used to store email address email count +# less than the limit value. +# email_out_limit is used to store email address email count +# beyond the limit. +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1. +# case the email in email_in_limit but email count less than the +# limit value, its value will +1. +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit. +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar. +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 4/7] send-internet-mail: add email mapping
by Luan Shengde 08 Jan '21

08 Jan '21

example: name => email_addr logo => email_addr email_addr_r => email_addr_d an email address can be set multi mappings, you can set a name or email to a specified email address when you write either a name or an email address in the send address bar, the email address will be converted to the specified address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..4c068b0 --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,49 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# email address translation according to the email_mapping queue +# email_mapping is used to set email mapping, example: +# email_mapping: +# name => email_address_d +# email_address_r1 => email_address_d +# email_address_r2 => email_address_d +# ... +# can set multi keys for an email address, the key can be a name, +# an email, or some logo else +# case add the key is added to email address bar, the key will be +# transferred to the destination email address +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 08 Jan '21

08 Jan '21

daily timing work: reset email rate limit queues Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..62f27ea --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v5 compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 08 Jan '21

08 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH v5 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 08 Jan '21

08 Jan '21

[why] redis is used to read/store email counts from/to redis queue rufus-scheduler is used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 0

[PATCH v4 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 08 Jan '21

08 Jan '21

[why] redis is used to read/store email counts from/to redis queue rufus-scheduler is used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

2 2

review star
by Wu Fengguang 08 Jan '21

08 Jan '21

Jihui is our review star, thanks Jihui! jimmy@crystal ~/todo/bin% ./review-star 7 Review email records in the last 7 days total review paragraph point people 53 31 68 26 Wu Fengguang 87 15 18 10 Wei Jihui 37 10 20 9 Lu Kaiyi 32 4 11 8 Yu Chuan 58 15 33 7 Luan Shengde 24 5 8 6 Xiao Shenwei 20 11 12 6 Xu Xijian 10 5 11 5 Wu Zhende 32 3 3 4 Hu Xuejiao 19 8 5 4 Liu Shaofei 22 9 9 4 Ren Wen 28 2 2 3 Cao Xueliang 29 6 5 2 Cui Yili 10 8 8 2 Wang Yong 17 2 3 2 Li Ping 3 3 5 2 Zhang Dewan 12 4 7 1 Lin Jiaxin 23 4 6 1 Wang Chenglong 10 2 2 1 Li Yuanchao 13 1 1 1 Lu Weitao 5 1 1 1 Zhang Yale 33 2 2 1 Liu Yinsi 6 1 2 0 Bai Jing

1 0

git count
by Wu Fengguang 08 Jan '21

08 Jan '21

Yu Chuan is our code star again, thanks Yu Chuan! 10 Yu Chuan 6 Wei Jihui 6 Wu Fengguang 6 Li Yuanchao 5 Liu Shaofei 4 Xu Xijian 4 Hu Xuejiao 4 Lu Weitao 4 Wu Zhende 4 Xiao Shenwei 4 Lu Kaiyi 3 Wang Chenglong 3 Li Ping 2 Bai Jing 2 Cao Xueliang 2 Cui Yili 2 Lin Jiaxin 1 Zhang Yale 1 Luan Shengde 0 Tong Qunfeng 0 Du Kaitian 0 Liu Yinsi 0 Sun Yukui 0 Zhang Yu 0 Wang Yong 0 Ren Wen 0 Zhang Dewan 0 Shi Zhichao

1 0

help star
by Wu Fengguang 08 Jan '21

08 Jan '21

Jihui is our help star, thanks Jihui! 15.2 Wu Fengguang 13.0 Wei Jihui 12.1 Yu Chuan 11.6 Xiao Shenwei 9.6 Lu Weitao 8.3 Ren Wen 7.5 Cao Xueliang 7.3 Wu Zhende 6.8 Zhang Yale 6.6 Wang Yong 6.1 Zhang Dewan 6.0 Liu Shaofei 4.8 Liu Yinsi 4.3 Xu Xijian 3.6 Wang Chenglong 3.3 Du Kaitian 2.8 Li Ping 2.7 Lin Jiaxin 2.3 Bai Jing 2.2 Cui Yili 2.0 Luan Shengde 1.3 Hu Xuejiao 0.7 Lu Kaiyi 0.0 Tong Qunfeng 0.0 Li Yuanchao 0.0 Wang Shuai 0.0 Cui Xucui

1 0

[PATCH lkp-tests] adaptation/openeuler: add dependency of pgbench
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/openeuler | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/adaptation/openeuler b/distro/adaptation/openeuler index 09120489..1938a8aa 100644 --- a/distro/adaptation/openeuler +++ b/distro/adaptation/openeuler @@ -295,3 +295,4 @@ ruby-full: ruby ruby-devel rubygem-rdoc linux-base: openssh-server: openssh openssh-client: openssh +postgresql-server: postgresql -- 2.23.0

2 1

[PATCH lkp-tests] tests/ebizzy: modify the value of iterations
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/ebizzy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/ebizzy b/tests/ebizzy index dcc301871..394bc09ba 100755 --- a/tests/ebizzy +++ b/tests/ebizzy @@ -19,7 +19,7 @@ duration=${duration%%s} [ -n "$mmap" ] && opt_mmap=" -m " [ -n "$linear" ] && opt_linear=" -z " - +[[ "$iterations" =~ ^[0-9]{3}x$ ]] && iterations=100 for i in $(seq 1 $iterations) do echo Iteration: $i -- 2.23.0

3 2

[PATCH v2 compass-ci 2/2] lib: add email_mapping_client
by Luan Shengde 08 Jan '21

08 Jan '21

used to - add email mapping - delete email mapping - re-map email mapping - search email mapping - delete email mapping queue(delete all email mappings) Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/email_mapping_client.rb | 47 +++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 lib/email_mapping_client.rb diff --git a/lib/email_mapping_client.rb b/lib/email_mapping_client.rb new file mode 100755 index 0000000..002bf23 --- /dev/null +++ b/lib/email_mapping_client.rb @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to add/delete/re-map/search email mappings +class EmailMapping + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(r_name, d_email) + @r_name = r_name + @d_email = d_email + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_email_mapping + return false if @redis.hexists 'email_mapping', @r_name + + @redis.hset 'email_mapping', @r_name, @d_email + return true + end + + def delete_email_mapping + @redis.hdel 'email_mapping', @r_name + end + + def delete_mapping_queue + @redis.del 'email_mapping' + end + + def re_map_email + @redis.hset 'email_mapping', @r_name, @d_email + end + + def search_map + d_email = @redis.hget 'email_mapping', @r_name + + return d_email if d_email + + message = "#{@r_name} has not mapped an email yet." + return message + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin: add email-mapping
by Luan Shengde 08 Jan '21

08 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue options: -n, --name name|email|logo appoint a name|email|logo to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings --delete-queue delete mapping queue -s, --search search email mapping -h, --help show this message the name can be either a name, an email or something like logo the file content consists of names and emails, like: name1: email_address1 name2: email_address2 you can map multi name to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 99 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100755 sbin/email-mapping.rb diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..cfc70ec --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,99 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +require "#{ENV['CCI_SRC']}/lib/email_mapping_client" + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d]\n" + opts.banner += " email_mapping --delete-queue\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|logo', '--name name|email|logo', 'appoint a name|email|logo to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || nil + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.empty? + ARGV << '-h' +elsif (['-n', '-f', '--delete-queue'] - ARGV).eql? ['-n', '-f', '--delete-queue'] + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = EmailMapping.new(r_name, d_email) + if map_conf['re_mapping'] + email_mapping.re_map_email + elsif map_conf['del_map'] + email_mapping.delete_email_mapping + elsif map_conf['search_map'] + puts email_mapping.search_map + elsif map_conf['delete_queue'] + email_mapping.delete_mapping_queue + else + return if email_mapping.add_email_mapping + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 0

[PATCH compass-ci] uniform constant naming
by Lu Weitao 08 Jan '21

08 Jan '21

[Why] most file use "SEND_MAIL_HOST/PORT" [How] sed -i "s/[^_]MAIL_HOST/SEND_MAIL_HOST/g" `grep MAIL_HOST -rl compass-ci/*` sed -i "s/[^_]MAIL_PORT/SEND_MAIL_PORT/g" `grep MAIL_PORT -rl compass-ci/*` Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/delimiter/start | 4 ++-- container/extract-stats/start | 4 ++-- lib/constants.rb | 4 ++-- lib/mail_client.rb | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/container/delimiter/start b/container/delimiter/start index 5d7037d..cb44d84 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -9,8 +9,8 @@ require_relative '../defconfig.rb' names = Set.new %w[ ES_HOST ES_PORT - MAIL_HOST - MAIL_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT MONITOR_HOST MONITOR_PORT TASKQUEUE_HOST diff --git a/container/extract-stats/start b/container/extract-stats/start index 74cf917..6510564 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -12,8 +12,8 @@ names = Set.new %w[ REDIS_PORT ES_HOST ES_PORT - MAIL_HOST - MAIL_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT LAB ] diff --git a/lib/constants.rb b/lib/constants.rb index c3c7543..a274adc 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -8,8 +8,8 @@ config = cci_defaults ES_HOST = config['ES_HOST'] || '172.17.0.1' ES_PORT = config['ES_PORT'] || 9200 -MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' -MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' +SEND_MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 diff --git a/lib/mail_client.rb b/lib/mail_client.rb index f63e65d..6f5a099 100644 --- a/lib/mail_client.rb +++ b/lib/mail_client.rb @@ -8,8 +8,8 @@ require_relative 'constants.rb' # mail client class class MailClient - HOST = (ENV.key?('MAIL_HOST') ? ENV['MAIL_HOST'] : MAIL_HOST) - PORT = (ENV.key?('MAIL_PORT') ? ENV['MAIL_PORT'] : MAIL_PORT).to_i + HOST = (ENV.key?('SEND_MAIL_HOST') ? ENV['SEND_MAIL_HOST'] : SEND_MAIL_HOST) + PORT = (ENV.key?('SEND_MAIL_PORT') ? ENV['SEND_MAIL_PORT'] : SEND_MAIL_PORT).to_i def initialize(host = HOST, port = PORT) @host = host @port = port -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/auto_submit.yaml: add/update linux repos
by Lin Jiaxin 08 Jan '21

08 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit.yaml | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 69b73ff..22172eb 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -4,7 +4,19 @@ c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml l/linux/linux: -- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/linux-next: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/openeuler-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/debian-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/archlinux-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: - testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

3 2

[PATCH lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 08 Jan '21

08 Jan '21

[error] LKP: HOSTNAME vm-8p64g.2288hv5-2s44p-384g--b5-3554212, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [ 75.489958] Kernel tests: Boot OK! ... [ 82.149060] sysrq: Resetting [why] check network ok or not by checking network adapter is not accurate, use ping ip more accurate. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index d4b41f3bd..b7795b40a 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,13 +87,7 @@ net_devices_link() network_ok() { - local i - for i in /sys/class/net/*/ - do - [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 - done + ping -c 3 $LKP_SERVER | grep -q ttl && return 0 is_clearlinux && { net_devices_link up -- 2.23.0

2 2

[PATCH compass-ci 2/3] scheduler: remove prefix of 'kernel_uri' and 'initrds_uri'
by Ren Wen 08 Jan '21

08 Jan '21

Prefix 'kernel ' and 'initrd ' are just used by 'boot.ipxe'. Now need these uris without prefix for 'boot.libvirt', so remove them. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e7b9e48..9f03a04 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -432,7 +432,7 @@ class Job end private def set_kernel_uri - self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + + self["kernel_uri"] = "#{OS_HTTP_PREFIX}" + JobHelper.service_path("#{linux_vmlinuz_path}") end @@ -496,7 +496,7 @@ class Job initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| - initrds_uri_values << JSON::Any.new("initrd #{initrd}") + initrds_uri_values << JSON::Any.new("#{initrd}") end @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) -- 2.23.0

1 0

[PATCH compass-ci 1/3] scheduler: add a new job field 'kernel_base_params'
by Ren Wen 08 Jan '21

08 Jan '21

Kernel params like 'initrd=xxx' is just for 'boot.ipxe'. Now need kernel params without 'initrd=xxx' for 'boot.libvirt', so add a job field 'kernel_base_params' for this. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 8 +------- src/scheduler/kernel_params.cr | 15 ++++++++++----- 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..e7b9e48 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -489,13 +489,7 @@ class Job end private def initrds_basename - basenames = "" - - get_initrds().each do |initrd| - basenames += "initrd=#{File.basename(initrd)} " - end - - return basenames + return get_initrds().map { |initrd| "initrd=#{File.basename(initrd)}" } end private def set_initrds_uri diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 76606aa..2319a9a 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -2,7 +2,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Job - private def kernel_common_params return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" end @@ -15,10 +14,10 @@ class Job os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrds_basename()}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrds_basename()}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrds_basename()}", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "container" => "", } @@ -32,7 +31,13 @@ class Job private def set_kernel_params kernel_params_values = "#{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" kernel_params_values = kernel_params_values.split(" ").map(&.strip()).reject!(&.empty?) + # *kernel_base_params* is for boot.libvirt + @hash["kernel_base_params"] = JSON.parse(kernel_params_values.to_json) + + _initrds_basenames = [] of String + _initrds_basenames = initrds_basename() unless os_mount == "container" + kernel_params_values += _initrds_basenames + # *kernel_params* is for boot.ipxe @hash["kernel_params"] = JSON.parse(kernel_params_values.to_json) end - end -- 2.23.0

1 0

[PATCH v6 compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 08 Jan '21

08 Jan '21

before: support kvm for aarch64 after: support kvm for aarch64, x86_64, riscv64 test: /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-riscv64/z9.289591 /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-x86_64/z9.289594 /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-aarch64/z9.289571 [how] put public options like $kernel, $initrd, $smp together, set individual options like $machine, $bios, $nic according to different qemu command, then merge all options to run qemu. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 116 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 101 insertions(+), 15 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..363cf4c 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,12 +18,6 @@ check_logfile() } } -qemu_command() -{ - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm -} - write_logfile() { ipxe_script=ipxe_script @@ -66,12 +60,57 @@ check_option_value() [ -n "$initrds" ] || exit } -get_initrd() +set_initrd() { initrd=initrd cat $initrds > $initrd } +set_bios() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd +} + +set_helper() +{ + helper=/usr/libexec/qemu-bridge-helper + [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper +} + +set_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + +set_device() +{ + device="virtio-net-device,netdev=net0,mac=${mac}" +} + +set_netdev() +{ + netdev="bridge,br=br0,id=net0,helper=${helper}" +} + +set_qemu() +{ + qemu=( + qemu-system-aarch64 + qemu-kvm + qemu-system-x86_64 + qemu-system-riscv64 + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} + print_message() { echo $SCHED_PORT @@ -83,36 +122,83 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} + +individual_option() +{ + case "$qemu" in + qemu-system-aarch64) + arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + [ "$(arch)" == "x86_64" ] && arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-x86_64) + arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-riscv64) + arch_option=( + -machine virt + -device $device + -netdev $netdev + ) + ;; + esac +} - "${kvm[@]}" --append "${append}" +run_qemu() +{ + "${kvm[@]}" "${arch_option[@]}" --append "${append}" } check_logfile write_logfile -qemu_command parse_ipxe_script check_option_value -get_initrd + +set_initrd +set_bios +set_helper +set_nic +set_device +set_netdev +set_qemu print_message +public_option +individual_option + run_qemu -- 2.23.0

1 0

[PATCH v3 compass-ci] 3-code/git: modify the git download script
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 42 ++++++++++++++++-------------------------- 1 file changed, 16 insertions(+), 26 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 48741ec..c9b5df4 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -5,31 +5,21 @@ # fix files permissions cause container service start failed umask 002 -# download git trees - -cd /c || exit - -# git clone https://gitee.com/openeuler/compass-ci.git -# modify and manual run for now: -[ -d "compass-ci/.git" ] || { - git clone https://gitee.com/wu_fengguang/compass-ci.git - - cd /c/compass-ci || exit - [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" - - ln -s compass-ci /c/cci +download_repo() +{ + local repo="$1" + local commit_id="$2" + + [ -d "/c/$repo/.git" ] || { + git clone https://gitee.com/wu_fengguang/"$repo".git /c/"$repo" + + [ "$repo" == "compass-ci" ] && ln -s /c/"$repo" /c/cci + } + + cd /c/"$repo" || exit + [ -n "$commit_id" ] && git reset --hard "$commit_id" } -[ -d "lkp-tests/.git" ] || { - git clone https://gitee.com/wu_fengguang/lkp-tests.git - - cd /c/lkp-tests || exit - [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" -} - -[ -d "upstream-repos/.git" ] || { - git clone https://gitee.com/wu_fengguang/upstream-repos.git - - cd /c/upstream-repos || exit - [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" -} +download_repo "compass-ci" "$compass_commit_id" +download_repo "lkp-tests" "$lkp_commit_id" +download_repo "upstream-repos" "$upstream_commit_id" -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: modify the git download script
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 42 ++++++++++++++++-------------------------- 1 file changed, 16 insertions(+), 26 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 48741ec..6174916 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -5,31 +5,21 @@ # fix files permissions cause container service start failed umask 002 -# download git trees - -cd /c || exit - -# git clone https://gitee.com/openeuler/compass-ci.git -# modify and manual run for now: -[ -d "compass-ci/.git" ] || { - git clone https://gitee.com/wu_fengguang/compass-ci.git - - cd /c/compass-ci || exit - [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" - - ln -s compass-ci /c/cci +download_repo() +{ + local repo="$1" + local commit_id="$2" + + [ -d "/c/$repo/.git" ] || { + git clone https://gitee.com/wu_fengguang/"$repo".git /c/"$repo" + + [ "$repo" == "compass-ci" ] && ln -s /c/"$repo" /c/cci + } + + cd /c/"$repo" || exit + [ -n "$commit_id" ] && git reset --hard "$commit_id" } -[ -d "lkp-tests/.git" ] || { - git clone https://gitee.com/wu_fengguang/lkp-tests.git - - cd /c/lkp-tests || exit - [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" -} - -[ -d "upstream-repos/.git" ] || { - git clone https://gitee.com/wu_fengguang/upstream-repos.git - - cd /c/upstream-repos || exit - [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" -} +download_repo "compass-ci" "compass_commit_id" +download_repo "lkp-tests" "$lkp_commit_id" +download_repo "upstream-repos" "$upstream_commit_id" -- 2.23.0

2 1