- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/4] container/send-mail:mail-post.rb
by Luan Shengde 16 Oct '20

16 Oct '20

add setup smpt for intranet/internet smtp enable send mail to both curystal.ci and WAN email Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/mail-post.rb | 36 +++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/container/send-mail/mail-post.rb b/container/send-mail/mail-post.rb index 4eecf6e..c7c31d5 100755 --- a/container/send-mail/mail-post.rb +++ b/container/send-mail/mail-post.rb @@ -7,21 +7,21 @@ require 'sinatra' require 'json' require 'yaml' require 'open3' -require_relative 'send-mail.rb' +require_relative 'send-mail' set :bind, '0.0.0.0' -set :port, 11311 +set :port, ENV['SEND_MAIL_PORT'] post '/send_mail_yaml' do data = YAML.safe_load request.body.read - raise TypeError, data, 'request data type error' unless data.class.eql? Hash + raise TypeError, data, 'request data type error' unless data.instance_of?(Hash) mail_info = { 'references' => data['references'] || '', 'from' => data['from'] || 'team(a)crystal.ci', - 'subject' => data['subject'], - 'to' => data['to'], - 'body' => data['body'] + 'subject' => data['subject'] || '', + 'to' => data['to'] || '', + 'body' => data['body'] || '' } check_send_mail(mail_info) end @@ -32,17 +32,29 @@ post '/send_mail_text' do mail_info = { 'references' => data.references || '', 'from' => data.from || 'team(a)crystal.ci', - 'subject' => data.subject, - 'to' => data.to, - 'body' => data.body.decoded + 'subject' => data.subject || '', + 'to' => data.to[0] || '', + 'body' => data.body.decoded || '' } check_send_mail(mail_info) end def check_send_mail(mail_info) - raise TypeError, data, 'empty subject.' if mail_info['subject'].empty? - raise TypeError, data, 'empty email address.' if mail_info['to'].empty? - raise TypeError, data, 'empty email content.' if mail_info['body'].empty? + raise 'empty subject.' if mail_info['subject'].empty? + raise 'empty email address.' if mail_info['to'].empty? + raise 'empty email content.' if mail_info['body'].empty? + + check_smtp(mail_info) send_mail(mail_info) end + +def check_smtp(mail_info) + mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp + if [nil, 'crystal.ci'].include? mail_info['to'].split('@')[1] + intranet_smtp(mail_server) + else + internet_smtp + mail_info['from'] = 'compass-ci(a)qq.com' + end +end -- 2.23.0

1 0

[PATCH lkp-tests] tests: add get_ipmi_ip function to host-info
by Zhang Yu 16 Oct '20

16 Oct '20

If ipmitool returns error or empty, the return of "$(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" is null, and only add the field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/host-info b/tests/host-info index a57855dc..6ee8df21 100755 --- a/tests/host-info +++ b/tests/host-info @@ -80,6 +80,11 @@ get_partitions() fi } +get_ipmi_ip() +{ + echo "ipmi_ip: $(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" +} + main() { get_node @@ -87,6 +92,7 @@ main() get_partitions get_mac get_model_name + get_ipmi_ip } main -- 2.23.0

4 3

[PATCH compass-ci] feat(sparrow): add kernel param: net.ipv6.bindv6only=1
by Yu Chuan 16 Oct '20

16 Oct '20

Turn on the bindv6only function to avoid the use of tcp6 port for services that do not actually need tcp6. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2c7de3754fad..31588f40b602 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -10,6 +10,7 @@ grep -q '^committer:' /etc/group || groupadd -g 1999 committer cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 +net.ipv6.bindv6only=1 vm.max_map_count=262144 EOF -- 2.23.0

1 0

[PATCH compass-ci 1/2] web-backend-nginx: Restrict interface access.
by Zhang Yuhang 16 Oct '20

16 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend-nginx/nginx.conf | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/container/web-backend-nginx/nginx.conf b/container/web-backend-nginx/nginx.conf index 5ab1a26..b461a2c 100644 --- a/container/web-backend-nginx/nginx.conf +++ b/container/web-backend-nginx/nginx.conf @@ -6,7 +6,7 @@ events { http { include /etc/nginx/mime.types; default_type application/octet-stream; - include /etc/nginx/conf.d/*.conf; + include /etc/nginx/conf.d/*.conf; server { listen 11320 ssl; server_name api.compass-ci.openeuler.org; @@ -15,8 +15,7 @@ http { ssl_session_timeout 5m; ssl_ciphers BCDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4; ssl_prefer_server_ciphers on; - - location / { + location ~ ^/(get_repos|get_jobs|compare|compare_candidates)$ { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forward-For $proxy_add_x_forwarded_for; -- 2.23.0

1 0

[PATCH v6 compass-ci 2/6] container/mail-robot: answerback-email.rb
by Luan Shengde 16 Oct '20

16 Oct '20

AssignUuid: read email content check email available check if base url in upstream-repos check if commit url valid check pub_key assign account generate uuid write account/uuid to es send uuid to user Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/answerback-email.rb | 181 +++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100755 container/mail-robot/answerback-email.rb diff --git a/container/mail-robot/answerback-email.rb b/container/mail-robot/answerback-email.rb new file mode 100755 index 0000000..2cd15ef --- /dev/null +++ b/container/mail-robot/answerback-email.rb @@ -0,0 +1,181 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '/compass-ci/lib/es_client.rb' + +# Assign uuid/account +class AssignUuid + def initialize(mail_content) + @mail_content = mail_content + end + + def build_message(sum_infos) + message = <<~EMAIL_MESSAGE + To: #{sum_infos['email']} + Message-ID: #{sum_infos['message_id']} + Subject: Account Ready + + Dear #{sum_infos['email'].split('@')[0]} + + Thank you for joining us. + You can use the following info to submit jobs: + + First: add the info to ~/.config/compass-ci/defaults/{USER}.yaml + + uuid: #{sum_infos['account_uuid']} + lab: z9 + + Second: download lkp-tests and dependencies + git clone https://gitee.com_/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source {HOME}/.bashrc && source {HOME}/.bash_profile + + Third: submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + how to write job yaml + submit jobs/netperf.yaml + + Repalce the fields {...} with your local environment variable. + + regards + compass-ci + EMAIL_MESSAGE + + return message + end + + def url_addr + url_line = @mail_content.part[0].body.decoded.split(/\n/).find { |line| line =~ %r{https?://} } + message = 'No commit url found' + + raise message unless url_line + + url = url_line.split[-1] + + return url + end + + def check_email_available(email) + url = url_addr + + base_url = url.gsub(%r{/commit/[^/].*}, '') + + upstream_dir = '/upstream-repos' + check_url_in_upstream_repos(upstream_dir, base_url) + check_url_commit(url, base_url, email) + + return url + end + + def check_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + message = 'The url is not in upstream-repo list' + + raise StandardError, message, base_url if match_out.empty? + end + + def check_url_commit(url, base_url, email) + hub_name = url.split('/')[2] + check_gitee_commit(url, base_url, email) if hub_name.eql? 'gitee.com' + check_non_gitee_commit(url, email) unless hub_name.eql? 'gitee.com' + end + + def check_non_gitee_commit(url, email) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index email + + message = 'No email matched from the commit_id feedback.' + raise StandardError, message, url if email_index.nil? + end + + def check_gitee_commit(url, base_url, email) + repo_info = { + 'repos_dir' => '/repos_dir', + 'repo_dir' => url.split('/')[-3], + 'repo_url' => [base_url, 'git'].join('.'), + 'repo_path' => File.join('/repos_dir', url.split('/')[-3]), + 'commit_id' => url.split('/')[-1] + } + check_gitee_repo_commit(url, email, repo_info) + end + + def check_gitee_repo_commit(url, email, repo_info) + Dir.chdir repo_info['repos_dir'] + %x(/usr/bin/git clone #{repo_info['repo_url']}) + email_index = %x(/usr/bin/git -C #{repo_info['repo_dir']} show #{repo_info['commit_id']}).index email + + FileUtils.rm_rf repo_info['repo_dir'] + + message = 'No email matched from commit_id feedback.' if email_index.nil? + raise StandardError, message, url if email_index.nil? + end + + def pub_key + pub_key = @mail_content.part[1].body.decoded.split(/\r|\n/).join if @mail_content.part[1].filename == 'id_rsa.pub' + + message = 'No pub_key found' + + raise message unless pub_key + + return pub_key + end + + def account_info(pub_key) + account_info_str = %x(curl -XGET "#{ENV['J_IP']}:#{ENV['J_PORT']}/assign_account" -d "pub_key: #{pub_key}") + account_info = JSON.parse account_info_str + + message = 'No available jumper account returned.' + raise StandardError, message, account_info if account_info['account'].empty? + + return account_info + end + + def build_sum_infos + email = @mail_content.from[0] + account_uuid = %x(uuidgen).chomp + message_id = @mail_content.message_id + sum_infos = { + 'email' => email, + 'message_id' => message_id, + 'url' => check_email_available(email), + 'acct_infos' => account_info(pub_key), + 'account_uuid' => account_uuid, + 'pub_key' => pub_key + } + store_account_info(sum_infos) + sum_infos + end + + def send_uuid(sum_infos) + message = build_message(sum_infos) + + send_mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp + %x(curl -XPOST "#{send_mail_server}:#{ENV['S_M_PORT']}/send_mail_text" -d "#{message}") + end + + def send_account + sum_infos = build_sum_infos + + send_uuid(sum_infos) + end + + def store_account_info(sum_infos) + account_uuid_info = { + 'email' => sum_infos['email'], + 'name' => sum_infos['email'].split('@')[0], + 'uuid' => sum_infos['account_uuid'], + 'user' => sum_infos['acct_infos']['account'], + 'my_commit_url' => sum_infos['url'] + } + + es = ESClient.new(index: 'accounts') + es.put_source_by_id(sum_infos['email'], account_uuid_info) + end +end -- 2.23.0

3 11

[PATCH compass-ci 2/3] git_mirror: put repo info to es
by Li Yuanchao 16 Oct '20

16 Oct '20

move a method to pass rubocop limit: Class xxx has too many lines[101/100] Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5f7fa6b..fef4aca 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -9,6 +9,8 @@ require 'json' # gem install PriorityQueue require 'priority_queue' require 'English' +require 'elasticsearch' +require_relative 'constants.rb' # worker threads class GitMirror @@ -84,6 +86,7 @@ class MirrorMain @git_info = {} @defaults = {} @git_queue = Queue.new + @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info connection = Bunny.new('amqp://172.17.0.1:5672') connection.start @@ -108,16 +111,6 @@ class MirrorMain @defaults[project] = YAML.safe_load(File.open(defaults_file)) end - def load_repo_file(repodir, project, fork_name) - git_repo = "#{project}/#{fork_name}" - @git_info[git_repo] = YAML.safe_load(File.open(repodir)) - @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] - fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 - end - def traverse_repodir(repodir) if File.directory? repodir load_defaults(repodir) @@ -194,6 +187,17 @@ end # main thread class MirrorMain + def load_repo_file(repodir, project, fork_name) + git_repo = "#{project}/#{fork_name}" + @git_info[git_repo] = YAML.safe_load(File.open(repodir)) + @git_info[git_repo]['git_repo'] = git_repo + @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] + es_repo_update(git_repo, @git_info[git_repo]) + fork_stat_init(git_repo) + @priority_queue.push git_repo, @priority + @priority += 1 + end + def compare_refs(cur_refs, old_refs) new_refs = { heads: {} } cur_refs[:heads].each do |ref, commit_id| @@ -246,4 +250,12 @@ class MirrorMain load_repo_file(repo_dir, File.dirname(file), File.basename(file)) if File.file?(repo_dir) end end + + def es_repo_update(git_repo, repo_info) + body = { + "doc": repo_info, + "doc_as_upsert": true + } + @es_client.update(index: 'repo', type: '_doc', id: git_repo, body: body) + end end -- 2.23.0

2 1

[PATCH lkp-tests] tests: add get_ipmi_ip function to host-info
by Zhang Yu 16 Oct '20

16 Oct '20

If ipmitool returns error or empty, the return of "$(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" is null, and then only add the field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tests/host-info b/tests/host-info index a57855dc..cd77c1eb 100755 --- a/tests/host-info +++ b/tests/host-info @@ -69,8 +69,7 @@ get_partitions() # sdf 1 ssd=$(lsblk -d -o name,ROTA |grep 0$ | awk '{print $1}') hdd=$(lsblk -d -o name,ROTA |grep 1$ | awk '{print $1}') - if [ -n "$hdd" ]; then - echo "hdd_partitions:" + if [ -n "$hdd" ]; then echo "hdd_partitions:" get_part $hdd fi @@ -80,6 +79,11 @@ get_partitions() fi } +get_ipmi_ip() +{ + echo "ipmi_ip: $(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" +} + main() { get_node @@ -87,6 +91,7 @@ main() get_partitions get_mac get_model_name + get_ipmi_ip } main -- 2.23.0

1 1

[PATCH compass-ci] fix docker-pull can't be used
by Xiao Shenwei 16 Oct '20

16 Oct '20

when run multi-docker job, the PATH exclude $CCI_SRC, so docker-pull can't be used directly, use full path to replace Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index b454f68..c241d3c 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -69,7 +69,7 @@ end def run(hostname, load_path, hash) docker_image = hash['docker_image'] - system "docker-pull #{docker_image}" + system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, ENV['CCI_SRC'] + '/providers/docker/run.sh' -- 2.23.0

1 0

[PATCH compass-ci] sched: throw an exception when deps/pkg does not exist
by Cao Xueliang 16 Oct '20

16 Oct '20

When submitting a job, an exception is thrown when deps/pkg does not exist on the server. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..ef515b8 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -389,11 +389,21 @@ class Job program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") && File.exists?(deps_dest_file) - initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?(deps_dest_file) + initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) + else + raise "#{deps_dest_file} does not exist on the server." + end end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") && File.exists?(pkg_dest_file) - initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") + if File.exists?(pkg_dest_file) + initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) + else + raise "#{pkg_dest_file} does not exist on the server." + end end end end -- 2.23.0

1 1

[PATCH lkp-tests] lib/job.rb: load host config with atomic job
by Wei Jihui 15 Oct '20

15 Oct '20

according to commit c725e0f022b707ba385f8f3657e79, job = deepcopy(@job) without load_hosts_config it will keep to @job for "@job = deepcopy(job)" under "@jobs.each do|hash|" so this @job is without load_hosts_config. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..6359254e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -696,9 +696,11 @@ class Job each_job_init each_job(&block) @jobs.each do |hash| + @load_hosts_done = false @job = deepcopy(job) @job2 = hash load_defaults + load_hosts_config each_job_init each_job(&block) end -- 2.23.0

2 1