October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 4/4] container/send-mail: update start
by Luan Shengde 19 Oct '20

19 Oct '20

add setup default variables update heredoc Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/start | 89 +++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 27 deletions(-) diff --git a/container/send-mail/start b/container/send-mail/start index 80be86a..068fc1e 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -1,29 +1,68 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -. $CCI_SRC/container/defconfig.sh +require 'set' +require_relative '../defconfig' -docker_rm send-mail +docker_rm 'send-mail' -cmd=( - docker run - --restart=always - --name=send-mail - -u nobody - -d - -p 11311:11311 - -v /etc/localtime:/etc/localtime:ro - send-mail -) +names = Set.new %w[ + EMAIL_PSWD + SEND_MAIL_PORT + SMTP_SERVER_ADDR + SMTP_SERVER_PORT + EMAIL_DOMAIN + EMAIL_ADDR + DEFAULT_EMAIL +] -"${cmd[@]}" +defaults = relevant_defaults(names) -cat<<EOF +EMAIL_PSWD = defaults['EMAIL_PSWD'] +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] +SMTP_SERVER_ADDR = defaults['SMTP_SERVER_ADDR'] +SMTP_SERVER_PORT = defaults['SMTP_SERVER_PORT'] +EMAIL_DOMAIN = defaults['EMAIL_DOMAIN'] +EMAIL_ADDR = defaults['EMAIL_ADDR'] +DEFAULT_EMAIL = defaults['DEFAULT_EMAIL'] + +cmd = %W[ + docker run + --restart=always + --name=send-mail + -u nobody + -d + -e EMAIL_PSWD=#{EMAIL_PSWD} + -e SEND_MAIL_PORT=#{SEND_MAIL_PORT} + -e SMTP_SERVER_ADDR=#{SMTP_SERVER_ADDR} + -e SMTP_SERVER_PORT=#{SMTP_SERVER_PORT} + -e EMAIL_DOMAIN=#{EMAIL_DOMAIN} + -e EMAIL_ADDR=#{EMAIL_ADDR} + -e DEFAULT_EMAIL=#{DEFAULT_EMAIL} + -p #{SEND_MAIL_PORT}:#{SEND_MAIL_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-driver json-file + send-mail +] + +cmd += ['bash', '-c', '/usr/local/bin/mail-post.rb'] + +system(*cmd) + +puts <<HEREDOC port: 11311 + +default from email: + when sending crystal.ci email. + team(a)crystal.ci if no from email appiontted + when sending to WAN email + compass-ci(a)qq.com in all cases + --- data='{ -"from": "email_from_addr" # default: team(a)crystal.ci +"from": "email_from_addr" "subject": "email subject", "to": "email_to_addr", "body": "email message" @@ -32,19 +71,17 @@ data='{ or data=" -from: email_from_addr # default: team(a)crystal.ci +from: email_from_addr subject: email subject to: email_to_addr body: email message" -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_yaml' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_yaml' -d "$data" +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_yaml' -d "$data" --- data=" -From: email_from_addr # default: team(a)crystal.ci +From: email_from_addr To: email_to_addr Subject: email_subject @@ -53,8 +90,6 @@ mail_msg_line2 ... " -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_text' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_text' -d "$data" -EOF +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_text' -d "$data" +HEREDOC -- 2.23.0

2 1

[PATCH v2 compass-ci 2/4] container/send-mail: send-mail.rb
by Luan Shengde 19 Oct '20

19 Oct '20

add setup smtp for qq mail server enable send mail via smtp.qq.com Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/send-mail.rb | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index f459e78..844cad5 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -5,15 +5,29 @@ require 'mail' -mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp +# mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp # setup smtp config -smtp = { - address: mail_server, - enable_starttls_auto: false -} +def intranet_smtp(mail_server) + smtp = { + address: mail_server, + enable_starttls_auto: false + } + Mail.defaults { delivery_method :smtp, smtp } +end -Mail.defaults { delivery_method :smtp, smtp } +def internet_smtp + smtp = { + address: (ENV['SMTP_SERVER_ADDR']).to_s, + port: (ENV['SMPT_SERVER_PORT']), + domain: (ENV['EMAIL_DOMAIN']).to_s, + user_name: (ENV['EMAIL_ADDR']).to_s, + password: (ENV['EMAIL_PSWD']).to_s, + openssl_verify_mode: 'none', + enable_starttls_auto: true + } + Mail.defaults { delivery_method :smtp, smtp } +end # send mail def send_mail(mail_info) -- 2.23.0

2 1

[PATCH compass-ci 1/2] user-client/maintain: test os rootfs in z9 tbox
by Liu Yinsi 19 Oct '20

19 Oct '20

1. no needed my-qemu.sh after submit job in z9 tbox, remove. 2. monitor job_state boot/extract_finished to get job runtime. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 124 +++++++++--------- 1 file changed, 65 insertions(+), 59 deletions(-) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index 21e1428..2c3fc9f 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -1,62 +1,68 @@ -#!/bin/bash -e +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ +# # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -# os test rounds -test_times=1 - -test_logs=walk-test.report -test_yaml=iperf-walk-os.yaml -test_os=( - 'openeuler aarch64 20' - 'openeuler aarch64 20.03' - 'centos aarch64 7.6' - 'centos aarch64 7.8' - 'centos aarch64 8.1.1911' +require "#{ENV['LKP_SRC']}/lib/monitor" + +TEST_OS = [ + 'openeuler aarch64 20.03', + 'centos aarch64 7.6.1810', + 'centos aarch64 7.8.2003', + 'centos aarch64 8.1.1911', 'debian aarch64 sid' -) - -trap 'echo User Cancelled! && exit' SIGINT - -sec2date() { - date -d@$1 +"%Y%m%d_%H:%M:%S" -} - -logging() { - if [ 'new' == "$1" ] ; then - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" ${@:2} > $test_logs - else - printf "%-5s %-8s %-25s %-20s %-20s %-8s %-10s\n" $@ >> $test_logs - fi -} - -create_test_yaml() { - local hwarch os os_arch os_ver cur_os - hwarch=$(arch) - cur_os="$@" - os=$(echo $cur_os |cut -d' ' -f1) - os_arch=$(echo $cur_os |cut -d' ' -f2) - os_ver=$(echo $cur_os |cut -d' ' -f3) - sed "s/USER/$USER/" $test_yaml > test.yaml - sed -i "s/OS_ARCH/$os_arch/" test.yaml - sed -i "s/OS_VER/$os_ver/" test.yaml - sed -i "s/OS/$os/" test.yaml - echo "$hwarch $os/$os_arch/$os_ver" -} - -job_test() { - local job_id bef aft - job_id=$(submit test.yaml |awk '{print $NF}') - bef=$(date +%s) - ./my-qemu.sh - aft=$(date +%s) - logging "$1" "$2" "$3" "$(sec2date $bef)" "$(sec2date $aft)" "$((aft - bef))" "$job_id" -} - -logging new "Index" "hwArch" "OS" "Begin" "End" "Cost/sec" "Job_ID" -for round in $(seq $test_times) -do - for idx in "${!test_os[@]}" - do - job_test $round $(create_test_yaml "${test_os[$idx]}") - done -done +] + +MOUNT_TYPE = ['cifs', 'initramfs'] + +def logging(job_id, os_params, start_time, end_time, cost_time) + line = format("%-12s %-36s %-28s %-28s %-8s\n", job_id, os_params, start_time, end_time, cost_time) + f = File.new('walk-test.report', 'a') + f.write(line) + f.close +end + +def monitor_job(job_id, job_state) + res = monitor({'job_id' => job_id, 'job_state' => job_state}, {'stop' => true}, '600s') + if res == 0 + Time.now + end +end + +def monitor(query, actions, timeout) + monitor = Monitor.new + monitor.overrides = query + monitor.action = actions + monitor.run(timeout) +end + +def submit_job(os_array) + os = os_array[0] + os_arch = os_array[1] + os_version = os_array[2] + os_mount = os_array[3] + message = %x(submit -s 'os: #{os}' -s 'os_arch: #{os_arch}' -s 'os_version: #{os_version}' -s 'os_mount: #{os_mount}' iperf-walk-os.yaml) + puts message + + job_id = message.split('=')[1].chomp + os_params = os_array.join('/') + start_time = monitor_job(job_id, 'boot') + end_time = monitor_job(job_id, 'extract_finished') + cost_time = (end_time - start_time).floor + + logging(job_id, os_params, start_time, end_time, cost_time) +end + +def get_os_array(os_mount) + TEST_OS.each do | item | + os_array = item.split.push(os_mount) + + submit_job(os_array) + end +end + +system "walk-test.report" + +logging("Job_ID", "OS", "Begin", "End", "Cost/sec") + +MOUNT_TYPE.each {| os_mount | get_os_array(os_mount)} -- 2.23.0

3 4

[PATCH compass-ci] fix: taskqueue: simplify logs
by Tong Qunfeng 19 Oct '20

19 Oct '20

[why] current taskqueue show 4 lines for 1 debug message - empty line for split: - end user request ip:port, request body: 172.17.0.1:48956 {"domain":"..."} - response body: <-- - kemal frame log: 2020-10... 200 POST /add?queue=... 710.15µs cut it two 2 line, and duplicate information - kemal frame log has time in {2020-10...} and span {710.15us} Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/taskqueue/taskqueue.cr | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/src/taskqueue/taskqueue.cr b/src/taskqueue/taskqueue.cr index c5c202d..98c5623 100644 --- a/src/taskqueue/taskqueue.cr +++ b/src/taskqueue/taskqueue.cr @@ -18,16 +18,22 @@ class TaskQueue @@rate_limiter.bucket(:l1pms, 1_u32, 0.001.seconds) @@rate_limiter.bucket(:l100pms, 1_u32, 0.00001.seconds) + # logs example: + # from: {172.17.0.1:6952} body {"domain":"compass-ci"} <-- ack: {"id":"z9.134780"} + # | from: {172.17.0.1:6952} <-- ack: {} + # 2020-10-19 03:25:01 UTC 201 PUT /consume?queue=sched%2Fdc-2g%2Fidle 356.51µs + # + # "from" is puts by this debug_message + # "2020-10..." auto puts by kemal frame, time in {2020-10-19 03:25:01 UTC} and span {356.51us} def debug_message(env, response, time_in) - puts("\n") + logs = "from: {#{env.request.remote_address}}" - from_message = "#{time_in} --> #{env.request.remote_address}" - if env.request.body != nil - from_message += " #{env.request.body}" - end - puts(from_message) + logs += " body: #{env.request.body}" if env.request.body + + logs += " <-- ack: " + logs += response ? "#{response}" : "{}" - puts("#{Time.utc} <-- #{response}") + puts(logs) end def run -- 2.23.0

1 0

[PATCH compass-ci] add exception handing for docker.rb
by Xiao Shenwei 19 Oct '20

19 Oct '20

[why] when run multi-docker job on physical machine, run 50 client to request job, if server down or error like Connection refused. these client will exit. when the server recovers, need to run these client again. [how] add exception handing to ensure all client does not exit --- providers/docker/docker.rb | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index c241d3c..2239930 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -88,6 +88,19 @@ def main(hostname) run(hostname, load_path, hash) end +def loop_main(hostname) + loop do + begin + main(hostname) + rescue StandardError => e + puts e.message + sleep 10 + ensure + sleep 5 + end + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -99,10 +112,7 @@ def multi_docker(hostname, nr_container) pids = [] nr_container.to_i.times do |i| pid = Process.fork do - loop do - main("#{hostname}-#{i}") - sleep 5 - end + loop_main("#{hostname}-#{i}") end pids << pid end -- 2.23.0

3 4

[PATCH v2 compass-ci 1/5] container/send-internet-mail: mail-post.rb
by Luan Shengde 18 Oct '20

18 Oct '20

send mail interface: get send mail request build email info from the request data call send-internet-mail to send email with email info Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-internet-mail/mail-post.rb | 46 +++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100755 container/send-internet-mail/mail-post.rb diff --git a/container/send-internet-mail/mail-post.rb b/container/send-internet-mail/mail-post.rb new file mode 100755 index 0000000..3b33c7f --- /dev/null +++ b/container/send-internet-mail/mail-post.rb @@ -0,0 +1,46 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'yaml' +require 'open3' +require_relative 'send-internet-mail.rb' + +set :bind, '0.0.0.0' +set :port, 11312 + +post '/send_mail_yaml' do + data = YAML.safe_load request.body.read + raise TypeError, data, 'request data type error' unless data.class.eql? Hash + + mail_info = { + 'references' => data['references'] || '', + 'subject' => data['subject'] || '', + 'to' => data['to'] || '', + 'body' => data['body'] || '' + } + check_send_mail(mail_info) if data +end + +post '/send_mail_text' do + data = Mail.read_from_string(request.body.read) + + mail_info = { + 'references' => data.references || '', + 'subject' => data.subject || '', + 'to' => data.to || '', + 'body' => data.body.decoded || '' + } + check_send_mail(mail_info) if data +end + +def check_send_mail(mail_info) + raise 'No subject.' if mail_info['subject'].empty? + raise 'No email address.' if mail_info['to'].empty? + raise 'No email body.' if mail_info['body'].empty? + + send_mail(mail_info) +end -- 2.23.0

3 5

[PATCH lkp-tests] jobs/fio-basic-openeuler.yaml: add fio test args
by Wei Jihui 17 Oct '20

17 Oct '20

1. add disk args, it should be tested on physical machine 2. raise nr_task to 24, for numjobs is 24. 3. add direct args Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/fio-basic-openeuler.yaml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/jobs/fio-basic-openeuler.yaml b/jobs/fio-basic-openeuler.yaml index 3fcdaac8..67d4e998 100644 --- a/jobs/fio-basic-openeuler.yaml +++ b/jobs/fio-basic-openeuler.yaml @@ -3,12 +3,18 @@ suite: fio-basic category: benchmark runtime: 60 +disk: +- 1HDD +fs: +- ext4 +nr_task: 24 + fio-setup-basic: ioengine: libaio iodepth: 1024 - nr_task: 2 test_size: 1G rwmixread: 70 + direct: 1 rw: - write - read -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: load yaml config in advance
by Wei Jihui 17 Oct '20

17 Oct '20

[why] for commit c725e0f022b7: def each_jobs(&block) each_job_init - load_hosts_config job = deepcopy(@job) @job2 = {} load_defaults + load_hosts_config each_job_init each_job(&block) @jobs.each do |hash| 1. it exec load_defaults first, so it not load hosts file when start exec load_defaults. load_defaults call expand_params, expand_params maybe expand host params, but do not have load hosts file, load_one_defaults can load host file, so exec load_one_defaults in advance. 2. load_one_defaults(file, @job) use @job is because load_hosts_file is add host config to @job, so it should be executed before "job=deepcopy(@job)" Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..27c4bd93 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -400,6 +400,11 @@ class Job @file_loaded ||= {} end + Dir.glob(["/etc/compass-ci/defaults/*.yaml", + "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| + load_one_defaults(file, @job) + end + i = include_files job = deepcopy(@job) revise_hash(job, deepcopy(@job2), true) @@ -418,11 +423,6 @@ class Job end @jobx = nil - Dir.glob(["/etc/compass-ci/defaults/*.yaml", - "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| - load_one_defaults(file, job) - end - for_each_in(job, i.keys) do |_pk, _h, k, v| job['___'] = v -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: add define files when program is cci-depends
by Wei Jihui 17 Oct '20

17 Oct '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..9a08dc30 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -617,9 +617,9 @@ class Job program_name_list = programs_params_hash.keys || [] program_name_list.each do |program| - # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg + # if job is run as makepkg, makepkg-deps, pack-deps, cci-makepkg, cci-depends # then use makepkg's benchmark as program to get define_files - if %w(makepkg makepkg-deps pack-deps cci-makepkg).include?(program) + if %w(makepkg makepkg-deps pack-deps cci-makepkg cci-depends).include?(program) program = programs_params_hash[program]["benchmark"] end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: merge overrides to defaults
by Wei Jihui 17 Oct '20

17 Oct '20

[why] submit job.yaml testbox=vm-2p8g it should use vm-2p8g to load host file, not use defaults's testbox [slove] merge @overrides to @defaults, then use @defaults to load host file Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..4e398a5a 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -382,7 +382,8 @@ class Job if defaults.is_a?(Hash) && !defaults.empty? @defaults[source_file_symkey(file)] = nil revise_hash(@defaults, defaults, true) - load_hosts_config if check_load_hosts_config(defaults) + @defaults.merge!(@overrides) + load_hosts_config if check_load_hosts_config(@defaults) end @file_loaded[file] = true true -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: set flag true only when find hosts file
by Wei Jihui 17 Oct '20

17 Oct '20

it should set load_hosts_done is true only when find and load hosts file Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..a5fe450f 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -307,10 +307,10 @@ class Job hwconfig = load_yaml(hosts_file, nil) @job[source_file_symkey(hosts_file)] = nil @job.merge!(hwconfig) { |_k, a, _b| a } # job's key/value has priority over hwconfig + @load_hosts_done = true end @job['arch'] ||= 'aarch64' - @load_hosts_done = true end def check_set_tbox_group -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor.rb: return the monitored logs
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] When other people use monitor to get logs, the logs only output, not return. It's hard to deal the logs. [How] Append result logs to a list. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 2a10b5c1..baf172d3 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -11,7 +11,7 @@ require 'eventmachine' require 'json' class Monitor - attr_accessor :monitor_url, :query, :overrides, :action, :job + attr_accessor :monitor_url, :query, :overrides, :action, :job, :result def initialize(monitor_url = '', query = {}, action = {}) @monitor_url = monitor_url @@ -25,6 +25,7 @@ class Monitor @exit_status_code = 0 @defaults = {} load_default + @result = [] end def load_default @@ -122,6 +123,7 @@ class Monitor ws.on :message do |event| data = JSON.parse(event.data) + @result << data output(data) connect(data, ws) -- 2.23.0

1 0

[PATCH compass-ci] monitoring/parse_serial_logs.cr: fix ameba warning
by Wu Zhende 17 Oct '20

17 Oct '20

parse_serial_logs.cr:46:5 [W] Lint/UselessAssign: Useless assignment to variable `host` > host = File.basename(msg["serial_path"].to_s) ^ parse_serial_logs.cr:95:5 [W] Lint/UselessAssign: Useless assignment to variable `job_id` > job_id = matched.named_captures["job_id"] ^ parse_serial_logs.cr:101:5 [W] Lint/UselessAssign: Useless assignment to variable `job` > job = Elasticsearch::Client.new.get_job_content(job_id) ^ Finished in 10.93 milliseconds 1 inspected, 3 failures. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/parse_serial_logs.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index a000ca3..6c8a220 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -43,7 +43,7 @@ class SerialParser def host_in_msg(msg) return unless msg["serial_path"]? - host = File.basename(msg["serial_path"].to_s) + File.basename(msg["serial_path"].to_s) end def detect_start_or_end(msg, host, pattern_list) @@ -92,13 +92,13 @@ class SerialParser matched = msg["message"].to_s.match(/.*\/job_initrd_tmpfs\/(?<job_id>.*?)\//) return unless matched - job_id = matched.named_captures["job_id"] + matched.named_captures["job_id"] end def find_job(job_id) return unless job_id - job = Elasticsearch::Client.new.get_job_content(job_id) + Elasticsearch::Client.new.get_job_content(job_id) end def dump_cache(job, msg, host) -- 2.23.0

1 0

[PATCH compass-ci] container/fluentd: forward docker logs to master-fluentd
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] I use a fluentd to collect docker logs, and use three output plugins:stdout/es/mq. But when I use them at the same time, will meet some problems. This fluentd will be very slow. Logs will be stacked. Another problem is that dockerd always sends logs to the same worker of fluentd.The multi-process function of fluentd can't work. [How] Change the architecture. A fluentd collects docker logs, do stdout and forward logs to master-fluentd. master-fluentd send logs to es and mq. Like this: |-----------| |--------|--> stdout |--------------|---> es |docker logs| --> |fluentd | |master-fluentd| |-----------| |--------|--> forword---> |--------------|---> mq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/fluentd/docker-fluentd.conf | 33 ++++++++++----------------- 1 file changed, 12 insertions(+), 21 deletions(-) diff --git a/container/fluentd/docker-fluentd.conf b/container/fluentd/docker-fluentd.conf index 8e76441..ac73dfb 100644 --- a/container/fluentd/docker-fluentd.conf +++ b/container/fluentd/docker-fluentd.conf @@ -60,28 +60,19 @@ </store> <store> - @type elasticsearch - host 172.17.0.1 - port 9202 - suppress_type_name true - flush_interval 1s - index_name ${tag} - ssl_verify false - num_threads 2 - </store> - - <store> - @type rabbitmq - host 172.17.0.1 - exchange logging-test - exchange_type fanout - exchange_durable false - heartbeat 10 - <format> - @type json - </format> + @type forward + flush_interval 0 + send_timeout 60 + heartbeat_type udp + heartbeat_interval 1 + recover_wait 10 + hard_timeout 60 + <server> + master-fluentd + host 172.17.0.1 + port 24225 + </server> </store> - </match> <label @FLUENT_LOG> -- 2.23.0

1 0

[PATCH lkp-tests] skip build these container for multi-docker
by Xiao Shenwei 17 Oct '20

17 Oct '20

multi-docker do't need thses container, skip it Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- setup/simplify-ci | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index e3ee3f77..eb93c150 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -36,23 +36,22 @@ boot_ipxe() run_service() { + [[ $suite == "multi-docker" ]] && return + ( cd $CCI_SRC/container/dnsmasq || return - ./build - ./start + ./build && ./start boot_ipxe )& ( cd $CCI_SRC/container/qemu-efi || return - ./build - ./install + ./build && ./install )& ( cd $CCI_SRC/container/fluentd-base || return ./build cd $CCI_SRC/container/sub-fluentd || return - ./build - ./start + ./build && ./start )& } -- 2.23.0

1 1

[PATCH compass-ci] container/master-fluentd: add a new fluentd container
by Wu Zhende 17 Oct '20

17 Oct '20

[Why] I use a fluentd to collect docker logs, and use three output plugins:stdout/es/mq. But when I use them at the same time, will meet some problems. This fluentd will be very slow. Logs will be stacked. Another problem is that dockerd always sends logs to the same worker of fluentd.The multi-process function of fluentd can't work. [How] Change the architecture. A fluentd collects docker logs, do stdout and forward logs to master-fluentd. master-fluentd send logs to es and mq. Like this: |-----------| |--------|--> stdout |--------------|---> es |docker logs| --> |fluentd | |master-fluentd| |-----------| |--------|--> forword---> |--------------|---> mq Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/Dockerfile | 6 +++ container/master-fluentd/build | 6 +++ container/master-fluentd/build-depends | 1 + container/master-fluentd/docker-fluentd.conf | 48 ++++++++++++++++++++ container/master-fluentd/start | 26 +++++++++++ 5 files changed, 87 insertions(+) create mode 100644 container/master-fluentd/Dockerfile create mode 100755 container/master-fluentd/build create mode 100755 container/master-fluentd/build-depends create mode 100644 container/master-fluentd/docker-fluentd.conf create mode 100755 container/master-fluentd/start diff --git a/container/master-fluentd/Dockerfile b/container/master-fluentd/Dockerfile new file mode 100644 index 0000000..192ece3 --- /dev/null +++ b/container/master-fluentd/Dockerfile @@ -0,0 +1,6 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM fluentd-base:alpine + +COPY --chown=1090:1090 docker-fluentd.conf /fluentd/etc/docker-fluentd.conf diff --git a/container/master-fluentd/build b/container/master-fluentd/build new file mode 100755 index 0000000..7d0291d --- /dev/null +++ b/container/master-fluentd/build @@ -0,0 +1,6 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +system 'docker build -t master-fluentd:alpine .' diff --git a/container/master-fluentd/build-depends b/container/master-fluentd/build-depends new file mode 100755 index 0000000..29ee9d3 --- /dev/null +++ b/container/master-fluentd/build-depends @@ -0,0 +1 @@ +fluentd-base diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf new file mode 100644 index 0000000..ef97d5c --- /dev/null +++ b/container/master-fluentd/docker-fluentd.conf @@ -0,0 +1,48 @@ +<system> + workers 10 +</system> + +<source> + @type forward + bind 0.0.0.0 +</source> + +<match **> + @type copy + + <store> + @type elasticsearch + host 172.17.0.1 + port 9202 + suppress_type_name true + flush_interval 1s + index_name ${tag} + ssl_verify false + log_es_400_reason true + with_transporter_log true + </store> + + <store> + @type rabbitmq + host 172.17.0.1 + exchange logging-test + exchange_type fanout + exchange_durable false + heartbeat 10 + <format> + @type json + </format> + </store> +</match> + +<label @FLUENT_LOG> + <match fluent.*> + @type stdout + <format> + @type stdout + output_type single_value + message_key message + add_newline true + </format> + </match> +</label> diff --git a/container/master-fluentd/start b/container/master-fluentd/start new file mode 100755 index 0000000..9e02980 --- /dev/null +++ b/container/master-fluentd/start @@ -0,0 +1,26 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "master-fluentd" + +cmd = %w[ + docker run + --restart=always + --name master-fluentd + -v /etc/localtime:/etc/localtime:ro + -d + -u 1090:1090 + -p 24225:24224/tcp + -p 24225:24224/udp + -e FLUENTD_CONF=docker-fluentd.conf + --log-driver json-file + --log-opt max-size=1g + master-fluentd:alpine +] +cmd += ['sh', '-c', 'umask 002 & fluentd -c /fluentd/etc/docker-fluentd.conf'] + +system(*cmd) -- 2.23.0

1 0

[PATCH compass-ci] generate conserver configuration file
by Cui Yili 16 Oct '20

16 Oct '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/conserver/conserver-head.cf | 16 ++++++++++++ container/conserver/generate_conserver.rb | 32 +++++++++++++++++++++++ 2 files changed, 48 insertions(+) create mode 100644 container/conserver/conserver-head.cf create mode 100755 container/conserver/generate_conserver.rb diff --git a/container/conserver/conserver-head.cf b/container/conserver/conserver-head.cf new file mode 100644 index 0000000..6648f88 --- /dev/null +++ b/container/conserver/conserver-head.cf @@ -0,0 +1,16 @@ +# This is a fairly basic configuration file that interacts with one + +default * { + logfile /srv/cci/serial/logs/&; # '&' is replaced with console name + timestamp 1hab; # write timestamps + rw *; # allow all users + type exec; + master localhost; + portbase 2000; # port numbers start at 2001 and + portinc 100; # go up by 1 (port #1 == 2001, etc) +} + +access * { + trusted 127.0.0.1; +} + diff --git a/container/conserver/generate_conserver.rb b/container/conserver/generate_conserver.rb new file mode 100755 index 0000000..285c393 --- /dev/null +++ b/container/conserver/generate_conserver.rb @@ -0,0 +1,32 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +require 'fileutils' +require 'yaml' + +lab = ENV['lab'] +system 'git clone file:///$CCI_REPOS/lab-$lab.git' + +def generate_conserver(lab) + return unless File.exist?('conserver-head.cf') + + FileUtils.cp('conserver-head.cf', 'conserver.cf') + + host_dir = "lab-#{lab}/hosts/" + return unless Dir.exist?(host_dir) + + Dir.each_child(host_dir) do |host| + ipmi_ip = YAML.load_file("#{host_dir}#{host}")['ipmi_ip'] + next if ipmi_ip.nil? + + console = <<~HEREDOC + console #{host} { + exec /usr/local/bin/ipmi.sh #{ipmi_ip}; + } + HEREDOC + File.open('conserver.cf', 'a') { |f| f << console } + end +end + +generate_conserver(lab) -- 2.23.0

1 0

[PATCH compass-ci] generate conserver configuration file
by Cui Yili 16 Oct '20

16 Oct '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/conserver/conserver-head.cf | 16 ++++++++++++ container/conserver/generate_conserver.rb | 30 +++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 container/conserver/conserver-head.cf create mode 100755 container/conserver/generate_conserver.rb diff --git a/container/conserver/conserver-head.cf b/container/conserver/conserver-head.cf new file mode 100644 index 0000000..6648f88 --- /dev/null +++ b/container/conserver/conserver-head.cf @@ -0,0 +1,16 @@ +# This is a fairly basic configuration file that interacts with one + +default * { + logfile /srv/cci/serial/logs/&; # '&' is replaced with console name + timestamp 1hab; # write timestamps + rw *; # allow all users + type exec; + master localhost; + portbase 2000; # port numbers start at 2001 and + portinc 100; # go up by 1 (port #1 == 2001, etc) +} + +access * { + trusted 127.0.0.1; +} + diff --git a/container/conserver/generate_conserver.rb b/container/conserver/generate_conserver.rb new file mode 100755 index 0000000..1c120b1 --- /dev/null +++ b/container/conserver/generate_conserver.rb @@ -0,0 +1,30 @@ +#!/usr/bin/env ruby + +# frozen_string_literal: true + +require 'fileutils' +require 'yaml' + +lab = ENV['lab'] +system 'git clone file:///$CCI_REPOS/lab-$lab.git' + +FileUtils.cp('conserver-head.cf', 'conserver.cf') + +def generate_conserver(lab) + hostdir = "lab-#{lab}/hosts/" + return unless Dir.exist?(hostdir) + + puts hostdir + Dir.each_child(hostdir) do |host| + ipmi_ip = YAML.load_file("#{hostdir}#{host}")['ipmi_ip'] + next if ipmi_ip.nil? + + File.open('conserver.cf', 'a') do |f| + f << "console #{host} {\n" + f << " exec /usr/local/bin/ipmi.sh #{ipmi_ip}\n" + f << "}\n" + end + end +end + +generate_conserver(lab) -- 2.23.0

1 0

[PATCH v2 compass-ci] feat: build centos initramfs image with one command
by Wang Chenglong 16 Oct '20

16 Oct '20

Use the tool to create a new centos-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/packages-to-install. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/centos/README.md | 16 ++++++++ container/osimage/centos/create-image | 43 ++++++++++++++++++++ container/osimage/centos/lib | 41 +++++++++++++++++++ container/osimage/centos/packages-to-install | 9 ++++ container/osimage/centos/packages-to-remove | 11 +++++ container/osimage/centos/run | 27 ++++++++++++ 6 files changed, 147 insertions(+) create mode 100644 container/osimage/centos/README.md create mode 100755 container/osimage/centos/create-image create mode 100755 container/osimage/centos/lib create mode 100644 container/osimage/centos/packages-to-install create mode 100644 container/osimage/centos/packages-to-remove create mode 100755 container/osimage/centos/run diff --git a/container/osimage/centos/README.md b/container/osimage/centos/README.md new file mode 100644 index 0000000..b5c7b07 --- /dev/null +++ b/container/osimage/centos/README.md @@ -0,0 +1,16 @@ ++# Use the tool to create a new centos-${version} initramfs image. ++ ++Usage: ++ cd ${CCI_SRC}/rootfs/initramfs/centos/aarch64/${version}/ ++ ./build ++ ++Some configuration items: ++./bin/packages-to-install ++ If you want to pre-install the software, you can write the package names in ./bin/packages-to-install. ++ ++./bin/packages-to-remove ++ If you want remove some unnecessary files, you can write the names in ./bin/packages-to-remove. ++ ++$HOME/.config/compass-ci/rootfs-passwd ++ Set the password for the image into this file. + diff --git a/container/osimage/centos/create-image b/container/osimage/centos/create-image new file mode 100755 index 0000000..a3495b2 --- /dev/null +++ b/container/osimage/centos/create-image @@ -0,0 +1,43 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Install functions +install_packages() +{ + echo "install" + yum repolist + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +# Setup functions +setup_rootfs() +{ + echo "setup" + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + + +# Config password +setup_login() { + [ -n "$ROOT_NEW_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${image_name}" |grep -vf /tmp/packages-to-remove | cpio -o -Hnewc | gzip -9 > /tmp/${image_name} + chmod 644 /tmp/${image_name} +} + +install_packages +setup_rootfs +setup_login +pack_cgz diff --git a/container/osimage/centos/lib b/container/osimage/centos/lib new file mode 100755 index 0000000..53aae2a --- /dev/null +++ b/container/osimage/centos/lib @@ -0,0 +1,41 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${os_name}:${os_version}" +image_name="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs-passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker_image() +{ + docker pull $image + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[INFO] Pull docker image failed." + exit + fi +} + +# cp image package to host +cp_image() +{ + docker cp -a init_docker:/tmp/${image_name} $HOME/ + echo "result: $(ls $HOME/${image_name})" +} diff --git a/container/osimage/centos/packages-to-install b/container/osimage/centos/packages-to-install new file mode 100644 index 0000000..a09bc28 --- /dev/null +++ b/container/osimage/centos/packages-to-install @@ -0,0 +1,9 @@ +openssh-server +openssl +cifs-utils +hostname +curl +iproute +iputils +which +time diff --git a/container/osimage/centos/packages-to-remove b/container/osimage/centos/packages-to-remove new file mode 100644 index 0000000..07a4a23 --- /dev/null +++ b/container/osimage/centos/packages-to-remove @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/packages-to-remove +/tmp/packages-to-install diff --git a/container/osimage/centos/run b/container/osimage/centos/run new file mode 100755 index 0000000..699bbc6 --- /dev/null +++ b/container/osimage/centos/run @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/packages-to-remove:/tmp/packages-to-remove + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker_image +${cmd[@]} +cp_image +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

1 1

help star 2020-10-16
by Wu Fengguang 16 Oct '20

16 Oct '20

Yuchuan is our help star, thanks to kind and helpful Yuchuan! $ bin/help-star people/*/credit/2020-10-16.md 31.2 Wu Fengguang 29.2 Yu Chuan 20.8 Wei Jihui 18.8 Cao Xueliang 14.8 Wang Chenglong 12.2 Liu Shaofei 11.0 Shi Zhichao 10.3 Wang Yong 10.1 Lu Weitao 10.1 Xu Xijian 9.9 Bai Jing 9.7 Tong Qunfeng 9.5 Zhang Dewan 9.4 Liu Yinsi 8.3 Wu Zhende 8.0 Luan Shengde 7.5 Cui Yili 7.5 Lin Jiaxin 6.3 Xiao Shenwei 6.1 Zhang Yale 6.1 Du Kaitian 5.3 Zhang Yu 4.9 Zhang Yuhang 4.7 Li Yuanchao 4.0 Li Ping 3.9 Ren Wen 2.3 Zhou Shengrui 2.3 Sun Yukui 2.2 Liu Saofei 1.5 Wang yong 1.3 Hu Xuejiao 0.0 Ren Zhiwen 0.0 Sun Lijun 0.0 Liang Tao 0.0 Wang Shuai 0.0 Lu Kaiyi 0.0 Cui Xucui 0.0 Xie Zheng

1 0

[PATCH compass-ci 1/2] feat: build centos initramfs image with one command
by Wang Chenglong 16 Oct '20

16 Oct '20

Use the tool to create a new centos-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/packages-to-install. --- container/osimage/centos/README.md | 15 +++++++ container/osimage/centos/bin/create-image | 36 ++++++++++++++++ container/osimage/centos/bin/lib | 41 +++++++++++++++++++ .../osimage/centos/bin/packages-to-install | 2 + .../osimage/centos/bin/packages-to-remove | 8 ++++ container/osimage/centos/run | 26 ++++++++++++ 6 files changed, 128 insertions(+) create mode 100644 container/osimage/centos/README.md create mode 100755 container/osimage/centos/bin/create-image create mode 100755 container/osimage/centos/bin/lib create mode 100644 container/osimage/centos/bin/packages-to-install create mode 100644 container/osimage/centos/bin/packages-to-remove create mode 100755 container/osimage/centos/run diff --git a/container/osimage/centos/README.md b/container/osimage/centos/README.md new file mode 100644 index 0000000..000ca05 --- /dev/null +++ b/container/osimage/centos/README.md @@ -0,0 +1,15 @@ +# Use the tool to create a new centos-${version} initramfs image. + +Usage: + cd ${CCI_SRC}/rootfs/initramfs/centos/aarch64/${version}/ + ./build + +Some configuration items: +./bin/packages-to-install + If you want to pre-install the software, you can write the package names in ./bin/packages-to-install. + +./bin/packages-to-remove + If you want remove some unnecessary files, you can write the names in ./bin/packages-to-remove. + +$HOME/.config/compass-ci/rootfs.passwd + Set the password for the image into this file. diff --git a/container/osimage/centos/bin/create-image b/container/osimage/centos/bin/create-image new file mode 100755 index 0000000..39aba7a --- /dev/null +++ b/container/osimage/centos/bin/create-image @@ -0,0 +1,36 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +# Configure rootfs +make_rootfs() +{ + yum repolist + yum -y install --skip-broken $(</root/bin/packages-to-install) + yum clean all + rm -rf $(</root/bin/packages-to-remove) + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + +# Config password +pre_config_rootfs() { + [ -n "$ROOT_NEW_PASSWD" ] && { + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_NEW_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" # Change the password in shadow + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" # Configure ssh service + } +} + +# Pack cgz package +pack_cgz() +{ + echo "Packing package. Please wait." + cd / + find ./ ! -path "./${image_name}" | cpio -o -Hnewc | gzip -9 > /${image_name} + chmod 644 /${image_name} +} + +make_rootfs +pre_config_rootfs +pack_cgz diff --git a/container/osimage/centos/bin/lib b/container/osimage/centos/bin/lib new file mode 100755 index 0000000..955717d --- /dev/null +++ b/container/osimage/centos/bin/lib @@ -0,0 +1,41 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${os_name}:${os_version}" +image_name="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +# Check password file +check_passwd_file() { + root_pwd_file="$HOME/.config/compass-ci/rootfs.passwd" + export ROOT_NEW_PASSWD= + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs.passwd" + exit 1 + } + + export ROOT_NEW_PASSWD=$(cat "$root_pwd_file") +} + +# Pull docker image +pull_docker_image() +{ + docker pull $image + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[INFO] Run command in false path ." + exit + fi +} + +# cp image package to host +cp_image() +{ + docker cp -a init_docker:/${image_name} $HOME/ + echo "result: $(ls $HOME/${image_name})" +} diff --git a/container/osimage/centos/bin/packages-to-install b/container/osimage/centos/bin/packages-to-install new file mode 100644 index 0000000..5d31c40 --- /dev/null +++ b/container/osimage/centos/bin/packages-to-install @@ -0,0 +1,2 @@ +openssh-server +openssl diff --git a/container/osimage/centos/bin/packages-to-remove b/container/osimage/centos/bin/packages-to-remove new file mode 100644 index 0000000..ce079b9 --- /dev/null +++ b/container/osimage/centos/bin/packages-to-remove @@ -0,0 +1,8 @@ +/.dockerenv +/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo diff --git a/container/osimage/centos/run b/container/osimage/centos/run new file mode 100755 index 0000000..6710872 --- /dev/null +++ b/container/osimage/centos/run @@ -0,0 +1,26 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/bin/lib +. ${CCI_SRC}/container/defconfig.sh + +check_passwd_file "$root_pwd_file" + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/bin/:/root/bin + -e ROOT_NEW_PASSWD=$ROOT_NEW_PASSWD + -e image_name=$image_name + $image + /root/bin/create-image +) + +pull_docker_image +${cmd[@]} +cp_image +docker_rm init_docker &> /dev/null +echo "build finished" + -- 2.23.0

2 4

git count 2020-10-16
by Wu Fengguang 16 Oct '20

16 Oct '20

Yuchuan is code star of the week, thanks Yuchuan! wfg@crystal ~/todo% bin/git-count 17 13 Yu Chuan 10 Wu Fengguang 8 Cao Xueliang 8 Xiao Shenwei 7 Lu Weitao 7 Wei Jihui 6 Li Yuanchao 5 Hu Xuejiao 4 Liu Yinsi 4 Zhang Yu 4 Wang Chenglong 4 Luan Shengde 4 Ren Wen 3 Xu Xijian 3 Liu Shaofei 3 Lu Kaiyi 2 Bai Jing 2 Zhang Yuhang 2 Zhang Dewan 1 Wang Yong 1 Lin Jiaxin 1 Li Ping 1 Zhou Shengrui 0 chief 0 Du Kaitian 0 Ren Zhiwen 0 Sun Lijun 0 Sun Yukui 0 Shi Zhichao 0 Wu Zhende 0 Zhang Yale 0 Cui Yili 0 Liang Tao 0 Xie Zheng

1 0

review star 2020-10-16
by Wu Fengguang 16 Oct '20

16 Oct '20

Shenwei is our review star, thanks Shenwei! wfg@crystal ~/todo% bin/review-star Review email records in the last 7 days total review paragraph point people 266 148 293 63 Wu Fengguang 43 14 35 3 Yu Chuan 55 12 35 15 Tong Qunfeng 49 17 31 19 Xiao Shenwei 31 19 29 8 Liu Yinsi 30 7 21 3 Lu Weitao 34 12 18 3 Ren Wen 16 10 18 11 Xu Xijian 26 9 13 3 Wei Jihui 26 8 9 4 Cao Xueliang 5 3 7 0 Du Kaitian 3 3 6 0 Sun Yukui 27 2 5 0 Hu Xuejiao 60 2 4 3 Luan Shengde 44 2 4 0 Wang Chenglong 14 4 4 0 Cui Yili 2 2 4 0 Xie Zheng 19 1 3 0 Liu Shaofei 28 3 3 2 Zhang Yu 15 2 3 1 Li Yuanchao 16 3 2 1 Bai Jing 2 1 2 1 Lin Jiaxin 36 2 2 0 Zhang Yale 5 2 2 0 Zhou Shengrui 2 2 2 0 Wu Zhende 8 5 1 0 Lu Kaiyi 4 0 0 0 Wang Yong 4 0 0 0 Shi Zhichao 14 1 0 0 Zhang Dewan 15 0 0 0 Li Ping 26 0 0 0 Zhang Yuhang

1 0

[PATCH compass-ci] sched: use user lkp src in set_pp_initrd
by Cao Xueliang 16 Oct '20

16 Oct '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index effba9b..dc9d042 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -6,6 +6,7 @@ require "yaml" require "any_merge" require "scheduler/constants.cr" +require "scheduler/jobfile_operate.cr" struct JSON::Any def []=(key : String, value : String) @@ -80,6 +81,7 @@ class Job access_key access_key_file lkp_initrd_user + user_lkp_src kernel_uri kernel_append_root kernel_params @@ -126,6 +128,7 @@ class Job private def set_defaults append_init_field() + set_user_lkp_src() set_os_dir() set_result_root() set_result_service() @@ -381,6 +384,15 @@ class Job self["kernel_params"] = " #{kernel_common_params()} #{kernel_append_root} #{kernel_console()}" end + private def set_user_lkp_src + lkp_arch_cgz = "#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz" + if File.exists?(lkp_arch_cgz) + self["user_lkp_src"] = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) + else + raise "The #{lkp_arch_cgz} does not exist." + end + end + private def set_pp_initrd initrd_deps_arr = Array(String).new initrd_pkg_arr = Array(String).new @@ -392,11 +404,11 @@ class Job deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?("#{user_lkp_src}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") + if File.exists?("#{user_lkp_src}/pkg/#{program}") initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) end end -- 2.23.0

2 2

[PATCH v2 compass-ci] sched: use user lkp src in set_pp_initrd
by Cao Xueliang 16 Oct '20

16 Oct '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index effba9b..20afa65 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -6,6 +6,7 @@ require "yaml" require "any_merge" require "scheduler/constants.cr" +require "scheduler/jobfile_operate.cr" struct JSON::Any def []=(key : String, value : String) @@ -80,6 +81,7 @@ class Job access_key access_key_file lkp_initrd_user + user_lkp_src kernel_uri kernel_append_root kernel_params @@ -126,6 +128,7 @@ class Job private def set_defaults append_init_field() + set_user_lkp_src() set_os_dir() set_result_root() set_result_service() @@ -381,6 +384,13 @@ class Job self["kernel_params"] = " #{kernel_common_params()} #{kernel_append_root} #{kernel_console()}" end + private def set_user_lkp_src + lkp_arch_cgz = "#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz" + raise "The #{lkp_arch_cgz} does not exist." unless File.exists?(lkp_arch_cgz) + + self["user_lkp_src"] = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) + end + private def set_pp_initrd initrd_deps_arr = Array(String).new initrd_pkg_arr = Array(String).new @@ -392,11 +402,11 @@ class Job deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?("#{user_lkp_src}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") + if File.exists?("#{user_lkp_src}/pkg/#{program}") initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) end end -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-account.sh update es-mapping
by Lu Weitao 16 Oct '20

16 Oct '20

modify some fields to make them more specific Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-account.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sbin/es-account.sh b/sbin/es-account.sh index 2b739d5..635f4fc 100755 --- a/sbin/es-account.sh +++ b/sbin/es-account.sh @@ -18,16 +18,16 @@ else "my_commit_url": { "type": "keyword" }, - "uuid": { + "my_uuid": { "type": "keyword" }, - "email": { + "my_email": { "type": "keyword" }, - "name": { + "my_name": { "type": "keyword" }, - "user": { + "my_login_name": { "type": "keyword" } } -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/send-mail: update Dockerfile
by Luan Shengde 16 Oct '20

16 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/Dockerfile | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index 36b145c..0cf3eeb 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -11,9 +11,7 @@ RUN apt-get update && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone -EXPOSE 25 - COPY mail-post.rb /usr/local/bin/ COPY send-mail.rb /usr/local/bin/ -CMD ["/usr/local/bin/mail-post.rb"] +WORKDIR /usr/local/bin/ -- 2.23.0

1 0

[PATCH compass-ci 2/2] web-backend: add search feature to "/get_repos" api
by Zhang Yuhang 16 Oct '20

16 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 6 ++- src/lib/web_backend.rb | 82 ++++++++++++++++--------------- 2 files changed, 48 insertions(+), 40 deletions(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 60c7827..fcc765f 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -9,6 +9,7 @@ require "#{CCI_SRC}/src/lib/web_backend.rb" set :bind, '0.0.0.0' set :port, 32767 set :show_exceptions, false +set :environment, :production # GET /compare_candidates # return: {"query_conditions": {...}, "dimension": [...]} @@ -48,7 +49,10 @@ get '/get_jobs' do get_jobs(params) end -# GET /get_repos?page_size=12&page_num=3 +# GET /get_repos?git_repo=a&page_size=12&page_num=3 +# git_repo: +# > optional +# # page_size: # > optional # - default: 20 diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 31aa0c3..48cfadb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -95,7 +95,7 @@ def compare_candidates_body OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, - { os: 'debian', os_version: ['10', 'sid'] }, + { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], os_arch: %w[aarch64 x86_64], @@ -304,7 +304,7 @@ def get_jobs_body(params) banner: get_banner(git_repo, branches), jobs: jobs, fields: FIELDS - } + }.to_json end def get_jobs(params) @@ -314,66 +314,70 @@ def get_jobs(params) warn e.message return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get jobs error'] end - [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end -def get_repo_url(repo_file) - return unless File.file? repo_file +def get_repo_url(urls) + return unless urls.is_a?(Array) - urls = YAML.load_file(repo_file)['url'] urls.each do |url| return url if url[0, 4] == 'http' end urls[0] end -def get_repo(git_repo, repo_file = nil) - repo = { - git_repo: git_repo, - git_url: nil - } - return repo if !git_repo && !repo_file - - if !git_repo - git_repo = repo_file[UPSTREAM_REPOS_PATH.size + 1, repo_file.size - 1] - repo[:git_repo] = git_repo - elsif !repo_file - repo_file = File.join(UPSTREAM_REPOS_PATH, git_repo) +def get_repo(git_repo) + repo = nil + if git_repo + must = [{ regexp: { git_repo: ".*#{git_repo}.*" } }] + repo = query_repos(must, from: 0, size: 1)[0] end - - repo[:git_url] = get_repo_url(repo_file) - repo + repo || {} end -def repo_files_list - Dir["#{UPSTREAM_REPOS_PATH}/*/*/*"].sort +def query_repos(must, from: 0, size: 1) + query = { + query: { bool: { must: must } }, + size: size, + from: from, + sort: [{ + git_repo: { order: 'asc' } + }] + } + result = ES_CLIENT.search index: 'repo', body: query + repos = [] + result['hits']['hits'].each do |r| + r = r['_source'] + repos << { + git_url: get_repo_url(r['url']), + git_repo: r['git_repo'] + } + end + repos end -def get_repos_list(repo_files, from, finish, total) - repos_list = [] - - total.times do |index| - next if index < from - break if index >= finish +def search_repos(git_repo, page_size, page_num) + size = page_size + from = size * page_num + must = git_repo ? [{ regexp: { git_repo: ".*#{git_repo}.*" } }] : [] + count_query = { query: { bool: { must: must } } } + total = ES_CLIENT.count(index: 'repo', body: count_query)['count'] + return [], total if wrong_size?(size, from) - repos_list << get_repo(nil, repo_files[index]) - end - repos_list + return query_repos(must, from: from, size: size), total end def get_repos_body(params) - repo_files = repo_files_list page_size = get_positive_number(params[:page_size], 20) page_num = get_positive_number(params[:page_num], 1) - 1 + git_repo = params[:git_repo] - from = page_num * page_size - finish = from + page_size - total = repo_files.size + repos, total = search_repos(git_repo, page_size, page_num) { total: total, - repos: get_repos_list(repo_files, from, finish, total) - } + repos: repos + }.to_json end def get_repos(params) @@ -383,5 +387,5 @@ def get_repos(params) warn e.message return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] end - [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end -- 2.23.0

2 2

[PATCH v3 compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7785c2a..984b9a1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + unless hash_dup[key] == @hash[key] + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,7 +389,7 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) -- 2.23.0

1 0

[PATCH v2 compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7785c2a..a0cae68 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + unless hash_dup[key] == @hash[key] + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,9 +389,9 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}")) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end -- 2.23.0

1 0

[PATCH compass-ci] job.cr: def var osimage_dir for used multi times
by Xu Xijian 16 Oct '20

16 Oct '20

osimage_dir is used twice here, define a var fo it. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..fd6e79b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -299,10 +299,11 @@ class Job private def initramfs_initrds temp_initrds = [] of String + osimage_dir = "#{SRV_INITRD}/osimage/#{os_dir}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + "#{JobHelper.service_path("#{osimage_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

1 0

[PATCH compass-ci] job.cr: def var initrd_os_dir for used multi times
by Xu Xijian 16 Oct '20

16 Oct '20

initrd_os_dir is used twice here, define a var fo it. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..fc712ba 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -299,10 +299,11 @@ class Job private def initramfs_initrds temp_initrds = [] of String + initrd_os_dir = "#{SRV_INITRD}/osimage/#{os_dir}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + "#{JobHelper.service_path("#{initrd_os_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + "#{JobHelper.service_path("#{initrd_os_dir}/run-ipconfig.cgz")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

2 2

[PATCH lkp-tests] fix(iso2rootfs): modify the soft link name
by Yu Chuan 16 Oct '20

16 Oct '20

'dailybuild' is closer to logic than 'debug'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 7ab4a08b0d60..524ab6695f93 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,7 @@ get_daily_iso_checksum() mount_rootfs() { - DEBUG_ROOTFS_DIR="debug-versions/dailybuild" + DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +103,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DEBUG_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +255,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DEBUG_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +310,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DEBUG_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DEBUG_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -363,10 +363,11 @@ post_works() mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" + local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && - rm -f "${iso_version}-debug" && - ln -s "${DEBUG_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ - "${iso_version}-debug" + rm -f "${soft_link}" && + ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ + "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 2

[PATCH compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..ffe3749 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + if "#{hash_dup[key]}" != "#{@hash[key]}" + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,7 +389,7 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") && File.exists?(deps_dest_file) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end -- 2.23.0

2 2

[PATCH v8 lkp-tests] sshd: send emails to login services for users
by Zhang Yale 16 Oct '20

16 Oct '20

Provide a login port for user by email, determine whether to send the ssh_port message based on the uuid. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 89 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 20 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..1f7c4126 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,16 @@ #!/bin/sh # pub_key # email +# uuid +# ssh_port_len +# ssh_port_base + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +sshr_port=5050 +[ -n "$sshr_port_len" ] || sshr_port_len=2000 +[ -n "$sshr_port_base" ] || sshr_port_base=30000 run_ssh() { @@ -11,40 +21,79 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh +Subject: [NOTIFY Compass-ci] $testbox ready to use + +Dear $user_name: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: + login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch - Login: - ssh root@$PUB_IP - Due time: - $deadline +Regards +Compass-ci +" +} - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch +data_failure() +{ + data="To: $email +Subject: [NOTIFY Compass-ci] Applying $testbox failed + +Dear $user_name: + Sorry to inform you that your application failed. + You may need to wait for a while and then try again. Regards -compass-ci +Compass-ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + +get_port() +{ + [ -n "$uuid" ] || return + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ + -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] || continue + return + done + port="" +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$email" ] && [ -n "$uuid" ] || return 0 + user_name=$(echo "$email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +get_port +report_ssh_port $port run_email -- 2.23.0

2 2

[PATCH compass-ci 4/4] container/send-mail: update start
by Luan Shengde 16 Oct '20

16 Oct '20

add setup default variables update heredoc Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/start | 74 +++++++++++++++++++++++++-------------- 1 file changed, 47 insertions(+), 27 deletions(-) diff --git a/container/send-mail/start b/container/send-mail/start index 80be86a..e287322 100755 --- a/container/send-mail/start +++ b/container/send-mail/start @@ -1,29 +1,53 @@ -#!/bin/bash +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -. $CCI_SRC/container/defconfig.sh +require 'set' +require_relative '../defconfig' -docker_rm send-mail +docker_rm 'send-mail' -cmd=( - docker run - --restart=always - --name=send-mail - -u nobody - -d - -p 11311:11311 - -v /etc/localtime:/etc/localtime:ro - send-mail -) +names = Set.new %w[ + EMAIL_PSWD + SEND_MAIL_PORT +] -"${cmd[@]}" +defaults = relevant_defaults(names) -cat<<EOF +EMAIL_PSWD = defaults['EMAIL_PSWD'] +SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] + +cmd = %W[ + docker run + --restart=always + --name=send-mail + -u nobody + -d + -e EMAIL_PSWD=#{EMAIL_PSWD} + -e SEND_MAIL_PORT=#{SEND_MAIL_PORT} + -p #{SEND_MAIL_PORT}:#{SEND_MAIL_PORT} + -v /etc/localtime:/etc/localtime:ro + --log-driver json-file + send-mail +] + +cmd += ['bash', '-c', '/usr/local/bin/mail-post.rb'] + +system(*cmd) + +puts <<HEREDOC port: 11311 + +default from email: + when sending crystal.ci email. + team(a)crystal.ci if no from email appiontted + when sending to WAN email + compass-ci(a)qq.com in all cases + --- data='{ -"from": "email_from_addr" # default: team(a)crystal.ci +"from": "email_from_addr" "subject": "email subject", "to": "email_to_addr", "body": "email message" @@ -32,19 +56,17 @@ data='{ or data=" -from: email_from_addr # default: team(a)crystal.ci +from: email_from_addr subject: email subject to: email_to_addr body: email message" -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_yaml' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_yaml' -d "$data" +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_yaml' -d "$data" --- data=" -From: email_from_addr # default: team(a)crystal.ci +From: email_from_addr To: email_to_addr Subject: email_subject @@ -53,8 +75,6 @@ mail_msg_line2 ... " -when compass-ci: - curl -XPOST 'localhost:11311/send_mail_text' -d "$data" -when other server: - curl -XPOST '124.90.34.227:11311/send_mail_text' -d "$data" -EOF +Usage: + curl -XPOST '$send_mail_server:$send_mail_port/send_mail_text' -d "$data" +HEREDOC -- 2.23.0

1 0

[PATCH compass-ci 3/4] container/send-mail: update Dockerfile
by Luan Shengde 16 Oct '20

16 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/Dockerfile | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index 36b145c..0cf3eeb 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -11,9 +11,7 @@ RUN apt-get update && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone -EXPOSE 25 - COPY mail-post.rb /usr/local/bin/ COPY send-mail.rb /usr/local/bin/ -CMD ["/usr/local/bin/mail-post.rb"] +WORKDIR /usr/local/bin/ -- 2.23.0

1 0

[PATCH compass-ci 2/4] container/send-mail: send-mail.rb
by Luan Shengde 16 Oct '20

16 Oct '20

add setup smtp for qq.com enable send mail via smtp.qq.com Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/send-mail.rb | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/container/send-mail/send-mail.rb b/container/send-mail/send-mail.rb index f459e78..4fb2b7e 100755 --- a/container/send-mail/send-mail.rb +++ b/container/send-mail/send-mail.rb @@ -5,15 +5,29 @@ require 'mail' -mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp +# mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp # setup smtp config -smtp = { - address: mail_server, - enable_starttls_auto: false -} +def intranet_smtp(mail_server) + smtp = { + address: mail_server, + enable_starttls_auto: false + } + Mail.defaults { delivery_method :smtp, smtp } +end -Mail.defaults { delivery_method :smtp, smtp } +def internet_smtp + smtp = { + address: 'smtp.qq.com', + port: 25, + domain: 'qq.com', + user_name: 'compass-ci(a)qq.com', + password: (ENV['EMAIL_PSWD']).to_s, + openssl_verify_mode: 'none', + enable_starttls_auto: true + } + Mail.defaults { delivery_method :smtp, smtp } +end # send mail def send_mail(mail_info) -- 2.23.0

1 0

[PATCH compass-ci 1/4] container/send-mail:mail-post.rb
by Luan Shengde 16 Oct '20

16 Oct '20

add setup smpt for intranet/internet smtp enable send mail to both curystal.ci and WAN email Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/mail-post.rb | 36 +++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/container/send-mail/mail-post.rb b/container/send-mail/mail-post.rb index 4eecf6e..c7c31d5 100755 --- a/container/send-mail/mail-post.rb +++ b/container/send-mail/mail-post.rb @@ -7,21 +7,21 @@ require 'sinatra' require 'json' require 'yaml' require 'open3' -require_relative 'send-mail.rb' +require_relative 'send-mail' set :bind, '0.0.0.0' -set :port, 11311 +set :port, ENV['SEND_MAIL_PORT'] post '/send_mail_yaml' do data = YAML.safe_load request.body.read - raise TypeError, data, 'request data type error' unless data.class.eql? Hash + raise TypeError, data, 'request data type error' unless data.instance_of?(Hash) mail_info = { 'references' => data['references'] || '', 'from' => data['from'] || 'team(a)crystal.ci', - 'subject' => data['subject'], - 'to' => data['to'], - 'body' => data['body'] + 'subject' => data['subject'] || '', + 'to' => data['to'] || '', + 'body' => data['body'] || '' } check_send_mail(mail_info) end @@ -32,17 +32,29 @@ post '/send_mail_text' do mail_info = { 'references' => data.references || '', 'from' => data.from || 'team(a)crystal.ci', - 'subject' => data.subject, - 'to' => data.to, - 'body' => data.body.decoded + 'subject' => data.subject || '', + 'to' => data.to[0] || '', + 'body' => data.body.decoded || '' } check_send_mail(mail_info) end def check_send_mail(mail_info) - raise TypeError, data, 'empty subject.' if mail_info['subject'].empty? - raise TypeError, data, 'empty email address.' if mail_info['to'].empty? - raise TypeError, data, 'empty email content.' if mail_info['body'].empty? + raise 'empty subject.' if mail_info['subject'].empty? + raise 'empty email address.' if mail_info['to'].empty? + raise 'empty email content.' if mail_info['body'].empty? + + check_smtp(mail_info) send_mail(mail_info) end + +def check_smtp(mail_info) + mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp + if [nil, 'crystal.ci'].include? mail_info['to'].split('@')[1] + intranet_smtp(mail_server) + else + internet_smtp + mail_info['from'] = 'compass-ci(a)qq.com' + end +end -- 2.23.0

1 0

[PATCH lkp-tests] tests: add get_ipmi_ip function to host-info
by Zhang Yu 16 Oct '20

16 Oct '20

If ipmitool returns error or empty, the return of "$(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" is null, and only add the field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/tests/host-info b/tests/host-info index a57855dc..6ee8df21 100755 --- a/tests/host-info +++ b/tests/host-info @@ -80,6 +80,11 @@ get_partitions() fi } +get_ipmi_ip() +{ + echo "ipmi_ip: $(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" +} + main() { get_node @@ -87,6 +92,7 @@ main() get_partitions get_mac get_model_name + get_ipmi_ip } main -- 2.23.0

4 3

[PATCH compass-ci] feat(sparrow): add kernel param: net.ipv6.bindv6only=1
by Yu Chuan 16 Oct '20

16 Oct '20

Turn on the bindv6only function to avoid the use of tcp6 port for services that do not actually need tcp6. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/0-package/common | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2c7de3754fad..31588f40b602 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -10,6 +10,7 @@ grep -q '^committer:' /etc/group || groupadd -g 1999 committer cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 +net.ipv6.bindv6only=1 vm.max_map_count=262144 EOF -- 2.23.0

1 0

[PATCH compass-ci 1/2] web-backend-nginx: Restrict interface access.
by Zhang Yuhang 16 Oct '20

16 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend-nginx/nginx.conf | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/container/web-backend-nginx/nginx.conf b/container/web-backend-nginx/nginx.conf index 5ab1a26..b461a2c 100644 --- a/container/web-backend-nginx/nginx.conf +++ b/container/web-backend-nginx/nginx.conf @@ -6,7 +6,7 @@ events { http { include /etc/nginx/mime.types; default_type application/octet-stream; - include /etc/nginx/conf.d/*.conf; + include /etc/nginx/conf.d/*.conf; server { listen 11320 ssl; server_name api.compass-ci.openeuler.org; @@ -15,8 +15,7 @@ http { ssl_session_timeout 5m; ssl_ciphers BCDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4; ssl_prefer_server_ciphers on; - - location / { + location ~ ^/(get_repos|get_jobs|compare|compare_candidates)$ { proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forward-For $proxy_add_x_forwarded_for; -- 2.23.0

1 0

[PATCH v6 compass-ci 2/6] container/mail-robot: answerback-email.rb
by Luan Shengde 16 Oct '20

16 Oct '20

AssignUuid: read email content check email available check if base url in upstream-repos check if commit url valid check pub_key assign account generate uuid write account/uuid to es send uuid to user Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/mail-robot/answerback-email.rb | 181 +++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100755 container/mail-robot/answerback-email.rb diff --git a/container/mail-robot/answerback-email.rb b/container/mail-robot/answerback-email.rb new file mode 100755 index 0000000..2cd15ef --- /dev/null +++ b/container/mail-robot/answerback-email.rb @@ -0,0 +1,181 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'mail' +require_relative '/compass-ci/lib/es_client.rb' + +# Assign uuid/account +class AssignUuid + def initialize(mail_content) + @mail_content = mail_content + end + + def build_message(sum_infos) + message = <<~EMAIL_MESSAGE + To: #{sum_infos['email']} + Message-ID: #{sum_infos['message_id']} + Subject: Account Ready + + Dear #{sum_infos['email'].split('@')[0]} + + Thank you for joining us. + You can use the following info to submit jobs: + + First: add the info to ~/.config/compass-ci/defaults/{USER}.yaml + + uuid: #{sum_infos['account_uuid']} + lab: z9 + + Second: download lkp-tests and dependencies + git clone https://gitee.com_/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source {HOME}/.bashrc && source {HOME}/.bash_profile + + Third: submit job + reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md + how to write job yaml + submit jobs/netperf.yaml + + Repalce the fields {...} with your local environment variable. + + regards + compass-ci + EMAIL_MESSAGE + + return message + end + + def url_addr + url_line = @mail_content.part[0].body.decoded.split(/\n/).find { |line| line =~ %r{https?://} } + message = 'No commit url found' + + raise message unless url_line + + url = url_line.split[-1] + + return url + end + + def check_email_available(email) + url = url_addr + + base_url = url.gsub(%r{/commit/[^/].*}, '') + + upstream_dir = '/upstream-repos' + check_url_in_upstream_repos(upstream_dir, base_url) + check_url_commit(url, base_url, email) + + return url + end + + def check_url_in_upstream_repos(upstream_dir, base_url) + Dir.chdir(upstream_dir) + match_out = %x(grep -rn #{base_url}) + + message = 'The url is not in upstream-repo list' + + raise StandardError, message, base_url if match_out.empty? + end + + def check_url_commit(url, base_url, email) + hub_name = url.split('/')[2] + check_gitee_commit(url, base_url, email) if hub_name.eql? 'gitee.com' + check_non_gitee_commit(url, email) unless hub_name.eql? 'gitee.com' + end + + def check_non_gitee_commit(url, email) + url_fdback = %x(curl #{url}) + email_index = url_fdback.index email + + message = 'No email matched from the commit_id feedback.' + raise StandardError, message, url if email_index.nil? + end + + def check_gitee_commit(url, base_url, email) + repo_info = { + 'repos_dir' => '/repos_dir', + 'repo_dir' => url.split('/')[-3], + 'repo_url' => [base_url, 'git'].join('.'), + 'repo_path' => File.join('/repos_dir', url.split('/')[-3]), + 'commit_id' => url.split('/')[-1] + } + check_gitee_repo_commit(url, email, repo_info) + end + + def check_gitee_repo_commit(url, email, repo_info) + Dir.chdir repo_info['repos_dir'] + %x(/usr/bin/git clone #{repo_info['repo_url']}) + email_index = %x(/usr/bin/git -C #{repo_info['repo_dir']} show #{repo_info['commit_id']}).index email + + FileUtils.rm_rf repo_info['repo_dir'] + + message = 'No email matched from commit_id feedback.' if email_index.nil? + raise StandardError, message, url if email_index.nil? + end + + def pub_key + pub_key = @mail_content.part[1].body.decoded.split(/\r|\n/).join if @mail_content.part[1].filename == 'id_rsa.pub' + + message = 'No pub_key found' + + raise message unless pub_key + + return pub_key + end + + def account_info(pub_key) + account_info_str = %x(curl -XGET "#{ENV['J_IP']}:#{ENV['J_PORT']}/assign_account" -d "pub_key: #{pub_key}") + account_info = JSON.parse account_info_str + + message = 'No available jumper account returned.' + raise StandardError, message, account_info if account_info['account'].empty? + + return account_info + end + + def build_sum_infos + email = @mail_content.from[0] + account_uuid = %x(uuidgen).chomp + message_id = @mail_content.message_id + sum_infos = { + 'email' => email, + 'message_id' => message_id, + 'url' => check_email_available(email), + 'acct_infos' => account_info(pub_key), + 'account_uuid' => account_uuid, + 'pub_key' => pub_key + } + store_account_info(sum_infos) + sum_infos + end + + def send_uuid(sum_infos) + message = build_message(sum_infos) + + send_mail_server = `/sbin/ip route |awk '/default/ {print $3}'`.chomp + %x(curl -XPOST "#{send_mail_server}:#{ENV['S_M_PORT']}/send_mail_text" -d "#{message}") + end + + def send_account + sum_infos = build_sum_infos + + send_uuid(sum_infos) + end + + def store_account_info(sum_infos) + account_uuid_info = { + 'email' => sum_infos['email'], + 'name' => sum_infos['email'].split('@')[0], + 'uuid' => sum_infos['account_uuid'], + 'user' => sum_infos['acct_infos']['account'], + 'my_commit_url' => sum_infos['url'] + } + + es = ESClient.new(index: 'accounts') + es.put_source_by_id(sum_infos['email'], account_uuid_info) + end +end -- 2.23.0

3 11

[PATCH compass-ci 2/3] git_mirror: put repo info to es
by Li Yuanchao 16 Oct '20

16 Oct '20

move a method to pass rubocop limit: Class xxx has too many lines[101/100] Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5f7fa6b..fef4aca 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -9,6 +9,8 @@ require 'json' # gem install PriorityQueue require 'priority_queue' require 'English' +require 'elasticsearch' +require_relative 'constants.rb' # worker threads class GitMirror @@ -84,6 +86,7 @@ class MirrorMain @git_info = {} @defaults = {} @git_queue = Queue.new + @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") load_fork_info connection = Bunny.new('amqp://172.17.0.1:5672') connection.start @@ -108,16 +111,6 @@ class MirrorMain @defaults[project] = YAML.safe_load(File.open(defaults_file)) end - def load_repo_file(repodir, project, fork_name) - git_repo = "#{project}/#{fork_name}" - @git_info[git_repo] = YAML.safe_load(File.open(repodir)) - @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] - fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 - end - def traverse_repodir(repodir) if File.directory? repodir load_defaults(repodir) @@ -194,6 +187,17 @@ end # main thread class MirrorMain + def load_repo_file(repodir, project, fork_name) + git_repo = "#{project}/#{fork_name}" + @git_info[git_repo] = YAML.safe_load(File.open(repodir)) + @git_info[git_repo]['git_repo'] = git_repo + @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] + es_repo_update(git_repo, @git_info[git_repo]) + fork_stat_init(git_repo) + @priority_queue.push git_repo, @priority + @priority += 1 + end + def compare_refs(cur_refs, old_refs) new_refs = { heads: {} } cur_refs[:heads].each do |ref, commit_id| @@ -246,4 +250,12 @@ class MirrorMain load_repo_file(repo_dir, File.dirname(file), File.basename(file)) if File.file?(repo_dir) end end + + def es_repo_update(git_repo, repo_info) + body = { + "doc": repo_info, + "doc_as_upsert": true + } + @es_client.update(index: 'repo', type: '_doc', id: git_repo, body: body) + end end -- 2.23.0

2 1

[PATCH lkp-tests] tests: add get_ipmi_ip function to host-info
by Zhang Yu 16 Oct '20

16 Oct '20

If ipmitool returns error or empty, the return of "$(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" is null, and then only add the field Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/tests/host-info b/tests/host-info index a57855dc..cd77c1eb 100755 --- a/tests/host-info +++ b/tests/host-info @@ -69,8 +69,7 @@ get_partitions() # sdf 1 ssd=$(lsblk -d -o name,ROTA |grep 0$ | awk '{print $1}') hdd=$(lsblk -d -o name,ROTA |grep 1$ | awk '{print $1}') - if [ -n "$hdd" ]; then - echo "hdd_partitions:" + if [ -n "$hdd" ]; then echo "hdd_partitions:" get_part $hdd fi @@ -80,6 +79,11 @@ get_partitions() fi } +get_ipmi_ip() +{ + echo "ipmi_ip: $(ipmitool lan print |grep -E "IP Address\s*:" | awk '{print $4}')" +} + main() { get_node @@ -87,6 +91,7 @@ main() get_partitions get_mac get_model_name + get_ipmi_ip } main -- 2.23.0

1 1

[PATCH compass-ci] fix docker-pull can't be used
by Xiao Shenwei 16 Oct '20

16 Oct '20

when run multi-docker job, the PATH exclude $CCI_SRC, so docker-pull can't be used directly, use full path to replace Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/docker/docker.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index b454f68..c241d3c 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -69,7 +69,7 @@ end def run(hostname, load_path, hash) docker_image = hash['docker_image'] - system "docker-pull #{docker_image}" + system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, ENV['CCI_SRC'] + '/providers/docker/run.sh' -- 2.23.0

1 0

[PATCH compass-ci] sched: throw an exception when deps/pkg does not exist
by Cao Xueliang 16 Oct '20

16 Oct '20

When submitting a job, an exception is thrown when deps/pkg does not exist on the server. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..ef515b8 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -389,11 +389,21 @@ class Job program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") && File.exists?(deps_dest_file) - initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?(deps_dest_file) + initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) + else + raise "#{deps_dest_file} does not exist on the server." + end end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") && File.exists?(pkg_dest_file) - initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) + + if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") + if File.exists?(pkg_dest_file) + initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) + else + raise "#{pkg_dest_file} does not exist on the server." + end end end end -- 2.23.0

1 1

[PATCH lkp-tests] lib/job.rb: load host config with atomic job
by Wei Jihui 15 Oct '20

15 Oct '20

according to commit c725e0f022b707ba385f8f3657e79, job = deepcopy(@job) without load_hosts_config it will keep to @job for "@job = deepcopy(job)" under "@jobs.each do|hash|" so this @job is without load_hosts_config. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 14946f47..6359254e 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -696,9 +696,11 @@ class Job each_job_init each_job(&block) @jobs.each do |hash| + @load_hosts_done = false @job = deepcopy(job) @job2 = hash load_defaults + load_hosts_config each_job_init each_job(&block) end -- 2.23.0

2 1

[PATCH compass-ci 3/3] container: git-mirror: add elasticsearch in dockerfile
by Li Yuanchao 15 Oct '20

15 Oct '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/git-mirror/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/git-mirror/Dockerfile b/container/git-mirror/Dockerfile index 9c582f7..390bbcc 100644 --- a/container/git-mirror/Dockerfile +++ b/container/git-mirror/Dockerfile @@ -12,5 +12,5 @@ RUN apk update && \ apk add ruby-dev make gcc g++ RUN umask 002 && \ - gem install bunny json PriorityQueue --no-rdoc --no-ri + gem install bunny json PriorityQueue elasticsearch --no-rdoc --no-ri -- 2.23.0

1 0

[PATCH compass-ci 1/3] git_mirror: optimize variable name
by Li Yuanchao 15 Oct '20

15 Oct '20

the same thing, the same name Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 779b488..5f7fa6b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -54,7 +54,7 @@ class GitMirror def mirror_sync fork_info = @queue.pop - mirror_dir = "/srv/git/#{fork_info['forkdir']}.git" + mirror_dir = "/srv/git/#{fork_info['git_repo']}.git" possible_new_refs = false if File.directory?(mirror_dir) possible_new_refs = git_fetch(mirror_dir) @@ -62,7 +62,7 @@ class GitMirror FileUtils.mkdir_p(mirror_dir) possible_new_refs = git_clone(fork_info['url'], mirror_dir) end - feedback(fork_info['forkdir'], possible_new_refs) + feedback(fork_info['git_repo'], possible_new_refs) end def git_mirror @@ -109,11 +109,12 @@ class MirrorMain end def load_repo_file(repodir, project, fork_name) - @git_info["#{project}/#{fork_name}"] = YAML.safe_load(File.open(repodir)) - @git_info["#{project}/#{fork_name}"]['forkdir'] = "#{project}/#{fork_name}" - @git_info["#{project}/#{fork_name}"].merge!(@defaults[project]) if @defaults[project] - fork_stat_init("#{project}/#{fork_name}") - @priority_queue.push "#{project}/#{fork_name}", @priority + git_repo = "#{project}/#{fork_name}" + @git_info[git_repo] = YAML.safe_load(File.open(repodir)) + @git_info[git_repo]['git_repo'] = git_repo + @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] + fork_stat_init(git_repo) + @priority_queue.push git_repo, @priority @priority += 1 end -- 2.23.0

1 0