- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 3/3] container/mail-robot: optimize parsing email
by Luan Shengde 24 Mar '21

24 Mar '21

optimize code: use gem resource ruby-git instead of system command Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index e6c35f7..ce01fec 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -3,9 +3,13 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +# set default system encoding +Encoding.default_external = Encoding::UTF_8 + require 'json' require 'mail' require 'rest-client' +require 'git' # used to parse mail_content for my_commit_url and my_ssh_pubkey # be called by AssignAccount when it needs to extract required data: @@ -110,14 +114,15 @@ class ParseApplyAccountEmail repo_dir = repo_url.split('/')[-1] commit_id = url.split('/')[-1] - %x(/usr/bin/git -C /tmp clone --bare #{repo_url} #{repo_dir}) + Git.clone(repo_url, repo_dir, path: '/tmp/', bare: true) # get all commit IDs and check commit id exists - repo_commits = %x(/usr/bin/git -C /tmp/#{repo_dir} log --pretty=format:'%H').split(/\n/) + repo = Git.bare("/tmp/#{repo_dir}") + repo_commits = repo.lib.log_commits check_commit_exist(commit_id, repo_commits) # get the auther's email for the commit - author_email = %x(/usr/bin/git -C /tmp/#{repo_dir} show -s --format=%aE #{commit_id}).chomp + author_email = repo.gcommit(commit_id).author.email check_commit_email(author_email) FileUtils.rm_rf "/tmp/#{repo_dir}" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container/mail-robot: add use ENV variable
by Luan Shengde 24 Mar '21

24 Mar '21

use ENV variable instead of full-path string enable modify the path without re-edit the code. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 2 +- container/mail-robot/start | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 17705ec..97c3989 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -56,7 +56,7 @@ class ApplyAccount @es_port = ES_PORT # email info file for account issuers. - @account_issuer = YAML.load_file '/etc/compass-ci/account-issuer.yaml' + @account_issuer = YAML.load_file ENV['ACCOUNT_ISSUER'] @my_info = {} end diff --git a/container/mail-robot/start b/container/mail-robot/start index bd2a62d..ce18498 100755 --- a/container/mail-robot/start +++ b/container/mail-robot/start @@ -16,6 +16,7 @@ names = Set.new %w[ SCHED_HTTPS_PORT ES_PORT MAILDIR + ACCOUNT_ISSUER lab ] -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/mail-robot: add install ruby-git
by Luan Shengde 24 Mar '21

24 Mar '21

add install ruby-git in dockerfile Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile index e4b3e7c..41056a5 100644 --- a/container/mail-robot/Dockerfile +++ b/container/mail-robot/Dockerfile @@ -10,7 +10,7 @@ ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y git uuid-runtime ruby-listen ruby-json ruby-mail ruby-rest-client + apt-get install -y git uuid-runtime ruby-listen ruby-json ruby-mail ruby-rest-client ruby-git RUN gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ -- 2.23.0

1 0

[PATCH compass-ci] src/monitor: detect crash and send msg to mq queue.
by Li Ping 23 Mar '21

23 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/monitoring/filter.cr | 2 +- src/monitoring/parse_serial_logs.cr | 50 ++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 39cfbb6..b663f11 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,7 +56,7 @@ class Filter msg = JSON.parse(msg.to_s).as_h? return unless msg - @sp.save_dmesg_to_result_root(msg) + @sp.deal_serial_log(msg) @hash.keys.each do |query| if match_query(query.as_h, msg) send_msg(query, msg) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 0589523..9430d8c 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -4,6 +4,7 @@ require "../scheduler/elasticsearch_client" require "set" require "json" +require "../lib/mq" # This parses dmesg in a stream of serial log, finding a number of patterns # in various places of the dmesg and take actions accordingly. @@ -40,9 +41,15 @@ class SerialParser "Restarting system", ] + CRASH_PATTERNS = [ + "mount.nfs: Connection timed out", + "No space left on device", + ] + def initialize @host2head = Hash(String, Array(String)).new @host2rt = Hash(String, String).new + @mq = MQClient.instance end def host_in_msg(msg) @@ -59,6 +66,14 @@ class SerialParser end end + def detect_crash(msg, host) + message = msg["message"].to_s + CRASH_PATTERNS.each do |pattern| + matched = message.match(/.*(?<crash>#{pattern})/) + return matched.named_captures["crash"] unless matched.nil? + end + end + def delete_host(msg, host, signal) boundary_signal = detect_start_or_end(msg, host, signal) return unless boundary_signal @@ -67,10 +82,43 @@ class SerialParser @host2rt.delete(host) end - def save_dmesg_to_result_root(msg) + def mq_publish(msg, host) + crash_signal = detect_crash(msg, host) + return unless crash_signal + + job_id = "" + if @host2rt.has_key?(host) + job_id = File.basename(@host2rt[host]) + end + + mq_msg = { + "job_id" => job_id, + "testbox" => host, + "time" => msg["time"]? || Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800"), + "job_state" => "crash" + } + spawn mq_publish_check("job_mq", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def deal_serial_log(msg) host = host_in_msg(msg) return unless host + mq_publish(msg, host) + save_dmesg_to_result_root(msg, host) + end + + def save_dmesg_to_result_root(msg, host) delete_host(msg, host, START_PATTERNS) check_save = check_save_dmesg(msg, host) -- 2.23.0

1 0

[PATCH compass-ci] src/lib: add on_job_crash
by Li Ping 23 Mar '21

23 Mar '21

[how] mq publish the crash event, lifecycle will deal the job event. when job_state is crash and the job is consumed, then close the job and reboot crash machine, if no job, just reboot crash machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/lifecycle.cr | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 4bd2ac0..8f34903 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -122,6 +122,8 @@ class Lifecycle on_job_close(event) when "abnormal" on_abnormal_job(event) + when "crash" + on_job_crash(event) else on_other_job(event) end @@ -190,6 +192,17 @@ class Lifecycle update_cached_machine(job["testbox"].to_s, event) end + def on_job_crash(event) + event_job_id = event["job_id"].to_s + if @jobs[event_job_id]? + @jobs.delete(event_job_id) + spawn @scheduler_api.close_job(event_job_id, "crash", "lifecycle") + end + + testbox = event["testbox"].to_s + reboot_crash_machine(testbox, event) + end + def on_job_boot(event) event_job_id = event["job_id"]?.to_s @jobs[event_job_id] = event unless event_job_id.empty? @@ -298,6 +311,22 @@ class Lifecycle spawn @scheduler_api.close_job(job_id, reason, "lifecycle") end + def reboot_crash_machine(testbox, event) + @machines.delete(testbox) + machine = @es.get_tbox(testbox) + return unless machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) + + mq_queue = get_machine_reboot_queue(testbox) + machine.as_h.delete("history") + machine.as_h["testbox"] = JSON::Any.new(testbox) + @mq.publish_confirm(mq_queue, machine.to_json, durable: true) + + machine["state"] = "rebooting_queue" + machine["time"] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") + @es.update_tbox(testbox, machine.as_h) + end + def reboot_timeout_machine(testbox) @machines.delete(testbox) machine = @es.get_tbox(testbox) -- 2.23.0

1 0

[PATCH lab-z9 3/3] hosts: taishan200-2280-2s48p-256g--a65
by Wei Jihui 23 Mar '21

23 Mar '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a65 | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a65 diff --git a/hosts/taishan200-2280-2s48p-256g--a65 b/hosts/taishan200-2280-2s48p-256g--a65 new file mode 100644 index 0000000..341303c --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a65 @@ -0,0 +1,21 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88b4509 + - /dev/disk/by-id/scsi-350000399b88b30dd + - /dev/disk/by-id/scsi-350000399b88b4621 + - /dev/disk/by-id/scsi-350000399b88b4425 +mac_addr: + - 44:67:47:c9:e9:90 + - 44:67:47:c9:e9:91 + - 44:67:47:c9:e9:92 + - 44:67:47:c9:e9:93 + - 44:67:47:c9:e9:94 + - 44:67:47:c9:e9:95 + - 44:67:47:c9:e9:96 + - 44:67:47:c9:e9:97 +model_name: Kunpeng-920 +ipmi_ip: 9.3.4.6 -- 2.23.0

1 0

[PATCH lab-z9 2/3] hosts: add taishan200-2280-2s48p-256g--a64
by Wei Jihui 23 Mar '21

23 Mar '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a64 | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a64 diff --git a/hosts/taishan200-2280-2s48p-256g--a64 b/hosts/taishan200-2280-2s48p-256g--a64 new file mode 100644 index 0000000..f8aa111 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a64 @@ -0,0 +1,21 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399b88b2fa1 + - /dev/disk/by-id/scsi-350000399b88b4339 + - /dev/disk/by-id/scsi-350000399b88b4329 + - /dev/disk/by-id/scsi-350000399b88aefc1 +mac_addr: + - 44:67:47:c9:ea:b2 + - 44:67:47:c9:ea:b3 + - 44:67:47:c9:ea:b4 + - 44:67:47:c9:ea:b5 + - 44:67:47:c9:ea:b6 + - 44:67:47:c9:ea:b7 + - 44:67:47:c9:ea:b8 + - 44:67:47:c9:ea:b9 +model_name: Kunpeng-920 +ipmi_ip: 9.3.4.5 -- 2.23.0

1 0

[PATCH lab-z9 1/3] hosts: add taishan200-2280-2s48p-256g--a63
by Wei Jihui 23 Mar '21

23 Mar '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a63 | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a63 diff --git a/hosts/taishan200-2280-2s48p-256g--a63 b/hosts/taishan200-2280-2s48p-256g--a63 new file mode 100644 index 0000000..a8e6688 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a63 @@ -0,0 +1,21 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions: 4 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b0431 + - /dev/disk/by-id/scsi-350000399886adc71 + - /dev/disk/by-id/scsi-350000399886b66f5 + - /dev/disk/by-id/scsi-350000399886b6b75 +mac_addr: + - 44:67:47:c9:e5:44 + - 44:67:47:c9:e5:45 + - 44:67:47:c9:e5:46 + - 44:67:47:c9:e5:47 + - 44:67:47:c9:e5:48 + - 44:67:47:c9:e5:49 + - 44:67:47:c9:e5:4a + - 44:67:47:c9:e5:4b +model_name: Kunpeng-920 +ipmi_ip: 9.3.4.4 -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: change the way to calculate priority for repo
by Li Yuanchao 23 Mar '21

23 Mar '21

Change the way to calculate priority for repo, to make active repo be updated more times and more effective. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 50a6c75..5ba9154 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -115,7 +115,6 @@ class MirrorMain def initialize @feedback_queue = Queue.new @fork_stat = {} - @priority = 0 @priority_queue = PriorityQueue.new @git_info = {} @defaults = {} @@ -221,11 +220,9 @@ class MirrorMain def push_git_queue return if @git_queue.size >= 1 - fork_key = @priority_queue.delete_min_return_key + fork_key, old_pri = @priority_queue.delete_min do_push(fork_key) - priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 - @priority_queue.push fork_key, priority_set - @priority += 1 + @priority_queue.push fork_key, get_repo_priority(fork_key, old_pri) end def main_loop @@ -251,8 +248,7 @@ class MirrorMain @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo], belong) fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end def compare_refs(cur_refs, old_refs) @@ -388,8 +384,7 @@ class MirrorMain @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true, 'belong' => belong } fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end end @@ -474,6 +469,8 @@ end # main thread class MirrorMain + WEEK_SECONDS = 604800 + def merge_defaults(object_key, object, belong) return object if object_key == belong @@ -541,4 +538,25 @@ class MirrorMain return true end + + def get_repo_priority(git_repo, old_pri) + old_pri ||= 0 + mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}" + mirror_dir = "#{mirror_dir}.git" unless @git_info[git_repo]['is_submodule'] + + return old_pri + Math.cbrt(WEEK_SECONDS) unless File.directory?(mirror_dir) + + return cal_priority(mirror_dir, old_pri) + end + + def cal_priority(mirror_dir, old_pri) + last_commit_time = %x(git -C #{mirror_dir} log --pretty=format:"%ct" -1 2>/dev/null).to_i + return old_pri + Math.cbrt(WEEK_SECONDS) if last_commit_time.zero? + + t = Time.now.to_i + interval = t - last_commit_time + return old_pri + Math.cbrt(WEEK_SECONDS) if interval <= 0 + + return old_pri + Math.cbrt(interval) + end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] service/sshr: change port
by Wu Zhende 23 Mar '21

23 Mar '21

Restarting the sshr service will disconnect the existing sshr connection and the machine that has been applied for can't be accessed. Therefore, we can't restart the sshr service. Instead, we start a new sshr service and use a new port. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 2 +- container/scheduler/start | 4 ++-- container/ssh-r/start | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index ee93e79..4ec84a2 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -38,7 +38,7 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT'] ||= 5051 defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' diff --git a/container/scheduler/start b/container/scheduler/start index e655dd4..8d800f9 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -31,8 +31,8 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 -defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT'] ||= 5051 +defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' diff --git a/container/ssh-r/start b/container/ssh-r/start index 6c7a373..68a63f2 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -16,7 +16,7 @@ cmd=( -e TCP_FORWARDING=true -d -p 21000-23999:21000-23999 - -p 5050:22 + -p 5051:22 -v /etc/localtime:/etc/localtime:ro -v /srv/pub/sshr/keys/:/etc/ssh/keys ssh-r:0.001 -- 2.23.0

2 1