- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/ssh-r/start: rename docker name
by Wei Jihui 30 Mar '21

30 Mar '21

vm_name --> docker_name Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/ssh-r/start | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/ssh-r/start b/container/ssh-r/start index 68a63f2..17d4493 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -4,14 +4,14 @@ . $CCI_SRC/container/defconfig.sh -vm_name=ssh_r +docker_name=ssh_r -docker_rm ${vm_name} +#docker_rm ${docker_name} cmd=( docker run --restart=always - --name ${vm_name} + --name ${docker_name} -e GATEWAY_PORTS=true -e TCP_FORWARDING=true -d -- 2.23.0

2 1

[PATCH compass-ci] update container expose port
by Cui Yili 30 Mar '21

30 Mar '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/minio/start | 2 +- container/openresty-proxy-cache/start | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/container/minio/start b/container/minio/start index 1798cfa..2cbc9c8 100755 --- a/container/minio/start +++ b/container/minio/start @@ -7,7 +7,7 @@ docker_rm minio cmd=( docker run -d - -p 11321:9000 + -p 20014:9000 --name minio -v /etc/localtime:/etc/localtime:ro -v /srv/minio/:/srv/minio/ diff --git a/container/openresty-proxy-cache/start b/container/openresty-proxy-cache/start index a1f095d..59e2a28 100755 --- a/container/openresty-proxy-cache/start +++ b/container/openresty-proxy-cache/start @@ -11,8 +11,8 @@ cmd=( docker run -d --name openresty-proxy-cache --restart=always - -p 11333:80 - -p 11334:443 + -p 20015:80 + -p 20016:443 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf -v /srv/initrd/tar:/srv/initrd/tar:ro -v /etc/localtime:/etc/localtime:ro -- 2.23.0

1 0

[PATCH compass-ci] crystal-shards/build: delete same annotation about gitcache
by Cui Yili 30 Mar '21

30 Mar '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/crystal-shards/build | 3 --- 1 file changed, 3 deletions(-) diff --git a/container/crystal-shards/build b/container/crystal-shards/build index 00153ab..872282b 100755 --- a/container/crystal-shards/build +++ b/container/crystal-shards/build @@ -2,9 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# GITCACHE_HOST and GITCACHE_PORT parameters come from gitcache_host and gitcache_port in the job.yaml file you submitted. -# If job.yaml does not have these two parameters, git will use the local proxy cache, -# GITCACHE_HOST defaults to the local ip, GITCACHE_PORT default port is 5000 . ../defconfig.sh docker_skip_rebuild "alpine:crystal-shards" -- 2.23.0

1 0

[PATCH compass-ci] container: docker run param -it => -d
by Cao Xueliang 30 Mar '21

30 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/etcd/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/etcd/start b/container/etcd/start index 7657b86..db94833 100755 --- a/container/etcd/start +++ b/container/etcd/start @@ -8,7 +8,7 @@ docker_rm etcd-server cmd=( docker run - -it + -d -p 2379:2379 -p 2380:2380 -p 4001:4001 -- 2.23.0

1 0

[PATCH compass-ci 2/4] container/rpm-repo: add build script
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/build | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/rpm-repo/build diff --git a/container/rpm-repo/build b/container/rpm-repo/build new file mode 100755 index 0000000..458e8f6 --- /dev/null +++ b/container/rpm-repo/build @@ -0,0 +1,6 @@ +#!/bin/sh + +. ../defconfig.sh +docker_skip_rebuild "fedora:update-repodata" + +docker build -t fedora:update-repodata . -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror: fix rubocop errors
by Li Yuanchao 29 Mar '21

29 Mar '21

[error msg] git_mirror.rb:112:1: C: Metrics/ClassLength: Class has too many lines. [101/100] class MirrorMain ... ^^^^^^^^^^^^^^^^ git_mirror.rb:192:3: C: Metrics/AbcSize: Assignment Branch Condition size for handle_feedback is too high. [<4, 17, 6> 18.47/15] def handle_feedback ... ^^^^^^^^^^^^^^^^^^^ git_mirror.rb:286:3: C: Metrics/AbcSize: Assignment Branch Condition size for reload_fork_info is too high. [<5, 17, 2> 17.83/15] def reload_fork_info(upstream_repos) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ git_mirror.rb:293:121: C: Layout/LineLength: Line is too long. [149/120] changed_files = %x(git -C /srv/git/#{@git_info[upstream_repos]['belong']}/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ git_mirror.rb:328:3: C: Metrics/AbcSize: Assignment Branch Condition size for update_stat_new_refs is too high. [<6, 14, 2> 15.36/15] def update_stat_new_refs(git_repo) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ git_mirror.rb:508:7: C: Naming/PredicateName: Rename is_upstream_repo? to upstream_repo?. def is_upstream_repo?(git_repo) ^^^^^^^^^^^^^^^^^ Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 45 +++++++++++++++++++++++++++++---------------- 1 file changed, 29 insertions(+), 16 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5ba9154..c967c8b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -199,14 +199,7 @@ class MirrorMain update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - return reload_fork_info(git_repo) if is_upstream_repo?(git_repo) - - new_refs = check_new_refs(git_repo) - return if new_refs[:heads].empty? - - feedback_info[:new_refs] = new_refs - send_message(feedback_info) - new_refs_log(git_repo, new_refs[:heads].length) if last_commit_new?(git_repo) + handle_feedback_new_refs(git_repo, feedback_info) end def do_push(fork_key) @@ -283,14 +276,19 @@ class MirrorMain return new_refs end + def get_change_files(upstream_repos) + old_commit = @git_info[upstream_repos][:cur_refs][:heads]['refs/heads/master'] + new_refs = check_new_refs(upstream_repos) + new_commit = new_refs[:heads]['refs/heads/master'] + mirror_dir = "/srv/git/#{@git_info[upstream_repos]['belong']}/#{upstream_repos}.git" + %x(git -C #{mirror_dir} diff --name-only #{old_commit}...#{new_commit}) + end + def reload_fork_info(upstream_repos) if @git_info[upstream_repos][:cur_refs].empty? @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) else - old_commit = @git_info[upstream_repos][:cur_refs][:heads]['refs/heads/master'] - new_refs = check_new_refs(upstream_repos) - new_commit = new_refs[:heads]['refs/heads/master'] - changed_files = %x(git -C /srv/git/#{@git_info[upstream_repos]['belong']}/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) + changed_files = get_change_files(upstream_repos) reload(changed_files, @git_info[upstream_repos]['belong']) end end @@ -325,13 +323,17 @@ class MirrorMain @fork_stat[git_repo][:offset_fetch] = offset_fetch + 1 end + def update_new_refs_info(git_repo, offset_new_refs) + @fork_stat[git_repo][:new_refs_time][offset_new_refs] = Time.now.to_s + @fork_stat[git_repo][:offset_new_refs] = offset_new_refs + 1 + @fork_stat[git_repo][:new_refs_count] = update_new_refs_count(@fork_stat[git_repo][:new_refs_count]) + end + def update_stat_new_refs(git_repo) @fork_stat[git_repo][:priority] += 1 offset_new_refs = @fork_stat[git_repo][:offset_new_refs] offset_new_refs = 0 if offset_new_refs >= 10 - @fork_stat[git_repo][:new_refs_time][offset_new_refs] = Time.now.to_s - @fork_stat[git_repo][:offset_new_refs] = offset_new_refs + 1 - @fork_stat[git_repo][:new_refs_count] = update_new_refs_count(@fork_stat[git_repo][:new_refs_count]) + update_new_refs_info(git_repo, offset_new_refs) end def update_fork_stat(git_repo, possible_new_refs) @@ -471,6 +473,17 @@ end class MirrorMain WEEK_SECONDS = 604800 + def handle_feedback_new_refs(git_repo, feedback_info) + return reload_fork_info(git_repo) if upstream_repo?(git_repo) + + new_refs = check_new_refs(git_repo) + return if new_refs[:heads].empty? + + feedback_info[:new_refs] = new_refs + send_message(feedback_info) + new_refs_log(git_repo, new_refs[:heads].length) if last_commit_new?(git_repo) + end + def merge_defaults(object_key, object, belong) return object if object_key == belong @@ -505,7 +518,7 @@ class MirrorMain return url end - def is_upstream_repo?(git_repo) + def upstream_repo?(git_repo) @upstreams['upstreams'].each do |repo| return true if git_repo == repo['git_repo'] end -- 2.23.0

2 2

[PATCH compass-ci] container: add etcd to start-depends
by Cao Xueliang 29 Mar '21

29 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/extract-stats/start-depends | 1 + container/scheduler/start-depends | 1 + 2 files changed, 2 insertions(+) diff --git a/container/extract-stats/start-depends b/container/extract-stats/start-depends index efbe5a2..abd487d 100755 --- a/container/extract-stats/start-depends +++ b/container/extract-stats/start-depends @@ -1 +1,2 @@ taskqueue +etcd diff --git a/container/scheduler/start-depends b/container/scheduler/start-depends index 1aced31..09a0b1d 100755 --- a/container/scheduler/start-depends +++ b/container/scheduler/start-depends @@ -2,3 +2,4 @@ redis es taskqueue rabbitmq +etcd -- 2.23.0

1 0

[PATCH compass-ci] src/monitor: detect crash and send msg to mq queue.
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/monitoring/filter.cr | 2 +- src/monitoring/parse_serial_logs.cr | 50 ++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 39cfbb6..b663f11 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,7 +56,7 @@ class Filter msg = JSON.parse(msg.to_s).as_h? return unless msg - @sp.save_dmesg_to_result_root(msg) + @sp.deal_serial_log(msg) @hash.keys.each do |query| if match_query(query.as_h, msg) send_msg(query, msg) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 0589523..9430d8c 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -4,6 +4,7 @@ require "../scheduler/elasticsearch_client" require "set" require "json" +require "../lib/mq" # This parses dmesg in a stream of serial log, finding a number of patterns # in various places of the dmesg and take actions accordingly. @@ -40,9 +41,15 @@ class SerialParser "Restarting system", ] + CRASH_PATTERNS = [ + "mount.nfs: Connection timed out", + "No space left on device", + ] + def initialize @host2head = Hash(String, Array(String)).new @host2rt = Hash(String, String).new + @mq = MQClient.instance end def host_in_msg(msg) @@ -59,6 +66,14 @@ class SerialParser end end + def detect_crash(msg, host) + message = msg["message"].to_s + CRASH_PATTERNS.each do |pattern| + matched = message.match(/.*(?<crash>#{pattern})/) + return matched.named_captures["crash"] unless matched.nil? + end + end + def delete_host(msg, host, signal) boundary_signal = detect_start_or_end(msg, host, signal) return unless boundary_signal @@ -67,10 +82,43 @@ class SerialParser @host2rt.delete(host) end - def save_dmesg_to_result_root(msg) + def mq_publish(msg, host) + crash_signal = detect_crash(msg, host) + return unless crash_signal + + job_id = "" + if @host2rt.has_key?(host) + job_id = File.basename(@host2rt[host]) + end + + mq_msg = { + "job_id" => job_id, + "testbox" => host, + "time" => msg["time"]? || Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800"), + "job_state" => "crash" + } + spawn mq_publish_check("job_mq", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def deal_serial_log(msg) host = host_in_msg(msg) return unless host + mq_publish(msg, host) + save_dmesg_to_result_root(msg, host) + end + + def save_dmesg_to_result_root(msg, host) delete_host(msg, host, START_PATTERNS) check_save = check_save_dmesg(msg, host) -- 2.23.0

1 0

[PATCH compass-ci] src/lib: add on_job_crash
by Li Ping 29 Mar '21

29 Mar '21

[how] mq publish the crash event, lifecycle will deal the job event. when job_state is crash and the job is consumed, then close the job and reboot crash machine, if no job, just reboot crash machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/lifecycle.cr | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 4bd2ac0..8f34903 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -122,6 +122,8 @@ class Lifecycle on_job_close(event) when "abnormal" on_abnormal_job(event) + when "crash" + on_job_crash(event) else on_other_job(event) end @@ -190,6 +192,17 @@ class Lifecycle update_cached_machine(job["testbox"].to_s, event) end + def on_job_crash(event) + event_job_id = event["job_id"].to_s + if @jobs[event_job_id]? + @jobs.delete(event_job_id) + spawn @scheduler_api.close_job(event_job_id, "crash", "lifecycle") + end + + testbox = event["testbox"].to_s + reboot_crash_machine(testbox, event) + end + def on_job_boot(event) event_job_id = event["job_id"]?.to_s @jobs[event_job_id] = event unless event_job_id.empty? @@ -298,6 +311,22 @@ class Lifecycle spawn @scheduler_api.close_job(job_id, reason, "lifecycle") end + def reboot_crash_machine(testbox, event) + @machines.delete(testbox) + machine = @es.get_tbox(testbox) + return unless machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) + + mq_queue = get_machine_reboot_queue(testbox) + machine.as_h.delete("history") + machine.as_h["testbox"] = JSON::Any.new(testbox) + @mq.publish_confirm(mq_queue, machine.to_json, durable: true) + + machine["state"] = "rebooting_queue" + machine["time"] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") + @es.update_tbox(testbox, machine.as_h) + end + def reboot_timeout_machine(testbox) @machines.delete(testbox) machine = @es.get_tbox(testbox) -- 2.23.0

1 0

[PATCH compass-ci 4/5] container/updaterepo: add build script
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/updaterepo/build | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 container/updaterepo/build diff --git a/container/updaterepo/build b/container/updaterepo/build new file mode 100755 index 0000000..594558a --- /dev/null +++ b/container/updaterepo/build @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_skip_rebuild "update-repo" + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t update-repo ." + +system 'rm updaterepo' -- 2.23.0

1 0