- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 2/2] src/monitor: detect crash and send msg to mq queue.
by Li Ping 30 Mar '21

30 Mar '21

linux/kernel/panic.c: pr_emerg("Kernel panic - not syncing: %s\n", buf); When kernel panic happens, it will print the panic call stack, then the ending msg like: [ 14.405869] Kernel panic - not syncing: Fatal exception in interrupt some bug not cause kernel panic happens but also mean that job fails like: mount.nfs: Connection timed out No space left on device The above message would be detected, then send msg to mq queue to close job and reboot machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/monitoring/filter.cr | 2 +- src/monitoring/parse_serial_logs.cr | 47 +++++++++++++++++++++++++++-- 2 files changed, 45 insertions(+), 4 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 39cfbb6..b663f11 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,7 +56,7 @@ class Filter msg = JSON.parse(msg.to_s).as_h? return unless msg - @sp.save_dmesg_to_result_root(msg) + @sp.deal_serial_log(msg) @hash.keys.each do |query| if match_query(query.as_h, msg) send_msg(query, msg) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 451312c..ff3b84a 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -4,6 +4,7 @@ require "../scheduler/elasticsearch_client" require "set" require "json" +require "../lib/mq" # This parses dmesg in a stream of serial log, finding a number of patterns # in various places of the dmesg and take actions accordingly. @@ -40,9 +41,16 @@ class SerialParser "Restarting system", ] + CRASH_PATTERNS = [ + "mount.nfs: Connection timed out", + "No space left on device", + "Kernel panic - not syncing: ", + ] + def initialize @host2head = Hash(String, Array(String)).new @host2rt = Hash(String, String).new + @mq = MQClient.instance end def host_in_msg(msg) @@ -51,7 +59,7 @@ class SerialParser File.basename(msg["serial_path"].to_s) end - def detect_start_or_end(msg, pattern_list) + def detect_patterns(msg, pattern_list) message = msg["message"].to_s pattern_list.each do |pattern| matched = message.match(/.*(?<signal>#{pattern})/) @@ -60,17 +68,50 @@ class SerialParser end def delete_host(msg, host, signal) - boundary_signal = detect_start_or_end(msg, signal) + boundary_signal = detect_patterns(msg, signal) return unless boundary_signal @host2head.delete(host) @host2rt.delete(host) end - def save_dmesg_to_result_root(msg) + def mq_publish(msg, host) + crash_signal = detect_patterns(msg, CRASH_PATTERNS) + return unless crash_signal + + job_id = "" + if @host2rt.has_key?(host) + job_id = File.basename(@host2rt[host]) + end + + mq_msg = { + "job_id" => job_id, + "testbox" => host, + "time" => msg["time"]? || Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800"), + "job_state" => "crash" + } + spawn mq_publish_check("job_mq", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def deal_serial_log(msg) host = host_in_msg(msg) return unless host + mq_publish(msg, host) + save_dmesg_to_result_root(msg, host) + end + + def save_dmesg_to_result_root(msg, host) delete_host(msg, host, START_PATTERNS) check_save = check_save_dmesg(msg, host) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] src/lib: add on_job_crash
by Li Ping 30 Mar '21

30 Mar '21

[how] mq publish the crash event, lifecycle will deal the job event. when job_state is crash and the job is consumed, then close the job and reboot crash machine, if no job, just reboot crash machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/lifecycle.cr | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index adcb965..0e6cb82 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -122,6 +122,8 @@ class Lifecycle on_job_close(event) when "abnormal" on_abnormal_job(event) + when "crash" + on_job_crash(event) else on_other_job(event) end @@ -189,6 +191,16 @@ class Lifecycle update_cached_machine(job["testbox"].to_s, event) end + def on_job_crash(event) + event_job_id = event["job_id"].to_s + if @jobs[event_job_id]? + close_job(event_job_id, "crash") + end + + testbox = event["testbox"].to_s + reboot_crash_machine(testbox, event) + end + def on_job_boot(event) event_job_id = event["job_id"]?.to_s @jobs[event_job_id] = event unless event_job_id.empty? @@ -296,6 +308,15 @@ class Lifecycle spawn @scheduler_api.close_job(job_id, reason, "lifecycle") end + def reboot_crash_machine(testbox, event) + @machines.delete(testbox) + machine = @es.get_tbox(testbox) + return unless machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) + + reboot_machine(testbox, machine) + end + def reboot_timeout_machine(testbox) @machines.delete(testbox) machine = @es.get_tbox(testbox) @@ -308,6 +329,10 @@ class Lifecycle deadline = Time.parse(deadline.to_s, "%Y-%m-%dT%H:%M:%S", Time.local.location) return if Time.local < deadline + reboot_machine(testbox, machine) + end + + def reboot_machine(testbox, machine) mq_queue = get_machine_reboot_queue(testbox) machine.as_h.delete("history") machine.as_h["testbox"] = JSON::Any.new(testbox) -- 2.23.0

1 0

[PATCH compass-ci] etcd/Dockerfile: change source address
by Wang Yong 30 Mar '21

30 Mar '21

accelerate download Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- container/etcd/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/etcd/Dockerfile b/container/etcd/Dockerfile index 3654fc4..b3957e8 100644 --- a/container/etcd/Dockerfile +++ b/container/etcd/Dockerfile @@ -8,7 +8,7 @@ MAINTAINER Cao Xueliang <caoxl78320(a)163.com> ARG VERSION RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && apk add ca-certificates && \ - wget https://github.com/etcd-io/etcd/releases/download/${VERSION}/etcd-${VERSION… && \ + wget https://repo.huaweicloud.com/etcd/${VERSION}/etcd-${VERSION}-linux-arm64.ta… && \ tar zxvf etcd-${VERSION}-linux-arm64.tar.gz && \ mv etcd-${VERSION}-linux-arm64/etcd* /bin/ && \ rm -Rf etcd-${VERSION}-linux-arm64* /var/cache/apk/* -- 2.23.0

1 0

[PATCH v2 compass-ci] container/ssh-r/start: rename var for vm_name to docker_name
by Wei Jihui 30 Mar '21

30 Mar '21

vm_name --> docker_name Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/ssh-r/start | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/ssh-r/start b/container/ssh-r/start index 68a63f2..47a29eb 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -4,14 +4,14 @@ . $CCI_SRC/container/defconfig.sh -vm_name=ssh_r +docker_name=ssh_r -docker_rm ${vm_name} +docker_rm ${docker_name} cmd=( docker run --restart=always - --name ${vm_name} + --name ${docker_name} -e GATEWAY_PORTS=true -e TCP_FORWARDING=true -d -- 2.23.0

2 1

[PATCH compass-ci] container/taskqueue: compiling code every time
by Wu Zhende 30 Mar '21

30 Mar '21

Delete the complied taskqueue file. Otherwise, the code will be updated and the taskqueue file will not be recomplied. Uses old code all the time. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/taskqueue/build | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/container/taskqueue/build b/container/taskqueue/build index 4f69f85..518a27e 100755 --- a/container/taskqueue/build +++ b/container/taskqueue/build @@ -12,10 +12,8 @@ service_port=${TASKQUEUE_PORT:-3060} docker_skip_rebuild "taskqueue-alpine:v0.001-${service_port}" -if [[ ! -f "taskqueue" ]]; then - $CCI_SRC/container/compile . -else - echo -e "\n\033[34m- Info: using \033[31mexists\033[34m taskqueue -\033[0m\n" -fi +$CCI_SRC/container/compile . docker build -t taskqueue-alpine:v0.001-${service_port} . + +rm taskqueue -- 2.23.0

1 0

[PATCH compass-ci] monitoring/parse_serial_logs.cr: fix useless arg
by Wei Jihui 30 Mar '21

30 Mar '21

host is not used at detect_start_or_end Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/monitoring/parse_serial_logs.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 0589523..451312c 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -51,7 +51,7 @@ class SerialParser File.basename(msg["serial_path"].to_s) end - def detect_start_or_end(msg, host, pattern_list) + def detect_start_or_end(msg, pattern_list) message = msg["message"].to_s pattern_list.each do |pattern| matched = message.match(/.*(?<signal>#{pattern})/) @@ -60,7 +60,7 @@ class SerialParser end def delete_host(msg, host, signal) - boundary_signal = detect_start_or_end(msg, host, signal) + boundary_signal = detect_start_or_end(msg, signal) return unless boundary_signal @host2head.delete(host) -- 2.23.0

1 0

[PATCH lkp-tests] monitors/kmsg: not execute dmesg when in docker
by Lin Jiaxin 30 Mar '21

30 Mar '21

In order to avoid the following stderr: ==> /tmp/stderr <== dmesg: read kernel buffer failed: Operation not permitted Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- monitors/kmsg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/monitors/kmsg b/monitors/kmsg index 4edd90a9..a1cf7121 100755 --- a/monitors/kmsg +++ b/monitors/kmsg @@ -6,6 +6,8 @@ # serial console has better availability. is_virt && [ -z "$LKP_LOCAL_RUN" ] && exec > /dev/ttyS1 +[ -f /.dockerenv ] && exit 0 + if dmesg --help 2>&1 | grep -q -- --follow; then exec dmesg --follow --decode else -- 2.23.0

2 1

[PATCH compass-ci] src/lib/lifecycle.cr: use close_job close job
by Wei Jihui 30 Mar '21

30 Mar '21

unified use close_job to close job, reduce code Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/lib/lifecycle.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 4bd2ac0..adcb965 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -177,8 +177,7 @@ class Lifecycle event_job_id = event["job_id"].to_s return unless @jobs.has_key?(event_job_id) - @jobs.delete(event_job_id) - spawn @scheduler_api.close_job(event_job_id, "abnormal", "lifecycle") + close_job(event_job_id, "abnormal") end def on_job_close(event) @@ -222,8 +221,7 @@ class Lifecycle return if machine_job_id.empty? return unless @jobs.has_key?(machine_job_id) - @jobs.delete(machine_job_id) - spawn @scheduler_api.close_job(machine_job_id, "abnormal", "lifecycle") + close_job(machine_job_id, "abnormal") end def max_time(times) -- 2.23.0

1 0

[PATCH compass-ci 3/4] container/rpm-repo: add rpm-repo.rb for update repodata
by Li Ping 30 Mar '21

30 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/rpm-repo.rb | 73 ++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 container/rpm-repo/rpm-repo.rb diff --git a/container/rpm-repo/rpm-repo.rb b/container/rpm-repo/rpm-repo.rb new file mode 100755 index 0000000..18f0507 --- /dev/null +++ b/container/rpm-repo/rpm-repo.rb @@ -0,0 +1,73 @@ +#!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require "bunny" +require "fileutils" +require "json" + + +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || '172.17.0.1' +MQ_PORT = ENV['MQ_PORT'] || 5672 + +class MQClient + def initialize(hostname = "172.17.0.1", port = "5672") + @conn = Bunny.new(hostname: hostname, port: port) + @conn.start + @channel = @conn.create_channel + end + + def queue(queue_name, opts = {}) + @channel.queue(queue_name, opts) + end + + def ack(delivery_info) + @channel.ack(delivery_info.delivery_tag) + end +end + +class HandleRepo + def initialize + @mq = MQClient.new(MQ_HOST, MQ_PORT) + @update = [] + end + + def handle_new_rpm + queue = @mq.queue("update_repo") + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + rpm_info = JSON.parse(msg) + puts rpm_info + rpm_info["upload_rpms"].each do |rpm| + rpm_path = File.dirname(rpm).sub("upload", "testing") + FileUtils.mkdir_p(rpm_path) unless File.directory?(rpm_path) + + dest = File.join(rpm_path.to_s, File.basename(rpm)) + @update << dest + FileUtils.mv(rpm, dest) + system("createrepo --update $(dirname #{rpm_path})") + end + update_pub_dir + @mq.ack(info) + end + end + + def update_pub_dir + @update.each do |rpm| + pub_path = File.dirname(rpm).sub("testing", "pub") + FileUtils.mkdir_p(pub_path) unless File.directory?(pub_path) + + dest = File.join(pub_path, File.basename(rpm)) + FileUtils.cp(rpm, dest) + + repodata_dest = File.join(File.dirname(pub_path), "repodata") + repodata_src = File.dirname(rpm).sub("Packages", "repodata") + + FileUtils.rm_r(repodata_dest) if Dir.exist?(repodata_dest) + FileUtils.cp_r(repodata_src, File.dirname(repodata_dest)) + end + end +end + +hr = HandleRepo.new +hr.handle_new_rpm -- 2.23.0

4 5

[PATCH compass-ci 5/5] container/updaterepo: add update-repo start script
by Li Ping 30 Mar '21

30 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/updaterepo/start | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100755 container/updaterepo/start diff --git a/container/updaterepo/start b/container/updaterepo/start new file mode 100755 index 0000000..46e3ad6 --- /dev/null +++ b/container/updaterepo/start @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "update-repo" + +cmd = %W[ + docker run + --restart=always + --name update-repo + -u 1090:1090 + -p 8132:8132 + -d + -v /srv/rpm/:/srv/rpm/ + -v /etc/localtime:/etc/localtime:ro + update-repo +] + +cmd += ['sh', '-c', 'umask 002 && ./updaterepo'] + +system(*cmd) -- 2.23.0

2 2

[PATCH compass-ci 2/5] src/lib/updaterepo.cr: send messages to update-repo queue
by Li Ping 30 Mar '21

30 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/updaterepo.cr | 60 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 src/lib/updaterepo.cr diff --git a/src/lib/updaterepo.cr b/src/lib/updaterepo.cr new file mode 100644 index 0000000..4826fac --- /dev/null +++ b/src/lib/updaterepo.cr @@ -0,0 +1,60 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "json" +require "kemal" +require "./json_logger" +require "./mq" + +class Repo + def initialize(env : HTTP::Server::Context) + @mq = MQClient.instance + @env = env + @log = JSONLogger.new + end + + def upload_repo + begin + body = @env.request.body.not_nil!.gets_to_end + data = JSON.parse(body.to_s).as_h? + rescue e + @log.error(e) + end + + puts "-" * 50 + puts "post body:", data + + begin + # check if the parameters are complete + check_params_complete(data) + mq_publish(data) + rescue e + response = { "errcode" => "101", "errmsg" => "upload rpm failed" } + @log.info(response.to_json) + puts "error message: #{e.message}" + end + end + + def mq_publish(data) + mq_msg = data + puts mq_msg.to_json + + spawn mq_publish_check("update_repo", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def check_params_complete(params) + params = params.not_nil! + tmp_hash = {"errcode" => "101", "errmsg" => "no upload_rpms params"}.to_json + raise tmp_hash unless params["upload_rpms"]? + end +end -- 2.23.0

2 2

[PATCH v3 compass-ci 3/3] sparrow/3-code/dev-env: add key EMAIL_LIMIT_COUNT
by Luan Shengde 30 Mar '21

30 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 27258fc..7327083 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -12,6 +12,7 @@ server_name=$(hostname | cut -f1 -d.) : ${OS_HTTP_HOST:=$server_ip} : ${send_mail_host:=$server_ip} : ${SEND_MAIL_PORT:=10001} +: ${EMAIL_LIMIT_COUNT:=10} : ${srv_http_result_host:=$server_ip} : ${srv_http_os_host:=$server_ip} : ${srv_http_git_host:=$server_ip} @@ -90,6 +91,7 @@ SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT TASKQUEUE_HOST: $sched_host SEND_MAIL_HOST: $send_mail_host SEND_MAIL_PORT: $SEND_MAIL_PORT +EMAIL_LIMIT_COUNT: $EMAIL_LIMIT_COUNT LOGGING_ES_HOST: $LOGGING_ES_HOST LOGGING_ES_PORT: $LOGGING_ES_PORT INITRD_HTTP_HOST: $initrd_http_host -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] lib/mail-post/email_limit_queue: use ENV instead of static value
by Luan Shengde 30 Mar '21

30 Mar '21

use ENV variable instead of static value. enable user-defined new value without edit the code. just change the key and restart the service. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_limit_queue.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/mail-post/email_limit_queue.rb b/lib/mail-post/email_limit_queue.rb index c7e3954..62e9634 100755 --- a/lib/mail-post/email_limit_queue.rb +++ b/lib/mail-post/email_limit_queue.rb @@ -54,9 +54,9 @@ class EmailRateLimit end def change_queue(email) - return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + return unless (@redis.hget 'email_in_limit', email).to_i >= ENV['EMAIL_LIMIT_COUNT'].to_i @redis.hdel 'email_in_limit', email - @redis.hset 'email_out_limit', email, 10 + @redis.hset 'email_out_limit', email, ENV['EMAIL_LIMIT_COUNT'] end end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] container/send-internet-mail: add key EMAIL_LIMIT_COUNT
by Luan Shengde 30 Mar '21

30 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 136435d..57bf1f6 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -14,6 +14,7 @@ names = Set.new %w[ SEND_MAIL_PORT SENT_MAILDIR REDIS_PORT + EMAIL_LIMIT_COUNT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/3-code/dev-env: rename defaults vars
by Liu Yinsi 30 Mar '21

30 Mar '21

before: : {$A:=$server_ip}; $A=$A after: : {$a:=$server_ip}; $A=$a [why] when submit a job to run a testbox, sched will get variables(from /etc/compass-ci/scheduler), and export them in a running testbox env, but for selftest, it will deploy a new Compass CI, no need to use variables from the host(crystal or z9), it should be the testbox own server ip, so change defaults variables name. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8cb9aa5..27258fc 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -10,23 +10,23 @@ server_name=$(hostname | cut -f1 -d.) : ${ES_HOST:=$server_ip} : ${ES_PORT:=9200} : ${OS_HTTP_HOST:=$server_ip} -: ${SEND_MAIL_HOST:=$server_ip} +: ${send_mail_host:=$server_ip} : ${SEND_MAIL_PORT:=10001} -: ${SRV_HTTP_RESULT_HOST:=$server_ip} -: ${SRV_HTTP_OS_HOST:=$server_ip} -: ${SRV_HTTP_GIT_HOST:=$server_ip} -: ${SRV_HTTP_CCI_HOST:=$server_ip} +: ${srv_http_result_host:=$server_ip} +: ${srv_http_os_host:=$server_ip} +: ${srv_http_git_host:=$server_ip} +: ${srv_http_cci_host:=$server_ip} : ${SRV_HTTP_RESULT_PORT:=20007} : ${SRV_HTTP_OS_PORT:=20009} : ${SRV_HTTP_GIT_PORT:=20010} : ${SRV_HTTP_CCI_PORT:=20011} : ${LOGGING_ES_HOST:=$server_ip} : ${LOGGING_ES_PORT:=9202} -: ${INITRD_HTTP_HOST:=$server_ip} +: ${initrd_http_host:=$server_ip} : ${INITRD_HTTP_PORT:=8800} -: ${ASSIST_RESULT_HOST:=$server_ip} +: ${assist_result_host:=$server_ip} : ${ASSIST_RESULT_PORT:=8102} -: ${RESULT_WEBDAV_HOST:=$server_ip} +: ${result_webdav_host:=$server_ip} : ${RESULT_WEBDAV_PORT:=3080} : ${MASTER_FLUENTD_HOST:=$server_ip} : ${MASTER_FLUENTD_PORT:=24224} @@ -34,7 +34,7 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} -: ${GIT_SERVER:=$server_ip} +: ${git_server:=$server_ip} : ${SCHED_HTTPS_HOST:=$server_ip} : ${SCHED_HTTPS_PORT:=20002} : ${DOCKER_PROXY_HOST:=$server_ip} @@ -77,26 +77,26 @@ ES_PORT: $ES_PORT SCHED_HOST: $sched_host SCHED_PORT: $sched_port LKP_SERVER: $server_ip -GIT_SERVER: $GIT_SERVER +GIT_SERVER: $git_server OS_HTTP_HOST: $OS_HTTP_HOST -SRV_HTTP_RESULT_HOST: $SRV_HTTP_RESULT_HOST -SRV_HTTP_OS_HOST: $SRV_HTTP_OS_HOST -SRV_HTTP_GIT_HOST: $SRV_HTTP_GIT_HOST -SRV_HTTP_CCI_HOST: $SRV_HTTP_CCI_HOST +SRV_HTTP_RESULT_HOST: $srv_http_result_host +SRV_HTTP_OS_HOST: $srv_http_os_host +SRV_HTTP_GIT_HOST: $srv_http_git_host +SRV_HTTP_CCI_HOST: $srv_http_cci_host SRV_HTTP_RESULT_PORT: $SRV_HTTP_RESULT_PORT SRV_HTTP_OS_PORT: $SRV_HTTP_OS_PORT SRV_HTTP_GIT_PORT: $SRV_HTTP_GIT_PORT SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT TASKQUEUE_HOST: $sched_host -SEND_MAIL_HOST: $SEND_MAIL_HOST +SEND_MAIL_HOST: $send_mail_host SEND_MAIL_PORT: $SEND_MAIL_PORT LOGGING_ES_HOST: $LOGGING_ES_HOST LOGGING_ES_PORT: $LOGGING_ES_PORT -INITRD_HTTP_HOST: $INITRD_HTTP_HOST +INITRD_HTTP_HOST: $initrd_http_host INITRD_HTTP_PORT: $INITRD_HTTP_PORT -ASSIST_RESULT_HOST: $ASSIST_RESULT_HOST +ASSIST_RESULT_HOST: $assist_result_host ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT -RESULT_WEBDAV_HOST: $RESULT_WEBDAV_HOST +RESULT_WEBDAV_HOST: $result_webdav_host RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT -- 2.23.0

1 0

[PATCH compass-ci 1/4] container/rpm-repo: add Dockerfile
by Li Ping 30 Mar '21

30 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/rpm-repo/Dockerfile diff --git a/container/rpm-repo/Dockerfile b/container/rpm-repo/Dockerfile new file mode 100644 index 0000000..ae340e8 --- /dev/null +++ b/container/rpm-repo/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM fedora:latest + +RUN yum repolist +RUN yum install -y createrepo ruby && \ + yum clean all + +RUN umask 002 && \ + gem install bunny + +COPY --chown=1090:1090 rpm-repo.rb /usr/local/bin/ +CMD ["/usr/local/bin/rpm-repo.rb"] -- 2.23.0

2 2

[PATCH compass-ci 4/4] container/rpm-repo: add start script
by Li Ping 30 Mar '21

30 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/start | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100755 container/rpm-repo/start diff --git a/container/rpm-repo/start b/container/rpm-repo/start new file mode 100755 index 0000000..3027855 --- /dev/null +++ b/container/rpm-repo/start @@ -0,0 +1,33 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + MQ_HOST + LKP_SERVER +] + +defaults = relevant_defaults(names) +defaults['MQ_HOST'] ||= defaults['LKP_SERVER'] ||= '172.17.0.1' +defaults['LKP_SERVER'] ||= '172.17.0.1' +env = docker_env(defaults) + +docker_rm "update-repodata" + +cmd = %W[ + docker run + --restart=always + --name update-repodata + -u 1090:1090 +] + env + %W[ + -d + -v /etc/localtime:/etc/localtime:ro + -v /srv/rpm/:/srv/rpm/ + fedora:update-repodata +] + +system(*cmd) -- 2.23.0

2 2

[PATCH compass-ci] container/ssh-r/start: rename docker name
by Wei Jihui 30 Mar '21

30 Mar '21

vm_name --> docker_name Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/ssh-r/start | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/ssh-r/start b/container/ssh-r/start index 68a63f2..17d4493 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -4,14 +4,14 @@ . $CCI_SRC/container/defconfig.sh -vm_name=ssh_r +docker_name=ssh_r -docker_rm ${vm_name} +#docker_rm ${docker_name} cmd=( docker run --restart=always - --name ${vm_name} + --name ${docker_name} -e GATEWAY_PORTS=true -e TCP_FORWARDING=true -d -- 2.23.0

2 1

[PATCH compass-ci] update container expose port
by Cui Yili 30 Mar '21

30 Mar '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/minio/start | 2 +- container/openresty-proxy-cache/start | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/container/minio/start b/container/minio/start index 1798cfa..2cbc9c8 100755 --- a/container/minio/start +++ b/container/minio/start @@ -7,7 +7,7 @@ docker_rm minio cmd=( docker run -d - -p 11321:9000 + -p 20014:9000 --name minio -v /etc/localtime:/etc/localtime:ro -v /srv/minio/:/srv/minio/ diff --git a/container/openresty-proxy-cache/start b/container/openresty-proxy-cache/start index a1f095d..59e2a28 100755 --- a/container/openresty-proxy-cache/start +++ b/container/openresty-proxy-cache/start @@ -11,8 +11,8 @@ cmd=( docker run -d --name openresty-proxy-cache --restart=always - -p 11333:80 - -p 11334:443 + -p 20015:80 + -p 20016:443 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf -v /srv/initrd/tar:/srv/initrd/tar:ro -v /etc/localtime:/etc/localtime:ro -- 2.23.0

1 0

[PATCH compass-ci] crystal-shards/build: delete same annotation about gitcache
by Cui Yili 30 Mar '21

30 Mar '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/crystal-shards/build | 3 --- 1 file changed, 3 deletions(-) diff --git a/container/crystal-shards/build b/container/crystal-shards/build index 00153ab..872282b 100755 --- a/container/crystal-shards/build +++ b/container/crystal-shards/build @@ -2,9 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# GITCACHE_HOST and GITCACHE_PORT parameters come from gitcache_host and gitcache_port in the job.yaml file you submitted. -# If job.yaml does not have these two parameters, git will use the local proxy cache, -# GITCACHE_HOST defaults to the local ip, GITCACHE_PORT default port is 5000 . ../defconfig.sh docker_skip_rebuild "alpine:crystal-shards" -- 2.23.0

1 0

[PATCH compass-ci] container: docker run param -it => -d
by Cao Xueliang 30 Mar '21

30 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/etcd/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/etcd/start b/container/etcd/start index 7657b86..db94833 100755 --- a/container/etcd/start +++ b/container/etcd/start @@ -8,7 +8,7 @@ docker_rm etcd-server cmd=( docker run - -it + -d -p 2379:2379 -p 2380:2380 -p 4001:4001 -- 2.23.0

1 0

[PATCH compass-ci 2/4] container/rpm-repo: add build script
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/build | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/rpm-repo/build diff --git a/container/rpm-repo/build b/container/rpm-repo/build new file mode 100755 index 0000000..458e8f6 --- /dev/null +++ b/container/rpm-repo/build @@ -0,0 +1,6 @@ +#!/bin/sh + +. ../defconfig.sh +docker_skip_rebuild "fedora:update-repodata" + +docker build -t fedora:update-repodata . -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror: fix rubocop errors
by Li Yuanchao 29 Mar '21

29 Mar '21

[error msg] git_mirror.rb:112:1: C: Metrics/ClassLength: Class has too many lines. [101/100] class MirrorMain ... ^^^^^^^^^^^^^^^^ git_mirror.rb:192:3: C: Metrics/AbcSize: Assignment Branch Condition size for handle_feedback is too high. [<4, 17, 6> 18.47/15] def handle_feedback ... ^^^^^^^^^^^^^^^^^^^ git_mirror.rb:286:3: C: Metrics/AbcSize: Assignment Branch Condition size for reload_fork_info is too high. [<5, 17, 2> 17.83/15] def reload_fork_info(upstream_repos) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ git_mirror.rb:293:121: C: Layout/LineLength: Line is too long. [149/120] changed_files = %x(git -C /srv/git/#{@git_info[upstream_repos]['belong']}/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ git_mirror.rb:328:3: C: Metrics/AbcSize: Assignment Branch Condition size for update_stat_new_refs is too high. [<6, 14, 2> 15.36/15] def update_stat_new_refs(git_repo) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ git_mirror.rb:508:7: C: Naming/PredicateName: Rename is_upstream_repo? to upstream_repo?. def is_upstream_repo?(git_repo) ^^^^^^^^^^^^^^^^^ Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 45 +++++++++++++++++++++++++++++---------------- 1 file changed, 29 insertions(+), 16 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5ba9154..c967c8b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -199,14 +199,7 @@ class MirrorMain update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - return reload_fork_info(git_repo) if is_upstream_repo?(git_repo) - - new_refs = check_new_refs(git_repo) - return if new_refs[:heads].empty? - - feedback_info[:new_refs] = new_refs - send_message(feedback_info) - new_refs_log(git_repo, new_refs[:heads].length) if last_commit_new?(git_repo) + handle_feedback_new_refs(git_repo, feedback_info) end def do_push(fork_key) @@ -283,14 +276,19 @@ class MirrorMain return new_refs end + def get_change_files(upstream_repos) + old_commit = @git_info[upstream_repos][:cur_refs][:heads]['refs/heads/master'] + new_refs = check_new_refs(upstream_repos) + new_commit = new_refs[:heads]['refs/heads/master'] + mirror_dir = "/srv/git/#{@git_info[upstream_repos]['belong']}/#{upstream_repos}.git" + %x(git -C #{mirror_dir} diff --name-only #{old_commit}...#{new_commit}) + end + def reload_fork_info(upstream_repos) if @git_info[upstream_repos][:cur_refs].empty? @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) else - old_commit = @git_info[upstream_repos][:cur_refs][:heads]['refs/heads/master'] - new_refs = check_new_refs(upstream_repos) - new_commit = new_refs[:heads]['refs/heads/master'] - changed_files = %x(git -C /srv/git/#{@git_info[upstream_repos]['belong']}/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) + changed_files = get_change_files(upstream_repos) reload(changed_files, @git_info[upstream_repos]['belong']) end end @@ -325,13 +323,17 @@ class MirrorMain @fork_stat[git_repo][:offset_fetch] = offset_fetch + 1 end + def update_new_refs_info(git_repo, offset_new_refs) + @fork_stat[git_repo][:new_refs_time][offset_new_refs] = Time.now.to_s + @fork_stat[git_repo][:offset_new_refs] = offset_new_refs + 1 + @fork_stat[git_repo][:new_refs_count] = update_new_refs_count(@fork_stat[git_repo][:new_refs_count]) + end + def update_stat_new_refs(git_repo) @fork_stat[git_repo][:priority] += 1 offset_new_refs = @fork_stat[git_repo][:offset_new_refs] offset_new_refs = 0 if offset_new_refs >= 10 - @fork_stat[git_repo][:new_refs_time][offset_new_refs] = Time.now.to_s - @fork_stat[git_repo][:offset_new_refs] = offset_new_refs + 1 - @fork_stat[git_repo][:new_refs_count] = update_new_refs_count(@fork_stat[git_repo][:new_refs_count]) + update_new_refs_info(git_repo, offset_new_refs) end def update_fork_stat(git_repo, possible_new_refs) @@ -471,6 +473,17 @@ end class MirrorMain WEEK_SECONDS = 604800 + def handle_feedback_new_refs(git_repo, feedback_info) + return reload_fork_info(git_repo) if upstream_repo?(git_repo) + + new_refs = check_new_refs(git_repo) + return if new_refs[:heads].empty? + + feedback_info[:new_refs] = new_refs + send_message(feedback_info) + new_refs_log(git_repo, new_refs[:heads].length) if last_commit_new?(git_repo) + end + def merge_defaults(object_key, object, belong) return object if object_key == belong @@ -505,7 +518,7 @@ class MirrorMain return url end - def is_upstream_repo?(git_repo) + def upstream_repo?(git_repo) @upstreams['upstreams'].each do |repo| return true if git_repo == repo['git_repo'] end -- 2.23.0

2 2

[PATCH compass-ci] container: add etcd to start-depends
by Cao Xueliang 29 Mar '21

29 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/extract-stats/start-depends | 1 + container/scheduler/start-depends | 1 + 2 files changed, 2 insertions(+) diff --git a/container/extract-stats/start-depends b/container/extract-stats/start-depends index efbe5a2..abd487d 100755 --- a/container/extract-stats/start-depends +++ b/container/extract-stats/start-depends @@ -1 +1,2 @@ taskqueue +etcd diff --git a/container/scheduler/start-depends b/container/scheduler/start-depends index 1aced31..09a0b1d 100755 --- a/container/scheduler/start-depends +++ b/container/scheduler/start-depends @@ -2,3 +2,4 @@ redis es taskqueue rabbitmq +etcd -- 2.23.0

1 0

[PATCH compass-ci] src/monitor: detect crash and send msg to mq queue.
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/monitoring/filter.cr | 2 +- src/monitoring/parse_serial_logs.cr | 50 ++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 39cfbb6..b663f11 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,7 +56,7 @@ class Filter msg = JSON.parse(msg.to_s).as_h? return unless msg - @sp.save_dmesg_to_result_root(msg) + @sp.deal_serial_log(msg) @hash.keys.each do |query| if match_query(query.as_h, msg) send_msg(query, msg) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 0589523..9430d8c 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -4,6 +4,7 @@ require "../scheduler/elasticsearch_client" require "set" require "json" +require "../lib/mq" # This parses dmesg in a stream of serial log, finding a number of patterns # in various places of the dmesg and take actions accordingly. @@ -40,9 +41,15 @@ class SerialParser "Restarting system", ] + CRASH_PATTERNS = [ + "mount.nfs: Connection timed out", + "No space left on device", + ] + def initialize @host2head = Hash(String, Array(String)).new @host2rt = Hash(String, String).new + @mq = MQClient.instance end def host_in_msg(msg) @@ -59,6 +66,14 @@ class SerialParser end end + def detect_crash(msg, host) + message = msg["message"].to_s + CRASH_PATTERNS.each do |pattern| + matched = message.match(/.*(?<crash>#{pattern})/) + return matched.named_captures["crash"] unless matched.nil? + end + end + def delete_host(msg, host, signal) boundary_signal = detect_start_or_end(msg, host, signal) return unless boundary_signal @@ -67,10 +82,43 @@ class SerialParser @host2rt.delete(host) end - def save_dmesg_to_result_root(msg) + def mq_publish(msg, host) + crash_signal = detect_crash(msg, host) + return unless crash_signal + + job_id = "" + if @host2rt.has_key?(host) + job_id = File.basename(@host2rt[host]) + end + + mq_msg = { + "job_id" => job_id, + "testbox" => host, + "time" => msg["time"]? || Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800"), + "job_state" => "crash" + } + spawn mq_publish_check("job_mq", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def deal_serial_log(msg) host = host_in_msg(msg) return unless host + mq_publish(msg, host) + save_dmesg_to_result_root(msg, host) + end + + def save_dmesg_to_result_root(msg, host) delete_host(msg, host, START_PATTERNS) check_save = check_save_dmesg(msg, host) -- 2.23.0

1 0

[PATCH compass-ci] src/lib: add on_job_crash
by Li Ping 29 Mar '21

29 Mar '21

[how] mq publish the crash event, lifecycle will deal the job event. when job_state is crash and the job is consumed, then close the job and reboot crash machine, if no job, just reboot crash machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/lifecycle.cr | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 4bd2ac0..8f34903 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -122,6 +122,8 @@ class Lifecycle on_job_close(event) when "abnormal" on_abnormal_job(event) + when "crash" + on_job_crash(event) else on_other_job(event) end @@ -190,6 +192,17 @@ class Lifecycle update_cached_machine(job["testbox"].to_s, event) end + def on_job_crash(event) + event_job_id = event["job_id"].to_s + if @jobs[event_job_id]? + @jobs.delete(event_job_id) + spawn @scheduler_api.close_job(event_job_id, "crash", "lifecycle") + end + + testbox = event["testbox"].to_s + reboot_crash_machine(testbox, event) + end + def on_job_boot(event) event_job_id = event["job_id"]?.to_s @jobs[event_job_id] = event unless event_job_id.empty? @@ -298,6 +311,22 @@ class Lifecycle spawn @scheduler_api.close_job(job_id, reason, "lifecycle") end + def reboot_crash_machine(testbox, event) + @machines.delete(testbox) + machine = @es.get_tbox(testbox) + return unless machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) + + mq_queue = get_machine_reboot_queue(testbox) + machine.as_h.delete("history") + machine.as_h["testbox"] = JSON::Any.new(testbox) + @mq.publish_confirm(mq_queue, machine.to_json, durable: true) + + machine["state"] = "rebooting_queue" + machine["time"] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") + @es.update_tbox(testbox, machine.as_h) + end + def reboot_timeout_machine(testbox) @machines.delete(testbox) machine = @es.get_tbox(testbox) -- 2.23.0

1 0

[PATCH compass-ci 4/5] container/updaterepo: add build script
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/updaterepo/build | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 container/updaterepo/build diff --git a/container/updaterepo/build b/container/updaterepo/build new file mode 100755 index 0000000..594558a --- /dev/null +++ b/container/updaterepo/build @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_skip_rebuild "update-repo" + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t update-repo ." + +system 'rm updaterepo' -- 2.23.0

1 0

[PATCH compass-ci 3/5] container/updaterepo: add update-repo Dockerfile
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/updaterepo/Dockerfile | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 container/updaterepo/Dockerfile diff --git a/container/updaterepo/Dockerfile b/container/updaterepo/Dockerfile new file mode 100644 index 0000000..adab626 --- /dev/null +++ b/container/updaterepo/Dockerfile @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:scheduler-dev + +MAINTAINER Li Ping <1477412247(a)qq.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +ENV UPDATE_REPO_DIR /c/cci/update-repo + +RUN mkdir -p $UPDATE_REPO_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $UPDATE_REPO_DIR + +COPY --chown=1090:1090 updaterepo . -- 2.23.0

1 0

[PATCH compass-ci 1/5] add update-repo service
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/web_env.cr | 9 +++++++++ src/updaterepo.cr | 17 +++++++++++++++++ src/updaterepo/constants.cr | 4 ++++ src/updaterepo/updaterepo.cr | 16 ++++++++++++++++ 4 files changed, 46 insertions(+) create mode 100644 src/updaterepo.cr create mode 100644 src/updaterepo/constants.cr create mode 100644 src/updaterepo/updaterepo.cr diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index b3a90a8..d47e7e5 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -4,6 +4,7 @@ require "./sched" require "./lifecycle" require "./json_logger" +require "./updaterepo" class HTTP::Server # Instances of this class are passed to an `HTTP::Server` handler. @@ -31,5 +32,13 @@ class HTTP::Server def create_lifecycle @lifecycle = Lifecycle.new(self) end + + def repo + @repo ||= create_repo + end + + def create_repo + @repo = Repo.new(self) + end end end diff --git a/src/updaterepo.cr b/src/updaterepo.cr new file mode 100644 index 0000000..05fa4fe --- /dev/null +++ b/src/updaterepo.cr @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "./updaterepo/updaterepo" +require "./updaterepo/constants" +require "./lib/json_logger" + + +module Updaterepo + log = JSONLogger.new + + begin + Kemal.run(REPO_PORT) + rescue e + log.error(e) + end +end diff --git a/src/updaterepo/constants.cr b/src/updaterepo/constants.cr new file mode 100644 index 0000000..c9daaf2 --- /dev/null +++ b/src/updaterepo/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +REPO_PORT = (ENV.has_key?("REPO_PORT") ? ENV["REPO_PORT"] : 8132).to_i32 diff --git a/src/updaterepo/updaterepo.cr b/src/updaterepo/updaterepo.cr new file mode 100644 index 0000000..7ad9d20 --- /dev/null +++ b/src/updaterepo/updaterepo.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/updaterepo" +require "../lib/json_logger" + +module Updaterepo + VERSION = "0.1.0" + post "/upload" do |env| + env.repo.upload_repo + "done" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] sparrow/3-code/dev-env: add key EMAIL_LIMIT_COUNT
by Luan Shengde 29 Mar '21

29 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f96972e..3123c2a 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -12,6 +12,7 @@ server_name=$(hostname | cut -f1 -d.) : ${OS_HTTP_HOST:=$server_ip} : ${SEND_MAIL_HOST:=$server_ip} : ${SEND_MAIL_PORT:=10001} +: ${EMAIL_LIMIT_COUNT:=10} : ${SRV_HTTP_RESULT_HOST:=$server_ip} : ${SRV_HTTP_OS_HOST:=$server_ip} : ${SRV_HTTP_GIT_HOST:=$server_ip} @@ -91,6 +92,7 @@ SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT TASKQUEUE_HOST: $sched_host SEND_MAIL_HOST: $SEND_MAIL_HOST SEND_MAIL_PORT: $SEND_MAIL_PORT +EMAIL_LIMIT_COUNT: ${EMAIL_LIMIT_COUNT} LOGGING_ES_HOST: $LOGGING_ES_HOST LOGGING_ES_PORT: $LOGGING_ES_PORT INITRD_HTTP_HOST: $INITRD_HTTP_HOST -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] lib/mail-post/email_limit_queue: use ENV instead of static value
by Luan Shengde 29 Mar '21

29 Mar '21

use ENV variable instead of static value. enable user-defined new value without edit the code. just change the key and restart the service. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_limit_queue.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/mail-post/email_limit_queue.rb b/lib/mail-post/email_limit_queue.rb index c7e3954..62e9634 100755 --- a/lib/mail-post/email_limit_queue.rb +++ b/lib/mail-post/email_limit_queue.rb @@ -54,9 +54,9 @@ class EmailRateLimit end def change_queue(email) - return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + return unless (@redis.hget 'email_in_limit', email).to_i >= ENV['EMAIL_LIMIT_COUNT'].to_i @redis.hdel 'email_in_limit', email - @redis.hset 'email_out_limit', email, 10 + @redis.hset 'email_out_limit', email, ENV['EMAIL_LIMIT_COUNT'] end end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/send-internet-mail: add key EMAIL_LIMIT_COUNT
by Luan Shengde 29 Mar '21

29 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 136435d..57bf1f6 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -14,6 +14,7 @@ names = Set.new %w[ SEND_MAIL_PORT SENT_MAILDIR REDIS_PORT + EMAIL_LIMIT_COUNT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/send-internet-mail: add key EMAIL_LIMIT_COUNT for start
by Luan Shengde 29 Mar '21

29 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 136435d..57bf1f6 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -14,6 +14,7 @@ names = Set.new %w[ SEND_MAIL_PORT SENT_MAILDIR REDIS_PORT + EMAIL_LIMIT_COUNT ] defaults = relevant_defaults(names) -- 2.23.0

2 2

[PATCH compass-ci 7/7] scheduler: add get_new_seqno in redis client
by Cao Xueliang 29 Mar '21

29 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/redis_client.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/redis_client.cr b/src/scheduler/redis_client.cr index 030d1b7..ed2af6e 100644 --- a/src/scheduler/redis_client.cr +++ b/src/scheduler/redis_client.cr @@ -70,4 +70,8 @@ class Redis::Client def remove_finished_job(job_id : String) @client.hdel("sched/id2job", job_id) end + + def get_new_seqno + return @client.incr("#{QUEUE_NAME_BASE}/seqno") + end end -- 2.23.0

1 0

[PATCH compass-ci 6/7] scheduler: update job parameter in etcd
by Cao Xueliang 29 Mar '21

29 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 7 ------- src/scheduler/update_job_parameter.cr | 9 ++++++++- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f74df9b..6fd6657 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -166,13 +166,6 @@ class Sched @es.update_tbox(testbox.to_s, hash) end - def update_testbox_time(job_id) - job = @redis.get_job(job_id) - testbox = job["testbox"] - hash = {"time" => @env.get?("time").to_s} - @es.update_tbox(testbox, hash) - end - def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index 92b905e..28ab0f2 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -27,7 +27,7 @@ class Sched job_content[parameter] = value end - @redis.update_job(job_content) + update_id2job(job_content) # json log log = job_content.dup @@ -46,4 +46,11 @@ class Sched } spawn mq_publish_confirm(JOB_MQ, mq_msg.to_json) end + + def update_testbox_time(job_id) + job = get_id2job(job_id) + testbox = job["testbox"] + hash = {"time" => @env.get?("time").to_s} + @es.update_tbox(testbox, hash) + end end -- 2.23.0

1 0

[PATCH compass-ci 5/7] scheduler: move job from in_process to extract_stats in etcd
by Cao Xueliang 29 Mar '21

29 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/close_job.cr | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index a57b233..746f70a 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -8,7 +8,7 @@ class Sched @env.set "job_id", job_id - job = @redis.get_job(job_id) + job = get_id2job(job_id) # update job_state job_state = @env.params.query["job_state"]? @@ -21,17 +21,8 @@ class Sched raise "es set job content fail!" end - subqueue = job.subqueue - queue = (subqueue == "idle" ? job.queue : "#{job.queue}/#{subqueue}") - - response = @task_queue.hand_over_task( - "sched/#{queue}", "extract_stats", job_id - ) - if response[0] != 201 - raise "#{response}" - end - - @redis.remove_finished_job(job_id) + move_process2stats(job) + delete_id2job(job.id) job_state ||= "complete" @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) -- 2.23.0

1 0

[PATCH compass-ci 4/7] scheduler: use etcd list-watch to consume job
by Cao Xueliang 29 Mar '21

29 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 113 +++++++++++++++++++++++++++------ 1 file changed, 92 insertions(+), 21 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 78b10ad..2596f83 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -69,42 +69,113 @@ class Sched default_queues = [] of String queues_str.split(',', remove_empty: true) do |item| - default_queues << item.strip + default_queues << "sched/#{item.strip}/ready" end - sub_queues = [] of String - default_queues.each do |queue| - # TODO: this could be high cost and should be improved in future : keys(pattern). - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") - next if matched_queues.empty? + return default_queues.uniq + end - matched_queues.each do |mq| - match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) - sub_queues << match_data[1] if match_data + def get_job_from_queues(queues, testbox) + job = nil + etcd_job = consume_job(queues) + job_id = etcd_job.key.split("/")[-1] + puts "#{testbox} got the job #{job_id}" + if job_id + begin + job = @es.get_job(job_id.to_s) + rescue ex + @log.warn("Invalid job (id=#{job_id}) in es. Info: #{ex}") + @log.warn(ex.inspect_with_backtrace) end end - idle_queues = [] of String - delimiter_queues = [] of String - default_queues.each do |queue| - idle_queues << "#{queue}/idle" - delimiter_queues << "#{queue}/delimiter@localhost" + if job + job.update({"testbox" => testbox}) + job.set_result_root + @log.info(%({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state" +: "set result root"})) + set_id2job(job) end - all_queues = delimiter_queues + rand_queues(sub_queues) + idle_queues + return job + end + + def consume_job(queues) + job, revision = consume_by_list(queues) + return job if job - return all_queues.uniq + consume_by_watch(queues, revision) end - def get_job_from_queues(queues, testbox) - job = nil + def consume_by_list(queues) + jobs, revision = get_history_jobs(queues) + jobs.shuffle! + while true + return nil, revision if jobs.empty? + + job = jobs.delete_at(0) + return job, revision if ready2process(job) + end + end + def get_history_jobs(queues) + revisions = [] of Int64 + ec = EtcdClient.new + jobs = [] of Etcd::Model::Kv queues.each do |queue| - job = prepare_job("sched/#{queue}", testbox) - return job if job + job = ec.range_prefix(queue) + revisions << job.header.not_nil!.revision + jobs += job.kvs end - return job + return jobs, revisions.min + end + + def consume_by_watch(queues, revision) + ready_queues = split_ready_queues(queues) + channel = watch_queues(ready_queues, revision) + loop_handle_event(channel) + end + + def split_ready_queues(queues) + ready_queues = [] of String + queues.each do |queue| + tmp = queue.split("/ready")[0] + ready_queues << "#{tmp}/ready" + end + + ready_queues.uniq + end + + def watch_queues(queues, revision) + puts "watch #{queues}, revision #{revision}" + channel = Channel(Array(Etcd::Model::WatchEvent)).new + queues.each do |queue| + watcher = EtcdClient.new.watch_prefix(queue, start_revision: revision.to_i64, progress_notify: false ,filters: [Etcd::Watch::Filter::NODELETE]) do |events| + channel.send(events) + end + spawn { watcher.start } + Fiber.yield + end + + return channel + end + + def loop_handle_event(channel) + while true + events = channel.receive + events.each do |event| + return event.kv if ready2process(event.kv) + end + end + end + + def ready2process(job) + ec = EtcdClient.new + f_queue = job.key + t_queue = f_queue.gsub("/ready/", "/in_process/") + value = job.value + ec.move(f_queue, t_queue, value) end def get_job_boot(host, boot_type) -- 2.23.0

1 0

[PATCH compass-ci 3/7] scheduler: submit job to etcd db
by Cao Xueliang 29 Mar '21

29 Mar '21

before, we add job to taskqueue, then we save job to es, after, we save job to es at first, then we add job to etcd. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/submit_job.cr | 64 +++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index fe9efa1..c639ad7 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -58,18 +58,12 @@ class Sched next if (config["roles"].as_a.map(&.to_s) & roles).empty? queue = host.to_s - job_id = add_task("#{queue}/#{subqueue}", lab) + job_id = get_job_id(lab) # return when job_id is '0' # 2 Questions: # - how to deal with the jobs added to DB prior to this loop # - may consume job before all jobs done - return job_messages << { - "job_id" => "0", - "message" => "add task queue sched/#{queue} failed", - "job_state" => "submit", - } unless job_id - job_ids << job_id # add to job content when multi-test @@ -87,15 +81,15 @@ class Sched job["direct_macs"] = direct_macs.join(" ") job["direct_ips"] = direct_ips.join(" ") - response = add_job(job, job_id) - message = (response["error"]? ? response["error"]["root_cause"] : "") + status, msg = add_job(job, job_id) + job_id = "0" unless status job_messages << { "job_id" => job_id, - "message" => message.to_s, + "message" => msg, "job_state" => "submit", "result_root" => "/srv#{job.result_root}", } - return job_messages if response["error"]? + return job_messages unless status end cluster_id = job_ids[0] @@ -117,31 +111,31 @@ class Sched # success: [{"job_id" => job_id, "message" => "", job_state => "submit"}] # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) - queue = "#{job.queue}/#{job.subqueue}" + job_id = get_job_id(job.lab) - job_id = add_task(queue, job.lab) + status, msg = add_job(job, job_id) + job_id = "0" unless status return [{ - "job_id" => "0", - "message" => "add task queue sched/#{queue} failed", + "job_id" => job_id, + "message" => msg, "job_state" => "submit", - }] unless job_id - - response = add_job(job, job_id) - message = (response["error"]? ? response["error"]["root_cause"] : "") - - return [{ - "job_id" => job_id, - "message" => message.to_s, - "job_state" => "submit", "result_root" => "/srv#{job.result_root}", }] end - # return job_id - def add_task(queue, lab) - task_desc = JSON.parse(%({"domain": "compass-ci", "lab": "#{lab}"})) - response = @task_queue.add_task("sched/#{queue}", task_desc) - JSON.parse(response[1].to_json)["id"].to_s if response[0] == 200 + def add_task(job, job_id) + task_desc = JSON.parse(%({"domain": "compass-ci", "id": "#{job_id}"})) + key = "sched/#{job.queue}/ready/#{job.subqueue}/#{job_id}" + response = @etcd.put(key, task_desc) + if response + return true, "" + else + return false, "add the job to queue failed: job_id #{job_id}, queue #{key}" + end + end + + def get_job_id(lab) + "#{lab}.#{(a)redis.get_new_seqno()}" end def save_secrets(job, job_id) @@ -150,11 +144,19 @@ class Sched @redis.hash_set("id2secrets", job_id, job["secrets"]?.to_json) job.delete("secrets") end - # add job content to es and return a response + def add_job(job, job_id) + # deal the job fields at first save_secrets(job, job_id) job.update_id(job_id) - @es.set_job_content(job) + + # save the job to es + response = @es.set_job_content(job) + msg = (response["error"]? ? response["error"]["root_cause"] : "") + return false, msg.to_s if response["error"]? + + # set job_id to ready queue + add_task(job, job_id) end def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) -- 2.23.0

1 0

[PATCH compass-ci 2/7] extract-stats: use etcd list-watch to handle extract-stats queue
by Cao Xueliang 29 Mar '21

29 Mar '21

before, we use taskqueue to handle extract-stats queue, after, we use etcd list-watch to handle extract-stats queue. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats.cr | 2 +- src/extract-stats/extract_stats.cr | 67 +++++++++++++++++++------ src/extract-stats/stats_worker.cr | 78 +++++++++--------------------- 3 files changed, 75 insertions(+), 72 deletions(-) diff --git a/src/extract-stats.cr b/src/extract-stats.cr index 2f4b97d..0908a91 100644 --- a/src/extract-stats.cr +++ b/src/extract-stats.cr @@ -5,4 +5,4 @@ require "./extract-stats/extract_stats" # results data post processing -ExtractStats.in_extract_stats +ExtractStats.consume_tasks diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index 45310ad..4ae3e71 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -1,30 +1,67 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +require "../lib/etcd_client" require "./constants" require "./stats_worker" module ExtractStats - # Consume scheduler queue - def self.in_extract_stats - back_fill_task - STATS_WORKER_COUNT.times do - Process.fork do - consume_task + def self.consume_tasks + channel = Channel(String).new + revision = self.consume_by_list(EXTRACT_STATS_QUEUE_PATH, channel) + self.consume_by_watch(EXTRACT_STATS_QUEUE_PATH, revision, channel) + end + + def self.consume_by_list(queue, channel) + tasks, revision = self.get_history_tasks(queue) + self.handle_history_tasks(tasks, channel) + + return revision + end + + def self.consume_by_watch(queue, revision, channel) + self.watch_queue(queue, revision, channel) + self.handle_events(channel) + end + + def self.get_history_tasks(queue) + tasks = [] of Etcd::Model::Kv + range = EtcdClient.new.range_prefix(queue) + revision = range.header.not_nil!.revision + tasks += range.kvs + + return tasks, revision + end + + def self.handle_history_tasks(tasks, channel) + while true + return if tasks.empty? + + task = tasks.delete_at(0) + spawn { StatsWorker.new.handle(task.key, channel) } + + sleep 1 + end + end + + def self.watch_queue(queue, revision, channel) + watcher = EtcdClient.new.watch_prefix(queue, start_revision: revision.to_i64, filters: [Etcd::Watch::Filter::NODELETE]) do |events| + events.each do |event| + puts event + channel.send(event.kv.key) end end - # keep main-process alive - sleep + spawn { watcher.start } + Fiber.yield end - def self.consume_task - worker = StatsWorker.new - worker.consume_sched_queue(EXTRACT_STATS_QUEUE_PATH) - end + def self.handle_events(channel) + while true + key = channel.receive + spawn { StatsWorker.new.handle(key, channel) } - def self.back_fill_task - worker = StatsWorker.new - worker.back_fill_task(EXTRACT_STATS_QUEUE_PATH) + sleep 1 + end end end diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 44c327d..5e0100e 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -require "../lib/taskqueue_api" +require "../lib/etcd_client" require "../scheduler/elasticsearch_client" require "../scheduler/redis_client" require "../scheduler/constants" @@ -11,40 +11,25 @@ require "./constants.cr" class StatsWorker def initialize @es = Elasticsearch::Client.new - @tq = TaskQueueAPI.new + @etcd = EtcdClient.new @rc = RegressionClient.new end - def consume_sched_queue(queue_path : String) - loop do - begin - job_id = get_job_id(queue_path) - if job_id # will consume the job by post-processing - job = @es.get_job_content(job_id) - result_root = job["result_root"]? - result_post_processing(job_id, result_root.to_s, queue_path) - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - end - rescue e - STDERR.puts e.message - error_message = e.message - - # incase of many error message when task-queue, ES does not work - sleep(10) - @tq = TaskQueueAPI.new if error_message && error_message.includes?("3060': Connection refused") - end - end - end - - # get job_id from task-queue - def get_job_id(queue_path : String) - response = @tq.consume_task(queue_path) - if response[0] == 200 - JSON.parse(response[1].to_json)["id"].to_s - else - # will sleep 60s if no task in task-queue - sleep(60) - nil + def handle(queue_path, channel) + begin + res = @etcd.range(queue_path) + return nil if res.count == 0 + + job_id = queue_path.split("/")[-1] + job = @es.get_job_content(job_id) + result_root = job["result_root"]? + result_post_processing(job_id, result_root.to_s, queue_path) + @etcd.delete(queue_path) + rescue e + channel.send(queue_path) + STDERR.puts e.message + # incase of many error message when task-queue, ES does not work + sleep(10) end end @@ -61,10 +46,7 @@ class StatsWorker def store_stats_es(result_root : String, job_id : String, queue_path : String) stats_path = "#{result_root}/stats.json" - unless File.exists?(stats_path) - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - raise "#{stats_path} file not exists." - end + return unless File.exists?(stats_path) stats = File.open(stats_path) do |file| JSON.parse(file) @@ -81,7 +63,6 @@ class StatsWorker { :index => "jobs", :type => "_doc", :id => job_id, - :refresh => "wait_for", :body => {:doc => update_content}, } ) @@ -90,9 +71,10 @@ class StatsWorker unless new_error_ids.empty? sample_error_id = new_error_ids.sample STDOUT.puts "send a delimiter task: job_id is #{job_id}" - @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => sample_error_id, - "job_id" => job_id, - "lab" => LAB}.to_json)) + queue = "#{DELIMITER_TASK_QUEUE}/#{job_id}" + value = {"error_id" => sample_error_id} + @etcd.put(queue, value) + msg = %({"job_id": "#{job_id}", "new_error_id": "#{sample_error_id}"}) system "echo '#{msg}'" end @@ -130,20 +112,4 @@ class StatsWorker end error_ids end - - def back_fill_task(queue_path) - redis_client = Redis::Client.new - # this queue may have leftover task_ids - queue_name = "queues/#{queue_path}/in_process" - begin - job_ids = redis_client.(a)client.zrange(queue_name, 0, -1) - return if job_ids.empty? - - job_ids.each do |job_id| - @tq.hand_over_task(queue_path, queue_path, job_id.to_s) - end - rescue e - STDERR.puts e.message - end - end end -- 2.23.0

1 0

[PATCH compass-ci 1/7] src/scheduler: add opt job in etcd
by Cao Xueliang 29 Mar '21

29 Mar '21

Before, we operate the job in redis. After, we operate the job in etcd. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 3 +++ src/scheduler/opt_job_in_etcd.cr | 36 ++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) create mode 100644 src/scheduler/opt_job_in_etcd.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index b10a16d..f74df9b 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -7,6 +7,7 @@ require "yaml" require "./mq" require "./job" require "./web_env" +require "./etcd_client" require "./block_helper" require "./taskqueue_api" require "./remote_git_client" @@ -23,6 +24,7 @@ require "../scheduler/request_cluster_state" require "../scheduler/update_job_parameter" require "../scheduler/create_job_cpio" require "../scheduler/download_file" +require "../scheduler/opt_job_in_etcd" class Sched property es @@ -37,6 +39,7 @@ class Sched @redis = Redis::Client.instance @mq = MQClient.instance @task_queue = TaskQueueAPI.new + @etcd = EtcdClient.new @rgc = RemoteGitClient.new @env = env @log = env.log.as(JSONLogger) diff --git a/src/scheduler/opt_job_in_etcd.cr b/src/scheduler/opt_job_in_etcd.cr new file mode 100644 index 0000000..f38433a --- /dev/null +++ b/src/scheduler/opt_job_in_etcd.cr @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def update_id2job(job_content) + id = job_content["id"].to_s + job = get_id2job(id) + job.update(job_content) + @etcd.update("sched/id2job/#{id}", job.dump_to_json) + end + + def set_id2job(job : Job) + @etcd.put("sched/id2job/#{job.id}", job.dump_to_json) + end + + def get_id2job(id) + response = @etcd.range("sched/id2job/#{id}") + raise "get job from etcd failed. id: #{id}" unless response.count == 1 + Job.new(JSON.parse(response.kvs[0].value.not_nil!), id) + end + + def delete_id2job(id) + @etcd.delete("sched/id2job/#{id}") + end + + def update_tbox_wtmp(testbox, wtmp_hash) + @etcd.update("sched/tbox_wtmp/#{testbox}", wtmp_hash) + end + + def move_process2stats(job : Job) + f_queue = "sched/#{job.queue}/in_process/#{job.subqueue}/#{job.id}" + t_queue = "extract_stats/#{job.id}" + value = { "id" => "#{job.id}" } + @etcd.move(f_queue, t_queue, value) + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/4-docker/buildall: build and start all in parallel to speedup
by Liu Yinsi 29 Mar '21

29 Mar '21

before: function build_depends() are parallel, but do_one_build(), start_depends(), do_one_run() are not parallel. after: function build_depends(), do_one_build(), start_depends(), do_one_run() all are parallel. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 3cd5719..3611158 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -24,7 +24,7 @@ do_one_build() { local container=$1 local container_name=$(basename $container) - lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock + lockfile-create -q --use-pid --retry 100 --lock-name "build_${container_name}".lock mkdir $tmpdir/$container_name 2>/dev/null && ( @@ -33,7 +33,7 @@ do_one_build() [ "$?" == 0 ] && log_info "finish build $container" [ -x install ] && ./install ) - lockfile-remove --lock-name "$container_name".lock + lockfile-remove --lock-name "build_${container_name}".lock } start_depends() @@ -42,8 +42,9 @@ start_depends() for dep in $(cat $container/start-depends 2> /dev/null) do - start_depends $CONTAINER_PATH/$dep + start_depends $CONTAINER_PATH/$dep & done + wait do_one_run $container } @@ -52,6 +53,7 @@ do_one_run() { local container=$1 local container_name=$(basename $container) + lockfile-create -q --use-pid --retry 100 --lock-name "start_${container_name}".lock mkdir $tmpdir/start_$container_name 2>/dev/null && ( @@ -64,14 +66,31 @@ do_one_run() [ -x start ] && ./start [ "$container_name" == 'lkp-initrd' ] && ./run ) + lockfile-remove --lock-name "start_${container_name}".lock +} + +build_all() +{ + for dir in $CONTAINER_PATH/*/ + do + build_depends $dir & + done + wait } +start_all() +{ + for dir in $CONTAINER_PATH/*/ + do + start_depends $dir & + done + wait +} + + tmpdir=$(mktemp -d) -for dir in $CONTAINER_PATH/*/ -do - [ "$action" != "start-only" ] && build_depends $dir - start_depends $dir -done +[ "$action" == "start-only" ] || build_all +start_all rm -fr $tmpdir -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: build and start all in parallel to speedup
by Liu Yinsi 29 Mar '21

29 Mar '21

before: function build_depends() are parallel, but do_one_build(), start_depends(), do_one_run() are not parallel. after: function build_depends(), do_one_build(), start_depends(), do_one_run() all are parallel. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 3cd5719..f6b9b65 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -24,7 +24,7 @@ do_one_build() { local container=$1 local container_name=$(basename $container) - lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock + lockfile-create -q --use-pid --retry 100 --lock-name "build_${container_name}".lock mkdir $tmpdir/$container_name 2>/dev/null && ( @@ -33,7 +33,7 @@ do_one_build() [ "$?" == 0 ] && log_info "finish build $container" [ -x install ] && ./install ) - lockfile-remove --lock-name "$container_name".lock + lockfile-remove --lock-name "build_${container_name}".lock } start_depends() @@ -42,8 +42,9 @@ start_depends() for dep in $(cat $container/start-depends 2> /dev/null) do - start_depends $CONTAINER_PATH/$dep + start_depends $CONTAINER_PATH/$dep & done + wait do_one_run $container } @@ -52,6 +53,7 @@ do_one_run() { local container=$1 local container_name=$(basename $container) + lockfile-create -q --use-pid --retry 100 --lock-name "start_${container_name}".lock mkdir $tmpdir/start_$container_name 2>/dev/null && ( @@ -64,14 +66,31 @@ do_one_run() [ -x start ] && ./start [ "$container_name" == 'lkp-initrd' ] && ./run ) + lockfile-remove --lock-name "start_${container_name}".lock +} + +build_all() +{ + for dir in $CONTAINER_PATH/*/ + do + build_depends $dir & + done + wait } +start_all() +{ + for dir in $CONTAINER_PATH/*/ + do + start_depends $dir & + done + wait +} + + tmpdir=$(mktemp -d) -for dir in $CONTAINER_PATH/*/ -do - [ "$action" != "start-only" ] && build_depends $dir - start_depends $dir -done +[ "$action" != "start-only" ] && build_all +start_all rm -fr $tmpdir -- 2.23.0

3 3

[PATCH compass-ci] remove gitchche env
by Liu Yinsi 29 Mar '21

29 Mar '21

refer patch: container/gitcache: delete gitcache docker that is no longer in use Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/crystal-shards/build-depends | 1 - sparrow/3-code/dev-env | 4 ---- 2 files changed, 5 deletions(-) diff --git a/container/crystal-shards/build-depends b/container/crystal-shards/build-depends index 6eed7ec..6627b6f 100755 --- a/container/crystal-shards/build-depends +++ b/container/crystal-shards/build-depends @@ -1,2 +1 @@ crystal-base -gitcache diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f96972e..c42fba5 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -78,8 +78,6 @@ SCHED_PORT: $sched_port LKP_SERVER: $server_ip GIT_SERVER: $GIT_SERVER OS_HTTP_HOST: $OS_HTTP_HOST -GITCACHE_HOST: $gitcache_host -GITCACHE_PORT: $gitcache_port SRV_HTTP_RESULT_HOST: $SRV_HTTP_RESULT_HOST SRV_HTTP_OS_HOST: $SRV_HTTP_OS_HOST SRV_HTTP_GIT_HOST: $SRV_HTTP_GIT_HOST @@ -125,8 +123,6 @@ mkdir -p /etc/compass-ci/scheduler cat > /etc/compass-ci/scheduler/local-testbox-env.yaml <<-EOF LKP_SERVER: GIT_SERVER: -GITCACHE_HOST: -GITCACHE_PORT: SRV_HTTP_RESULT_HOST: SRV_HTTP_RESULT_PORT: SRV_HTTP_OS_HOST: -- 2.23.0

1 0

[PATCH v1 compass-ci] container: fix library libgc.so.1: No such file or directory
by Lu Weitao 29 Mar '21

29 Mar '21

use the compile env as the runtime env in extract-stats and taskqueue container Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/extract-stats/Dockerfile | 2 +- container/taskqueue/Dockerfile | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/extract-stats/Dockerfile b/container/extract-stats/Dockerfile index 3a8c8f3..08abc8c 100644 --- a/container/extract-stats/Dockerfile +++ b/container/extract-stats/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:scheduler-dev RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ diff --git a/container/taskqueue/Dockerfile b/container/taskqueue/Dockerfile index 28ac925..2d5b10a 100644 --- a/container/taskqueue/Dockerfile +++ b/container/taskqueue/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:scheduler-dev RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/mail-post/email_limit_queue: use ENV instead of static value
by Luan Shengde 29 Mar '21

29 Mar '21

use ENV variable instead of static value. enable user-defined new value without edit the code. just restart the service. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_limit_queue.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/mail-post/email_limit_queue.rb b/lib/mail-post/email_limit_queue.rb index c7e3954..62e9634 100755 --- a/lib/mail-post/email_limit_queue.rb +++ b/lib/mail-post/email_limit_queue.rb @@ -54,9 +54,9 @@ class EmailRateLimit end def change_queue(email) - return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + return unless (@redis.hget 'email_in_limit', email).to_i >= ENV['EMAIL_LIMIT_COUNT'].to_i @redis.hdel 'email_in_limit', email - @redis.hset 'email_out_limit', email, 10 + @redis.hset 'email_out_limit', email, ENV['EMAIL_LIMIT_COUNT'] end end -- 2.23.0

1 0

[PATCH compass-ci] container/gitcache: delete gitcache docker that is no longer in use
by Cui Yili 28 Mar '21

28 Mar '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/gitcache/Dockerfile | 15 --------------- container/gitcache/build | 10 ---------- container/gitcache/start | 20 -------------------- 3 files changed, 45 deletions(-) delete mode 100644 container/gitcache/Dockerfile delete mode 100755 container/gitcache/build delete mode 100755 container/gitcache/start diff --git a/container/gitcache/Dockerfile b/container/gitcache/Dockerfile deleted file mode 100644 index d92a141..0000000 --- a/container/gitcache/Dockerfile +++ /dev/null @@ -1,15 +0,0 @@ -# Origin: https://github.com/git-cloner/gitcache -# Copyright (C) 2019-2020 Eric D.little51 -# SPDX-License-Identifier: GPL-3.0 - -FROM golang:alpine - -ENV GO111MODULE on -ENV GOPROXY https://goproxy.cn - - -RUN apk add --no-cache git && \ - git clone https://github.com/git-cloner/gitcache /gitcache && \ - cd /gitcache && go build - -ENTRYPOINT ["/gitcache/gitcache"] diff --git a/container/gitcache/build b/container/gitcache/build deleted file mode 100755 index d7f3523..0000000 --- a/container/gitcache/build +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/bash -# Origin: https://github.com/git-cloner/gitcache -# Copyright (C) 2019-2020 Eric D.little51 -# SPDX-License-Identifier: GPL-3.0 - -. ../defconfig.sh - -docker_skip_rebuild "gitcache" - -docker build -t gitcache . diff --git a/container/gitcache/start b/container/gitcache/start deleted file mode 100755 index 0e1de1b..0000000 --- a/container/gitcache/start +++ /dev/null @@ -1,20 +0,0 @@ -#!/bin/bash -# Origin: https://github.com/git-cloner/gitcache -# Copyright (C) 2019-2020 Eric D.little51 -# SPDX-License-Identifier: GPL-3.0 - -. $CCI_SRC/container/defconfig.sh - -docker_rm gitcache - -cmd=( - docker run -dt - --name gitcache - --restart=always - -p 5000:5000 - -v /srv/cache/gitcache/:/var/gitcache/ - -v /etc/localtime:/etc/localtime:ro - gitcache -) - -"${cmd[@]}" -- 2.23.0

2 1

[PATCH compass-ci] container: add etcd container
by Cao Xueliang 28 Mar '21

28 Mar '21

add etcd container Dockerfile, build and start script. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/etcd/Dockerfile | 16 ++++++++++++++++ container/etcd/build | 12 ++++++++++++ container/etcd/start | 28 ++++++++++++++++++++++++++++ 3 files changed, 56 insertions(+) create mode 100644 container/etcd/Dockerfile create mode 100755 container/etcd/build create mode 100755 container/etcd/start diff --git a/container/etcd/Dockerfile b/container/etcd/Dockerfile new file mode 100644 index 0000000..3654fc4 --- /dev/null +++ b/container/etcd/Dockerfile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.13 + +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> + +ARG VERSION +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && apk add ca-certificates && \ + wget https://github.com/etcd-io/etcd/releases/download/${VERSION}/etcd-${VERSION… && \ + tar zxvf etcd-${VERSION}-linux-arm64.tar.gz && \ + mv etcd-${VERSION}-linux-arm64/etcd* /bin/ && \ + rm -Rf etcd-${VERSION}-linux-arm64* /var/cache/apk/* + +VOLUME /data diff --git a/container/etcd/build b/container/etcd/build new file mode 100755 index 0000000..6ae25dd --- /dev/null +++ b/container/etcd/build @@ -0,0 +1,12 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ../defconfig.sh + +version=v3.4.15 +docker_skip_rebuild "etcd-${version}:alpine" + +docker build -t etcd-${version}:alpine --build-arg VERSION=${version} . + + diff --git a/container/etcd/start b/container/etcd/start new file mode 100755 index 0000000..7657b86 --- /dev/null +++ b/container/etcd/start @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +docker_rm etcd-server + +cmd=( + docker run + -it + -p 2379:2379 + -p 2380:2380 + -p 4001:4001 + -p 7001:7001 + -v /srv/etcd/dir:/data + -e ETCD_UNSUPPORTED_ARCH=arm64 + --name etcd-server + etcd-v3.4.15:alpine + /bin/etcd + --name node-1 + --data-dir /data + --initial-cluster-state new + --listen-client-urls http://0.0.0.0:2379 + --advertise-client-urls http://0.0.0.0:2379 +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] 0-package/os/openEuler: update lockfile rpm download URL
by Liu Yinsi 27 Mar '21

27 Mar '21

The old URL is invalid. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/openEuler | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 61a37c4..9cb7b04 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -28,8 +28,8 @@ pkgs=( yum install -y --skip-broken "${pkgs[@]}" -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… +wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… +wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm rm -rf *.rpm -- 2.23.0

1 0

[PATCH compass-ci] container/web-backend: fix SyntaxError)
by Lu Weitao 27 Mar '21

27 Mar '21

[Why] Traceback (most recent call last): 2: from ./web-backend:9:in `<main>' 1: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require' /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require': /home/lwt/compass-ci/src/lib/web_backend.rb:681: syntax error, unexpected end-of-input, expecting keyword_end (SyntaxError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 1 + src/lib/web_backend.rb | 1 + 2 files changed, 2 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index c7d1ec6..1172160 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -170,6 +170,7 @@ end # } get '/get_jobs_summary' do group_jobs_stats(params) +end # GET /get_job_error?suite=virttest&tbox_group=vm-2p8g # Response diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 7f26b9c..932d362 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -614,6 +614,7 @@ def group_jobs_stats(params) return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end # ------------------------------------------------------------------------------------------- # job error table like: -- 2.23.0

1 0

[PATCH v2 compass-ci] netdata: modify the service port
by Yu Chuan 27 Mar '21

27 Mar '21

[Why] According our latest port policy, modify the service port: - netdata: 20012 => 20013 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/netdata/start | 4 +++- sparrow/3-code/dev-env | 1 + 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/container/netdata/start b/container/netdata/start index 18d225b8fcae..93eabb45dba7 100755 --- a/container/netdata/start +++ b/container/netdata/start @@ -4,13 +4,15 @@ . $CCI_SRC/container/defconfig.sh +load_cci_defaults + docker_rm netdata cmd=( docker run -d --restart=always --name netdata - -p 20012:19999 + -p ${NETDATA_PORT:=20013}:19999 -v /proc:/host/proc:ro -v /sys:/host/sys:ro -v /etc/localtime:/etc/localtime:ro diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f96972eb102d..209c6663238f 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -50,6 +50,7 @@ server_name=$(hostname | cut -f1 -d.) : ${REDIS_PORT:=6379} : ${MAILDIR:=/srv/cci/Maildir/.compass-ci} : ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} +: ${NETDATA_PORT:=20013} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF -- 2.23.0

1 0