- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 2/2] src/monitor: detect crash and send msg to mq queue.
by Li Ping 30 Mar '21

30 Mar '21

linux/kernel/panic.c: pr_emerg("Kernel panic - not syncing: %s\n", buf); When kernel panic happens, it will print the panic call stack, then the ending msg like: [ 14.405869] Kernel panic - not syncing: Fatal exception in interrupt some bug not cause kernel panic happens but also mean that job fails like: mount.nfs: Connection timed out No space left on device The above message would be detected, then send msg to mq queue to close job and reboot machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/monitoring/filter.cr | 2 +- src/monitoring/parse_serial_logs.cr | 47 +++++++++++++++++++++++++++-- 2 files changed, 45 insertions(+), 4 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 39cfbb6..b663f11 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,7 +56,7 @@ class Filter msg = JSON.parse(msg.to_s).as_h? return unless msg - @sp.save_dmesg_to_result_root(msg) + @sp.deal_serial_log(msg) @hash.keys.each do |query| if match_query(query.as_h, msg) send_msg(query, msg) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 451312c..ff3b84a 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -4,6 +4,7 @@ require "../scheduler/elasticsearch_client" require "set" require "json" +require "../lib/mq" # This parses dmesg in a stream of serial log, finding a number of patterns # in various places of the dmesg and take actions accordingly. @@ -40,9 +41,16 @@ class SerialParser "Restarting system", ] + CRASH_PATTERNS = [ + "mount.nfs: Connection timed out", + "No space left on device", + "Kernel panic - not syncing: ", + ] + def initialize @host2head = Hash(String, Array(String)).new @host2rt = Hash(String, String).new + @mq = MQClient.instance end def host_in_msg(msg) @@ -51,7 +59,7 @@ class SerialParser File.basename(msg["serial_path"].to_s) end - def detect_start_or_end(msg, pattern_list) + def detect_patterns(msg, pattern_list) message = msg["message"].to_s pattern_list.each do |pattern| matched = message.match(/.*(?<signal>#{pattern})/) @@ -60,17 +68,50 @@ class SerialParser end def delete_host(msg, host, signal) - boundary_signal = detect_start_or_end(msg, signal) + boundary_signal = detect_patterns(msg, signal) return unless boundary_signal @host2head.delete(host) @host2rt.delete(host) end - def save_dmesg_to_result_root(msg) + def mq_publish(msg, host) + crash_signal = detect_patterns(msg, CRASH_PATTERNS) + return unless crash_signal + + job_id = "" + if @host2rt.has_key?(host) + job_id = File.basename(@host2rt[host]) + end + + mq_msg = { + "job_id" => job_id, + "testbox" => host, + "time" => msg["time"]? || Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800"), + "job_state" => "crash" + } + spawn mq_publish_check("job_mq", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def deal_serial_log(msg) host = host_in_msg(msg) return unless host + mq_publish(msg, host) + save_dmesg_to_result_root(msg, host) + end + + def save_dmesg_to_result_root(msg, host) delete_host(msg, host, START_PATTERNS) check_save = check_save_dmesg(msg, host) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] src/lib: add on_job_crash
by Li Ping 30 Mar '21

30 Mar '21

[how] mq publish the crash event, lifecycle will deal the job event. when job_state is crash and the job is consumed, then close the job and reboot crash machine, if no job, just reboot crash machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/lifecycle.cr | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index adcb965..0e6cb82 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -122,6 +122,8 @@ class Lifecycle on_job_close(event) when "abnormal" on_abnormal_job(event) + when "crash" + on_job_crash(event) else on_other_job(event) end @@ -189,6 +191,16 @@ class Lifecycle update_cached_machine(job["testbox"].to_s, event) end + def on_job_crash(event) + event_job_id = event["job_id"].to_s + if @jobs[event_job_id]? + close_job(event_job_id, "crash") + end + + testbox = event["testbox"].to_s + reboot_crash_machine(testbox, event) + end + def on_job_boot(event) event_job_id = event["job_id"]?.to_s @jobs[event_job_id] = event unless event_job_id.empty? @@ -296,6 +308,15 @@ class Lifecycle spawn @scheduler_api.close_job(job_id, reason, "lifecycle") end + def reboot_crash_machine(testbox, event) + @machines.delete(testbox) + machine = @es.get_tbox(testbox) + return unless machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) + + reboot_machine(testbox, machine) + end + def reboot_timeout_machine(testbox) @machines.delete(testbox) machine = @es.get_tbox(testbox) @@ -308,6 +329,10 @@ class Lifecycle deadline = Time.parse(deadline.to_s, "%Y-%m-%dT%H:%M:%S", Time.local.location) return if Time.local < deadline + reboot_machine(testbox, machine) + end + + def reboot_machine(testbox, machine) mq_queue = get_machine_reboot_queue(testbox) machine.as_h.delete("history") machine.as_h["testbox"] = JSON::Any.new(testbox) -- 2.23.0

1 0

[PATCH compass-ci] etcd/Dockerfile: change source address
by Wang Yong 30 Mar '21

30 Mar '21

accelerate download Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- container/etcd/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/etcd/Dockerfile b/container/etcd/Dockerfile index 3654fc4..b3957e8 100644 --- a/container/etcd/Dockerfile +++ b/container/etcd/Dockerfile @@ -8,7 +8,7 @@ MAINTAINER Cao Xueliang <caoxl78320(a)163.com> ARG VERSION RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && apk add ca-certificates && \ - wget https://github.com/etcd-io/etcd/releases/download/${VERSION}/etcd-${VERSION… && \ + wget https://repo.huaweicloud.com/etcd/${VERSION}/etcd-${VERSION}-linux-arm64.ta… && \ tar zxvf etcd-${VERSION}-linux-arm64.tar.gz && \ mv etcd-${VERSION}-linux-arm64/etcd* /bin/ && \ rm -Rf etcd-${VERSION}-linux-arm64* /var/cache/apk/* -- 2.23.0

1 0

[PATCH v2 compass-ci] container/ssh-r/start: rename var for vm_name to docker_name
by Wei Jihui 30 Mar '21

30 Mar '21

vm_name --> docker_name Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/ssh-r/start | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/ssh-r/start b/container/ssh-r/start index 68a63f2..47a29eb 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -4,14 +4,14 @@ . $CCI_SRC/container/defconfig.sh -vm_name=ssh_r +docker_name=ssh_r -docker_rm ${vm_name} +docker_rm ${docker_name} cmd=( docker run --restart=always - --name ${vm_name} + --name ${docker_name} -e GATEWAY_PORTS=true -e TCP_FORWARDING=true -d -- 2.23.0

2 1

[PATCH compass-ci] container/taskqueue: compiling code every time
by Wu Zhende 30 Mar '21

30 Mar '21

Delete the complied taskqueue file. Otherwise, the code will be updated and the taskqueue file will not be recomplied. Uses old code all the time. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/taskqueue/build | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/container/taskqueue/build b/container/taskqueue/build index 4f69f85..518a27e 100755 --- a/container/taskqueue/build +++ b/container/taskqueue/build @@ -12,10 +12,8 @@ service_port=${TASKQUEUE_PORT:-3060} docker_skip_rebuild "taskqueue-alpine:v0.001-${service_port}" -if [[ ! -f "taskqueue" ]]; then - $CCI_SRC/container/compile . -else - echo -e "\n\033[34m- Info: using \033[31mexists\033[34m taskqueue -\033[0m\n" -fi +$CCI_SRC/container/compile . docker build -t taskqueue-alpine:v0.001-${service_port} . + +rm taskqueue -- 2.23.0

1 0

[PATCH compass-ci] monitoring/parse_serial_logs.cr: fix useless arg
by Wei Jihui 30 Mar '21

30 Mar '21

host is not used at detect_start_or_end Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/monitoring/parse_serial_logs.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 0589523..451312c 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -51,7 +51,7 @@ class SerialParser File.basename(msg["serial_path"].to_s) end - def detect_start_or_end(msg, host, pattern_list) + def detect_start_or_end(msg, pattern_list) message = msg["message"].to_s pattern_list.each do |pattern| matched = message.match(/.*(?<signal>#{pattern})/) @@ -60,7 +60,7 @@ class SerialParser end def delete_host(msg, host, signal) - boundary_signal = detect_start_or_end(msg, host, signal) + boundary_signal = detect_start_or_end(msg, signal) return unless boundary_signal @host2head.delete(host) -- 2.23.0

1 0

[PATCH lkp-tests] monitors/kmsg: not execute dmesg when in docker
by Lin Jiaxin 30 Mar '21

30 Mar '21

In order to avoid the following stderr: ==> /tmp/stderr <== dmesg: read kernel buffer failed: Operation not permitted Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- monitors/kmsg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/monitors/kmsg b/monitors/kmsg index 4edd90a9..a1cf7121 100755 --- a/monitors/kmsg +++ b/monitors/kmsg @@ -6,6 +6,8 @@ # serial console has better availability. is_virt && [ -z "$LKP_LOCAL_RUN" ] && exec > /dev/ttyS1 +[ -f /.dockerenv ] && exit 0 + if dmesg --help 2>&1 | grep -q -- --follow; then exec dmesg --follow --decode else -- 2.23.0

2 1

[PATCH compass-ci] src/lib/lifecycle.cr: use close_job close job
by Wei Jihui 30 Mar '21

30 Mar '21

unified use close_job to close job, reduce code Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/lib/lifecycle.cr | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 4bd2ac0..adcb965 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -177,8 +177,7 @@ class Lifecycle event_job_id = event["job_id"].to_s return unless @jobs.has_key?(event_job_id) - @jobs.delete(event_job_id) - spawn @scheduler_api.close_job(event_job_id, "abnormal", "lifecycle") + close_job(event_job_id, "abnormal") end def on_job_close(event) @@ -222,8 +221,7 @@ class Lifecycle return if machine_job_id.empty? return unless @jobs.has_key?(machine_job_id) - @jobs.delete(machine_job_id) - spawn @scheduler_api.close_job(machine_job_id, "abnormal", "lifecycle") + close_job(machine_job_id, "abnormal") end def max_time(times) -- 2.23.0

1 0

[PATCH compass-ci 3/4] container/rpm-repo: add rpm-repo.rb for update repodata
by Li Ping 30 Mar '21

30 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/rpm-repo.rb | 73 ++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 container/rpm-repo/rpm-repo.rb diff --git a/container/rpm-repo/rpm-repo.rb b/container/rpm-repo/rpm-repo.rb new file mode 100755 index 0000000..18f0507 --- /dev/null +++ b/container/rpm-repo/rpm-repo.rb @@ -0,0 +1,73 @@ +#!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require "bunny" +require "fileutils" +require "json" + + +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || '172.17.0.1' +MQ_PORT = ENV['MQ_PORT'] || 5672 + +class MQClient + def initialize(hostname = "172.17.0.1", port = "5672") + @conn = Bunny.new(hostname: hostname, port: port) + @conn.start + @channel = @conn.create_channel + end + + def queue(queue_name, opts = {}) + @channel.queue(queue_name, opts) + end + + def ack(delivery_info) + @channel.ack(delivery_info.delivery_tag) + end +end + +class HandleRepo + def initialize + @mq = MQClient.new(MQ_HOST, MQ_PORT) + @update = [] + end + + def handle_new_rpm + queue = @mq.queue("update_repo") + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + rpm_info = JSON.parse(msg) + puts rpm_info + rpm_info["upload_rpms"].each do |rpm| + rpm_path = File.dirname(rpm).sub("upload", "testing") + FileUtils.mkdir_p(rpm_path) unless File.directory?(rpm_path) + + dest = File.join(rpm_path.to_s, File.basename(rpm)) + @update << dest + FileUtils.mv(rpm, dest) + system("createrepo --update $(dirname #{rpm_path})") + end + update_pub_dir + @mq.ack(info) + end + end + + def update_pub_dir + @update.each do |rpm| + pub_path = File.dirname(rpm).sub("testing", "pub") + FileUtils.mkdir_p(pub_path) unless File.directory?(pub_path) + + dest = File.join(pub_path, File.basename(rpm)) + FileUtils.cp(rpm, dest) + + repodata_dest = File.join(File.dirname(pub_path), "repodata") + repodata_src = File.dirname(rpm).sub("Packages", "repodata") + + FileUtils.rm_r(repodata_dest) if Dir.exist?(repodata_dest) + FileUtils.cp_r(repodata_src, File.dirname(repodata_dest)) + end + end +end + +hr = HandleRepo.new +hr.handle_new_rpm -- 2.23.0

4 5

[PATCH compass-ci 5/5] container/updaterepo: add update-repo start script
by Li Ping 30 Mar '21

30 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/updaterepo/start | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100755 container/updaterepo/start diff --git a/container/updaterepo/start b/container/updaterepo/start new file mode 100755 index 0000000..46e3ad6 --- /dev/null +++ b/container/updaterepo/start @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_rm "update-repo" + +cmd = %W[ + docker run + --restart=always + --name update-repo + -u 1090:1090 + -p 8132:8132 + -d + -v /srv/rpm/:/srv/rpm/ + -v /etc/localtime:/etc/localtime:ro + update-repo +] + +cmd += ['sh', '-c', 'umask 002 && ./updaterepo'] + +system(*cmd) -- 2.23.0

2 2

[PATCH compass-ci 2/5] src/lib/updaterepo.cr: send messages to update-repo queue
by Li Ping 30 Mar '21

30 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/updaterepo.cr | 60 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 src/lib/updaterepo.cr diff --git a/src/lib/updaterepo.cr b/src/lib/updaterepo.cr new file mode 100644 index 0000000..4826fac --- /dev/null +++ b/src/lib/updaterepo.cr @@ -0,0 +1,60 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "json" +require "kemal" +require "./json_logger" +require "./mq" + +class Repo + def initialize(env : HTTP::Server::Context) + @mq = MQClient.instance + @env = env + @log = JSONLogger.new + end + + def upload_repo + begin + body = @env.request.body.not_nil!.gets_to_end + data = JSON.parse(body.to_s).as_h? + rescue e + @log.error(e) + end + + puts "-" * 50 + puts "post body:", data + + begin + # check if the parameters are complete + check_params_complete(data) + mq_publish(data) + rescue e + response = { "errcode" => "101", "errmsg" => "upload rpm failed" } + @log.info(response.to_json) + puts "error message: #{e.message}" + end + end + + def mq_publish(data) + mq_msg = data + puts mq_msg.to_json + + spawn mq_publish_check("update_repo", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def check_params_complete(params) + params = params.not_nil! + tmp_hash = {"errcode" => "101", "errmsg" => "no upload_rpms params"}.to_json + raise tmp_hash unless params["upload_rpms"]? + end +end -- 2.23.0

2 2

[PATCH v3 compass-ci 3/3] sparrow/3-code/dev-env: add key EMAIL_LIMIT_COUNT
by Luan Shengde 30 Mar '21

30 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 27258fc..7327083 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -12,6 +12,7 @@ server_name=$(hostname | cut -f1 -d.) : ${OS_HTTP_HOST:=$server_ip} : ${send_mail_host:=$server_ip} : ${SEND_MAIL_PORT:=10001} +: ${EMAIL_LIMIT_COUNT:=10} : ${srv_http_result_host:=$server_ip} : ${srv_http_os_host:=$server_ip} : ${srv_http_git_host:=$server_ip} @@ -90,6 +91,7 @@ SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT TASKQUEUE_HOST: $sched_host SEND_MAIL_HOST: $send_mail_host SEND_MAIL_PORT: $SEND_MAIL_PORT +EMAIL_LIMIT_COUNT: $EMAIL_LIMIT_COUNT LOGGING_ES_HOST: $LOGGING_ES_HOST LOGGING_ES_PORT: $LOGGING_ES_PORT INITRD_HTTP_HOST: $initrd_http_host -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] lib/mail-post/email_limit_queue: use ENV instead of static value
by Luan Shengde 30 Mar '21

30 Mar '21

use ENV variable instead of static value. enable user-defined new value without edit the code. just change the key and restart the service. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_limit_queue.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/mail-post/email_limit_queue.rb b/lib/mail-post/email_limit_queue.rb index c7e3954..62e9634 100755 --- a/lib/mail-post/email_limit_queue.rb +++ b/lib/mail-post/email_limit_queue.rb @@ -54,9 +54,9 @@ class EmailRateLimit end def change_queue(email) - return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + return unless (@redis.hget 'email_in_limit', email).to_i >= ENV['EMAIL_LIMIT_COUNT'].to_i @redis.hdel 'email_in_limit', email - @redis.hset 'email_out_limit', email, 10 + @redis.hset 'email_out_limit', email, ENV['EMAIL_LIMIT_COUNT'] end end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] container/send-internet-mail: add key EMAIL_LIMIT_COUNT
by Luan Shengde 30 Mar '21

30 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 136435d..57bf1f6 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -14,6 +14,7 @@ names = Set.new %w[ SEND_MAIL_PORT SENT_MAILDIR REDIS_PORT + EMAIL_LIMIT_COUNT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/3-code/dev-env: rename defaults vars
by Liu Yinsi 30 Mar '21

30 Mar '21

before: : {$A:=$server_ip}; $A=$A after: : {$a:=$server_ip}; $A=$a [why] when submit a job to run a testbox, sched will get variables(from /etc/compass-ci/scheduler), and export them in a running testbox env, but for selftest, it will deploy a new Compass CI, no need to use variables from the host(crystal or z9), it should be the testbox own server ip, so change defaults variables name. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 8cb9aa5..27258fc 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -10,23 +10,23 @@ server_name=$(hostname | cut -f1 -d.) : ${ES_HOST:=$server_ip} : ${ES_PORT:=9200} : ${OS_HTTP_HOST:=$server_ip} -: ${SEND_MAIL_HOST:=$server_ip} +: ${send_mail_host:=$server_ip} : ${SEND_MAIL_PORT:=10001} -: ${SRV_HTTP_RESULT_HOST:=$server_ip} -: ${SRV_HTTP_OS_HOST:=$server_ip} -: ${SRV_HTTP_GIT_HOST:=$server_ip} -: ${SRV_HTTP_CCI_HOST:=$server_ip} +: ${srv_http_result_host:=$server_ip} +: ${srv_http_os_host:=$server_ip} +: ${srv_http_git_host:=$server_ip} +: ${srv_http_cci_host:=$server_ip} : ${SRV_HTTP_RESULT_PORT:=20007} : ${SRV_HTTP_OS_PORT:=20009} : ${SRV_HTTP_GIT_PORT:=20010} : ${SRV_HTTP_CCI_PORT:=20011} : ${LOGGING_ES_HOST:=$server_ip} : ${LOGGING_ES_PORT:=9202} -: ${INITRD_HTTP_HOST:=$server_ip} +: ${initrd_http_host:=$server_ip} : ${INITRD_HTTP_PORT:=8800} -: ${ASSIST_RESULT_HOST:=$server_ip} +: ${assist_result_host:=$server_ip} : ${ASSIST_RESULT_PORT:=8102} -: ${RESULT_WEBDAV_HOST:=$server_ip} +: ${result_webdav_host:=$server_ip} : ${RESULT_WEBDAV_PORT:=3080} : ${MASTER_FLUENTD_HOST:=$server_ip} : ${MASTER_FLUENTD_PORT:=24224} @@ -34,7 +34,7 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} -: ${GIT_SERVER:=$server_ip} +: ${git_server:=$server_ip} : ${SCHED_HTTPS_HOST:=$server_ip} : ${SCHED_HTTPS_PORT:=20002} : ${DOCKER_PROXY_HOST:=$server_ip} @@ -77,26 +77,26 @@ ES_PORT: $ES_PORT SCHED_HOST: $sched_host SCHED_PORT: $sched_port LKP_SERVER: $server_ip -GIT_SERVER: $GIT_SERVER +GIT_SERVER: $git_server OS_HTTP_HOST: $OS_HTTP_HOST -SRV_HTTP_RESULT_HOST: $SRV_HTTP_RESULT_HOST -SRV_HTTP_OS_HOST: $SRV_HTTP_OS_HOST -SRV_HTTP_GIT_HOST: $SRV_HTTP_GIT_HOST -SRV_HTTP_CCI_HOST: $SRV_HTTP_CCI_HOST +SRV_HTTP_RESULT_HOST: $srv_http_result_host +SRV_HTTP_OS_HOST: $srv_http_os_host +SRV_HTTP_GIT_HOST: $srv_http_git_host +SRV_HTTP_CCI_HOST: $srv_http_cci_host SRV_HTTP_RESULT_PORT: $SRV_HTTP_RESULT_PORT SRV_HTTP_OS_PORT: $SRV_HTTP_OS_PORT SRV_HTTP_GIT_PORT: $SRV_HTTP_GIT_PORT SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT TASKQUEUE_HOST: $sched_host -SEND_MAIL_HOST: $SEND_MAIL_HOST +SEND_MAIL_HOST: $send_mail_host SEND_MAIL_PORT: $SEND_MAIL_PORT LOGGING_ES_HOST: $LOGGING_ES_HOST LOGGING_ES_PORT: $LOGGING_ES_PORT -INITRD_HTTP_HOST: $INITRD_HTTP_HOST +INITRD_HTTP_HOST: $initrd_http_host INITRD_HTTP_PORT: $INITRD_HTTP_PORT -ASSIST_RESULT_HOST: $ASSIST_RESULT_HOST +ASSIST_RESULT_HOST: $assist_result_host ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT -RESULT_WEBDAV_HOST: $RESULT_WEBDAV_HOST +RESULT_WEBDAV_HOST: $result_webdav_host RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT -- 2.23.0

1 0

[PATCH compass-ci 1/4] container/rpm-repo: add Dockerfile
by Li Ping 30 Mar '21

30 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/rpm-repo/Dockerfile diff --git a/container/rpm-repo/Dockerfile b/container/rpm-repo/Dockerfile new file mode 100644 index 0000000..ae340e8 --- /dev/null +++ b/container/rpm-repo/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM fedora:latest + +RUN yum repolist +RUN yum install -y createrepo ruby && \ + yum clean all + +RUN umask 002 && \ + gem install bunny + +COPY --chown=1090:1090 rpm-repo.rb /usr/local/bin/ +CMD ["/usr/local/bin/rpm-repo.rb"] -- 2.23.0

2 2

[PATCH compass-ci 4/4] container/rpm-repo: add start script
by Li Ping 30 Mar '21

30 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/start | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100755 container/rpm-repo/start diff --git a/container/rpm-repo/start b/container/rpm-repo/start new file mode 100755 index 0000000..3027855 --- /dev/null +++ b/container/rpm-repo/start @@ -0,0 +1,33 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + MQ_HOST + LKP_SERVER +] + +defaults = relevant_defaults(names) +defaults['MQ_HOST'] ||= defaults['LKP_SERVER'] ||= '172.17.0.1' +defaults['LKP_SERVER'] ||= '172.17.0.1' +env = docker_env(defaults) + +docker_rm "update-repodata" + +cmd = %W[ + docker run + --restart=always + --name update-repodata + -u 1090:1090 +] + env + %W[ + -d + -v /etc/localtime:/etc/localtime:ro + -v /srv/rpm/:/srv/rpm/ + fedora:update-repodata +] + +system(*cmd) -- 2.23.0

2 2

[PATCH compass-ci] container/ssh-r/start: rename docker name
by Wei Jihui 30 Mar '21

30 Mar '21

vm_name --> docker_name Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/ssh-r/start | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/ssh-r/start b/container/ssh-r/start index 68a63f2..17d4493 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -4,14 +4,14 @@ . $CCI_SRC/container/defconfig.sh -vm_name=ssh_r +docker_name=ssh_r -docker_rm ${vm_name} +#docker_rm ${docker_name} cmd=( docker run --restart=always - --name ${vm_name} + --name ${docker_name} -e GATEWAY_PORTS=true -e TCP_FORWARDING=true -d -- 2.23.0

2 1

[PATCH compass-ci] update container expose port
by Cui Yili 30 Mar '21

30 Mar '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/minio/start | 2 +- container/openresty-proxy-cache/start | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/container/minio/start b/container/minio/start index 1798cfa..2cbc9c8 100755 --- a/container/minio/start +++ b/container/minio/start @@ -7,7 +7,7 @@ docker_rm minio cmd=( docker run -d - -p 11321:9000 + -p 20014:9000 --name minio -v /etc/localtime:/etc/localtime:ro -v /srv/minio/:/srv/minio/ diff --git a/container/openresty-proxy-cache/start b/container/openresty-proxy-cache/start index a1f095d..59e2a28 100755 --- a/container/openresty-proxy-cache/start +++ b/container/openresty-proxy-cache/start @@ -11,8 +11,8 @@ cmd=( docker run -d --name openresty-proxy-cache --restart=always - -p 11333:80 - -p 11334:443 + -p 20015:80 + -p 20016:443 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf -v /srv/initrd/tar:/srv/initrd/tar:ro -v /etc/localtime:/etc/localtime:ro -- 2.23.0

1 0

[PATCH compass-ci] crystal-shards/build: delete same annotation about gitcache
by Cui Yili 30 Mar '21

30 Mar '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/crystal-shards/build | 3 --- 1 file changed, 3 deletions(-) diff --git a/container/crystal-shards/build b/container/crystal-shards/build index 00153ab..872282b 100755 --- a/container/crystal-shards/build +++ b/container/crystal-shards/build @@ -2,9 +2,6 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# GITCACHE_HOST and GITCACHE_PORT parameters come from gitcache_host and gitcache_port in the job.yaml file you submitted. -# If job.yaml does not have these two parameters, git will use the local proxy cache, -# GITCACHE_HOST defaults to the local ip, GITCACHE_PORT default port is 5000 . ../defconfig.sh docker_skip_rebuild "alpine:crystal-shards" -- 2.23.0

1 0

[PATCH compass-ci] container: docker run param -it => -d
by Cao Xueliang 30 Mar '21

30 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/etcd/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/etcd/start b/container/etcd/start index 7657b86..db94833 100755 --- a/container/etcd/start +++ b/container/etcd/start @@ -8,7 +8,7 @@ docker_rm etcd-server cmd=( docker run - -it + -d -p 2379:2379 -p 2380:2380 -p 4001:4001 -- 2.23.0

1 0

[PATCH compass-ci 2/4] container/rpm-repo: add build script
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/rpm-repo/build | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100755 container/rpm-repo/build diff --git a/container/rpm-repo/build b/container/rpm-repo/build new file mode 100755 index 0000000..458e8f6 --- /dev/null +++ b/container/rpm-repo/build @@ -0,0 +1,6 @@ +#!/bin/sh + +. ../defconfig.sh +docker_skip_rebuild "fedora:update-repodata" + +docker build -t fedora:update-repodata . -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror: fix rubocop errors
by Li Yuanchao 29 Mar '21

29 Mar '21

[error msg] git_mirror.rb:112:1: C: Metrics/ClassLength: Class has too many lines. [101/100] class MirrorMain ... ^^^^^^^^^^^^^^^^ git_mirror.rb:192:3: C: Metrics/AbcSize: Assignment Branch Condition size for handle_feedback is too high. [<4, 17, 6> 18.47/15] def handle_feedback ... ^^^^^^^^^^^^^^^^^^^ git_mirror.rb:286:3: C: Metrics/AbcSize: Assignment Branch Condition size for reload_fork_info is too high. [<5, 17, 2> 17.83/15] def reload_fork_info(upstream_repos) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ git_mirror.rb:293:121: C: Layout/LineLength: Line is too long. [149/120] changed_files = %x(git -C /srv/git/#{@git_info[upstream_repos]['belong']}/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ git_mirror.rb:328:3: C: Metrics/AbcSize: Assignment Branch Condition size for update_stat_new_refs is too high. [<6, 14, 2> 15.36/15] def update_stat_new_refs(git_repo) ... ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ git_mirror.rb:508:7: C: Naming/PredicateName: Rename is_upstream_repo? to upstream_repo?. def is_upstream_repo?(git_repo) ^^^^^^^^^^^^^^^^^ Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 45 +++++++++++++++++++++++++++++---------------- 1 file changed, 29 insertions(+), 16 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 5ba9154..c967c8b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -199,14 +199,7 @@ class MirrorMain update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - return reload_fork_info(git_repo) if is_upstream_repo?(git_repo) - - new_refs = check_new_refs(git_repo) - return if new_refs[:heads].empty? - - feedback_info[:new_refs] = new_refs - send_message(feedback_info) - new_refs_log(git_repo, new_refs[:heads].length) if last_commit_new?(git_repo) + handle_feedback_new_refs(git_repo, feedback_info) end def do_push(fork_key) @@ -283,14 +276,19 @@ class MirrorMain return new_refs end + def get_change_files(upstream_repos) + old_commit = @git_info[upstream_repos][:cur_refs][:heads]['refs/heads/master'] + new_refs = check_new_refs(upstream_repos) + new_commit = new_refs[:heads]['refs/heads/master'] + mirror_dir = "/srv/git/#{@git_info[upstream_repos]['belong']}/#{upstream_repos}.git" + %x(git -C #{mirror_dir} diff --name-only #{old_commit}...#{new_commit}) + end + def reload_fork_info(upstream_repos) if @git_info[upstream_repos][:cur_refs].empty? @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) else - old_commit = @git_info[upstream_repos][:cur_refs][:heads]['refs/heads/master'] - new_refs = check_new_refs(upstream_repos) - new_commit = new_refs[:heads]['refs/heads/master'] - changed_files = %x(git -C /srv/git/#{@git_info[upstream_repos]['belong']}/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) + changed_files = get_change_files(upstream_repos) reload(changed_files, @git_info[upstream_repos]['belong']) end end @@ -325,13 +323,17 @@ class MirrorMain @fork_stat[git_repo][:offset_fetch] = offset_fetch + 1 end + def update_new_refs_info(git_repo, offset_new_refs) + @fork_stat[git_repo][:new_refs_time][offset_new_refs] = Time.now.to_s + @fork_stat[git_repo][:offset_new_refs] = offset_new_refs + 1 + @fork_stat[git_repo][:new_refs_count] = update_new_refs_count(@fork_stat[git_repo][:new_refs_count]) + end + def update_stat_new_refs(git_repo) @fork_stat[git_repo][:priority] += 1 offset_new_refs = @fork_stat[git_repo][:offset_new_refs] offset_new_refs = 0 if offset_new_refs >= 10 - @fork_stat[git_repo][:new_refs_time][offset_new_refs] = Time.now.to_s - @fork_stat[git_repo][:offset_new_refs] = offset_new_refs + 1 - @fork_stat[git_repo][:new_refs_count] = update_new_refs_count(@fork_stat[git_repo][:new_refs_count]) + update_new_refs_info(git_repo, offset_new_refs) end def update_fork_stat(git_repo, possible_new_refs) @@ -471,6 +473,17 @@ end class MirrorMain WEEK_SECONDS = 604800 + def handle_feedback_new_refs(git_repo, feedback_info) + return reload_fork_info(git_repo) if upstream_repo?(git_repo) + + new_refs = check_new_refs(git_repo) + return if new_refs[:heads].empty? + + feedback_info[:new_refs] = new_refs + send_message(feedback_info) + new_refs_log(git_repo, new_refs[:heads].length) if last_commit_new?(git_repo) + end + def merge_defaults(object_key, object, belong) return object if object_key == belong @@ -505,7 +518,7 @@ class MirrorMain return url end - def is_upstream_repo?(git_repo) + def upstream_repo?(git_repo) @upstreams['upstreams'].each do |repo| return true if git_repo == repo['git_repo'] end -- 2.23.0

2 2

[PATCH compass-ci] container: add etcd to start-depends
by Cao Xueliang 29 Mar '21

29 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/extract-stats/start-depends | 1 + container/scheduler/start-depends | 1 + 2 files changed, 2 insertions(+) diff --git a/container/extract-stats/start-depends b/container/extract-stats/start-depends index efbe5a2..abd487d 100755 --- a/container/extract-stats/start-depends +++ b/container/extract-stats/start-depends @@ -1 +1,2 @@ taskqueue +etcd diff --git a/container/scheduler/start-depends b/container/scheduler/start-depends index 1aced31..09a0b1d 100755 --- a/container/scheduler/start-depends +++ b/container/scheduler/start-depends @@ -2,3 +2,4 @@ redis es taskqueue rabbitmq +etcd -- 2.23.0

1 0

[PATCH compass-ci] src/monitor: detect crash and send msg to mq queue.
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/monitoring/filter.cr | 2 +- src/monitoring/parse_serial_logs.cr | 50 ++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 39cfbb6..b663f11 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,7 +56,7 @@ class Filter msg = JSON.parse(msg.to_s).as_h? return unless msg - @sp.save_dmesg_to_result_root(msg) + @sp.deal_serial_log(msg) @hash.keys.each do |query| if match_query(query.as_h, msg) send_msg(query, msg) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 0589523..9430d8c 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -4,6 +4,7 @@ require "../scheduler/elasticsearch_client" require "set" require "json" +require "../lib/mq" # This parses dmesg in a stream of serial log, finding a number of patterns # in various places of the dmesg and take actions accordingly. @@ -40,9 +41,15 @@ class SerialParser "Restarting system", ] + CRASH_PATTERNS = [ + "mount.nfs: Connection timed out", + "No space left on device", + ] + def initialize @host2head = Hash(String, Array(String)).new @host2rt = Hash(String, String).new + @mq = MQClient.instance end def host_in_msg(msg) @@ -59,6 +66,14 @@ class SerialParser end end + def detect_crash(msg, host) + message = msg["message"].to_s + CRASH_PATTERNS.each do |pattern| + matched = message.match(/.*(?<crash>#{pattern})/) + return matched.named_captures["crash"] unless matched.nil? + end + end + def delete_host(msg, host, signal) boundary_signal = detect_start_or_end(msg, host, signal) return unless boundary_signal @@ -67,10 +82,43 @@ class SerialParser @host2rt.delete(host) end - def save_dmesg_to_result_root(msg) + def mq_publish(msg, host) + crash_signal = detect_crash(msg, host) + return unless crash_signal + + job_id = "" + if @host2rt.has_key?(host) + job_id = File.basename(@host2rt[host]) + end + + mq_msg = { + "job_id" => job_id, + "testbox" => host, + "time" => msg["time"]? || Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800"), + "job_state" => "crash" + } + spawn mq_publish_check("job_mq", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def deal_serial_log(msg) host = host_in_msg(msg) return unless host + mq_publish(msg, host) + save_dmesg_to_result_root(msg, host) + end + + def save_dmesg_to_result_root(msg, host) delete_host(msg, host, START_PATTERNS) check_save = check_save_dmesg(msg, host) -- 2.23.0

1 0

[PATCH compass-ci] src/lib: add on_job_crash
by Li Ping 29 Mar '21

29 Mar '21

[how] mq publish the crash event, lifecycle will deal the job event. when job_state is crash and the job is consumed, then close the job and reboot crash machine, if no job, just reboot crash machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/lifecycle.cr | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 4bd2ac0..8f34903 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -122,6 +122,8 @@ class Lifecycle on_job_close(event) when "abnormal" on_abnormal_job(event) + when "crash" + on_job_crash(event) else on_other_job(event) end @@ -190,6 +192,17 @@ class Lifecycle update_cached_machine(job["testbox"].to_s, event) end + def on_job_crash(event) + event_job_id = event["job_id"].to_s + if @jobs[event_job_id]? + @jobs.delete(event_job_id) + spawn @scheduler_api.close_job(event_job_id, "crash", "lifecycle") + end + + testbox = event["testbox"].to_s + reboot_crash_machine(testbox, event) + end + def on_job_boot(event) event_job_id = event["job_id"]?.to_s @jobs[event_job_id] = event unless event_job_id.empty? @@ -298,6 +311,22 @@ class Lifecycle spawn @scheduler_api.close_job(job_id, reason, "lifecycle") end + def reboot_crash_machine(testbox, event) + @machines.delete(testbox) + machine = @es.get_tbox(testbox) + return unless machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) + + mq_queue = get_machine_reboot_queue(testbox) + machine.as_h.delete("history") + machine.as_h["testbox"] = JSON::Any.new(testbox) + @mq.publish_confirm(mq_queue, machine.to_json, durable: true) + + machine["state"] = "rebooting_queue" + machine["time"] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") + @es.update_tbox(testbox, machine.as_h) + end + def reboot_timeout_machine(testbox) @machines.delete(testbox) machine = @es.get_tbox(testbox) -- 2.23.0

1 0

[PATCH compass-ci 4/5] container/updaterepo: add build script
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/updaterepo/build | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 container/updaterepo/build diff --git a/container/updaterepo/build b/container/updaterepo/build new file mode 100755 index 0000000..594558a --- /dev/null +++ b/container/updaterepo/build @@ -0,0 +1,14 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig.rb' + +docker_skip_rebuild "update-repo" + +CCI_SRC = ENV['CCI_SRC'] +system "bash #{CCI_SRC}/container/compile ." +system "docker build -t update-repo ." + +system 'rm updaterepo' -- 2.23.0

1 0

[PATCH compass-ci 3/5] container/updaterepo: add update-repo Dockerfile
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/updaterepo/Dockerfile | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 container/updaterepo/Dockerfile diff --git a/container/updaterepo/Dockerfile b/container/updaterepo/Dockerfile new file mode 100644 index 0000000..adab626 --- /dev/null +++ b/container/updaterepo/Dockerfile @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:scheduler-dev + +MAINTAINER Li Ping <1477412247(a)qq.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + +RUN apk update && \ + apk add --no-cache 'ruby-dev' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'cpio' 'bash' + +ENV UPDATE_REPO_DIR /c/cci/update-repo + +RUN mkdir -p $UPDATE_REPO_DIR && \ + chown -R 1090:1090 /c + +WORKDIR $UPDATE_REPO_DIR + +COPY --chown=1090:1090 updaterepo . -- 2.23.0

1 0

[PATCH compass-ci 1/5] add update-repo service
by Li Ping 29 Mar '21

29 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/web_env.cr | 9 +++++++++ src/updaterepo.cr | 17 +++++++++++++++++ src/updaterepo/constants.cr | 4 ++++ src/updaterepo/updaterepo.cr | 16 ++++++++++++++++ 4 files changed, 46 insertions(+) create mode 100644 src/updaterepo.cr create mode 100644 src/updaterepo/constants.cr create mode 100644 src/updaterepo/updaterepo.cr diff --git a/src/lib/web_env.cr b/src/lib/web_env.cr index b3a90a8..d47e7e5 100644 --- a/src/lib/web_env.cr +++ b/src/lib/web_env.cr @@ -4,6 +4,7 @@ require "./sched" require "./lifecycle" require "./json_logger" +require "./updaterepo" class HTTP::Server # Instances of this class are passed to an `HTTP::Server` handler. @@ -31,5 +32,13 @@ class HTTP::Server def create_lifecycle @lifecycle = Lifecycle.new(self) end + + def repo + @repo ||= create_repo + end + + def create_repo + @repo = Repo.new(self) + end end end diff --git a/src/updaterepo.cr b/src/updaterepo.cr new file mode 100644 index 0000000..05fa4fe --- /dev/null +++ b/src/updaterepo.cr @@ -0,0 +1,17 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "./updaterepo/updaterepo" +require "./updaterepo/constants" +require "./lib/json_logger" + + +module Updaterepo + log = JSONLogger.new + + begin + Kemal.run(REPO_PORT) + rescue e + log.error(e) + end +end diff --git a/src/updaterepo/constants.cr b/src/updaterepo/constants.cr new file mode 100644 index 0000000..c9daaf2 --- /dev/null +++ b/src/updaterepo/constants.cr @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +REPO_PORT = (ENV.has_key?("REPO_PORT") ? ENV["REPO_PORT"] : 8132).to_i32 diff --git a/src/updaterepo/updaterepo.cr b/src/updaterepo/updaterepo.cr new file mode 100644 index 0000000..7ad9d20 --- /dev/null +++ b/src/updaterepo/updaterepo.cr @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "kemal" + +require "../lib/web_env" +require "../lib/updaterepo" +require "../lib/json_logger" + +module Updaterepo + VERSION = "0.1.0" + post "/upload" do |env| + env.repo.upload_repo + "done" + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 3/3] sparrow/3-code/dev-env: add key EMAIL_LIMIT_COUNT
by Luan Shengde 29 Mar '21

29 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f96972e..3123c2a 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -12,6 +12,7 @@ server_name=$(hostname | cut -f1 -d.) : ${OS_HTTP_HOST:=$server_ip} : ${SEND_MAIL_HOST:=$server_ip} : ${SEND_MAIL_PORT:=10001} +: ${EMAIL_LIMIT_COUNT:=10} : ${SRV_HTTP_RESULT_HOST:=$server_ip} : ${SRV_HTTP_OS_HOST:=$server_ip} : ${SRV_HTTP_GIT_HOST:=$server_ip} @@ -91,6 +92,7 @@ SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT TASKQUEUE_HOST: $sched_host SEND_MAIL_HOST: $SEND_MAIL_HOST SEND_MAIL_PORT: $SEND_MAIL_PORT +EMAIL_LIMIT_COUNT: ${EMAIL_LIMIT_COUNT} LOGGING_ES_HOST: $LOGGING_ES_HOST LOGGING_ES_PORT: $LOGGING_ES_PORT INITRD_HTTP_HOST: $INITRD_HTTP_HOST -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] lib/mail-post/email_limit_queue: use ENV instead of static value
by Luan Shengde 29 Mar '21

29 Mar '21

use ENV variable instead of static value. enable user-defined new value without edit the code. just change the key and restart the service. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_limit_queue.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/mail-post/email_limit_queue.rb b/lib/mail-post/email_limit_queue.rb index c7e3954..62e9634 100755 --- a/lib/mail-post/email_limit_queue.rb +++ b/lib/mail-post/email_limit_queue.rb @@ -54,9 +54,9 @@ class EmailRateLimit end def change_queue(email) - return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + return unless (@redis.hget 'email_in_limit', email).to_i >= ENV['EMAIL_LIMIT_COUNT'].to_i @redis.hdel 'email_in_limit', email - @redis.hset 'email_out_limit', email, 10 + @redis.hset 'email_out_limit', email, ENV['EMAIL_LIMIT_COUNT'] end end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/send-internet-mail: add key EMAIL_LIMIT_COUNT
by Luan Shengde 29 Mar '21

29 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 136435d..57bf1f6 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -14,6 +14,7 @@ names = Set.new %w[ SEND_MAIL_PORT SENT_MAILDIR REDIS_PORT + EMAIL_LIMIT_COUNT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/send-internet-mail: add key EMAIL_LIMIT_COUNT for start
by Luan Shengde 29 Mar '21

29 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 136435d..57bf1f6 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -14,6 +14,7 @@ names = Set.new %w[ SEND_MAIL_PORT SENT_MAILDIR REDIS_PORT + EMAIL_LIMIT_COUNT ] defaults = relevant_defaults(names) -- 2.23.0

2 2

[PATCH compass-ci 7/7] scheduler: add get_new_seqno in redis client
by Cao Xueliang 29 Mar '21

29 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/redis_client.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/redis_client.cr b/src/scheduler/redis_client.cr index 030d1b7..ed2af6e 100644 --- a/src/scheduler/redis_client.cr +++ b/src/scheduler/redis_client.cr @@ -70,4 +70,8 @@ class Redis::Client def remove_finished_job(job_id : String) @client.hdel("sched/id2job", job_id) end + + def get_new_seqno + return @client.incr("#{QUEUE_NAME_BASE}/seqno") + end end -- 2.23.0

1 0

[PATCH compass-ci 6/7] scheduler: update job parameter in etcd
by Cao Xueliang 29 Mar '21

29 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 7 ------- src/scheduler/update_job_parameter.cr | 9 ++++++++- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index f74df9b..6fd6657 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -166,13 +166,6 @@ class Sched @es.update_tbox(testbox.to_s, hash) end - def update_testbox_time(job_id) - job = @redis.get_job(job_id) - testbox = job["testbox"] - hash = {"time" => @env.get?("time").to_s} - @es.update_tbox(testbox, hash) - end - def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index 92b905e..28ab0f2 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -27,7 +27,7 @@ class Sched job_content[parameter] = value end - @redis.update_job(job_content) + update_id2job(job_content) # json log log = job_content.dup @@ -46,4 +46,11 @@ class Sched } spawn mq_publish_confirm(JOB_MQ, mq_msg.to_json) end + + def update_testbox_time(job_id) + job = get_id2job(job_id) + testbox = job["testbox"] + hash = {"time" => @env.get?("time").to_s} + @es.update_tbox(testbox, hash) + end end -- 2.23.0

1 0

[PATCH compass-ci 5/7] scheduler: move job from in_process to extract_stats in etcd
by Cao Xueliang 29 Mar '21

29 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/close_job.cr | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index a57b233..746f70a 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -8,7 +8,7 @@ class Sched @env.set "job_id", job_id - job = @redis.get_job(job_id) + job = get_id2job(job_id) # update job_state job_state = @env.params.query["job_state"]? @@ -21,17 +21,8 @@ class Sched raise "es set job content fail!" end - subqueue = job.subqueue - queue = (subqueue == "idle" ? job.queue : "#{job.queue}/#{subqueue}") - - response = @task_queue.hand_over_task( - "sched/#{queue}", "extract_stats", job_id - ) - if response[0] != 201 - raise "#{response}" - end - - @redis.remove_finished_job(job_id) + move_process2stats(job) + delete_id2job(job.id) job_state ||= "complete" @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) -- 2.23.0

1 0

[PATCH compass-ci 4/7] scheduler: use etcd list-watch to consume job
by Cao Xueliang 29 Mar '21

29 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 113 +++++++++++++++++++++++++++------ 1 file changed, 92 insertions(+), 21 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 78b10ad..2596f83 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -69,42 +69,113 @@ class Sched default_queues = [] of String queues_str.split(',', remove_empty: true) do |item| - default_queues << item.strip + default_queues << "sched/#{item.strip}/ready" end - sub_queues = [] of String - default_queues.each do |queue| - # TODO: this could be high cost and should be improved in future : keys(pattern). - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") - next if matched_queues.empty? + return default_queues.uniq + end - matched_queues.each do |mq| - match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) - sub_queues << match_data[1] if match_data + def get_job_from_queues(queues, testbox) + job = nil + etcd_job = consume_job(queues) + job_id = etcd_job.key.split("/")[-1] + puts "#{testbox} got the job #{job_id}" + if job_id + begin + job = @es.get_job(job_id.to_s) + rescue ex + @log.warn("Invalid job (id=#{job_id}) in es. Info: #{ex}") + @log.warn(ex.inspect_with_backtrace) end end - idle_queues = [] of String - delimiter_queues = [] of String - default_queues.each do |queue| - idle_queues << "#{queue}/idle" - delimiter_queues << "#{queue}/delimiter@localhost" + if job + job.update({"testbox" => testbox}) + job.set_result_root + @log.info(%({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state" +: "set result root"})) + set_id2job(job) end - all_queues = delimiter_queues + rand_queues(sub_queues) + idle_queues + return job + end + + def consume_job(queues) + job, revision = consume_by_list(queues) + return job if job - return all_queues.uniq + consume_by_watch(queues, revision) end - def get_job_from_queues(queues, testbox) - job = nil + def consume_by_list(queues) + jobs, revision = get_history_jobs(queues) + jobs.shuffle! + while true + return nil, revision if jobs.empty? + + job = jobs.delete_at(0) + return job, revision if ready2process(job) + end + end + def get_history_jobs(queues) + revisions = [] of Int64 + ec = EtcdClient.new + jobs = [] of Etcd::Model::Kv queues.each do |queue| - job = prepare_job("sched/#{queue}", testbox) - return job if job + job = ec.range_prefix(queue) + revisions << job.header.not_nil!.revision + jobs += job.kvs end - return job + return jobs, revisions.min + end + + def consume_by_watch(queues, revision) + ready_queues = split_ready_queues(queues) + channel = watch_queues(ready_queues, revision) + loop_handle_event(channel) + end + + def split_ready_queues(queues) + ready_queues = [] of String + queues.each do |queue| + tmp = queue.split("/ready")[0] + ready_queues << "#{tmp}/ready" + end + + ready_queues.uniq + end + + def watch_queues(queues, revision) + puts "watch #{queues}, revision #{revision}" + channel = Channel(Array(Etcd::Model::WatchEvent)).new + queues.each do |queue| + watcher = EtcdClient.new.watch_prefix(queue, start_revision: revision.to_i64, progress_notify: false ,filters: [Etcd::Watch::Filter::NODELETE]) do |events| + channel.send(events) + end + spawn { watcher.start } + Fiber.yield + end + + return channel + end + + def loop_handle_event(channel) + while true + events = channel.receive + events.each do |event| + return event.kv if ready2process(event.kv) + end + end + end + + def ready2process(job) + ec = EtcdClient.new + f_queue = job.key + t_queue = f_queue.gsub("/ready/", "/in_process/") + value = job.value + ec.move(f_queue, t_queue, value) end def get_job_boot(host, boot_type) -- 2.23.0

1 0

[PATCH compass-ci 3/7] scheduler: submit job to etcd db
by Cao Xueliang 29 Mar '21

29 Mar '21

before, we add job to taskqueue, then we save job to es, after, we save job to es at first, then we add job to etcd. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/submit_job.cr | 64 +++++++++++++++++++------------------ 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index fe9efa1..c639ad7 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -58,18 +58,12 @@ class Sched next if (config["roles"].as_a.map(&.to_s) & roles).empty? queue = host.to_s - job_id = add_task("#{queue}/#{subqueue}", lab) + job_id = get_job_id(lab) # return when job_id is '0' # 2 Questions: # - how to deal with the jobs added to DB prior to this loop # - may consume job before all jobs done - return job_messages << { - "job_id" => "0", - "message" => "add task queue sched/#{queue} failed", - "job_state" => "submit", - } unless job_id - job_ids << job_id # add to job content when multi-test @@ -87,15 +81,15 @@ class Sched job["direct_macs"] = direct_macs.join(" ") job["direct_ips"] = direct_ips.join(" ") - response = add_job(job, job_id) - message = (response["error"]? ? response["error"]["root_cause"] : "") + status, msg = add_job(job, job_id) + job_id = "0" unless status job_messages << { "job_id" => job_id, - "message" => message.to_s, + "message" => msg, "job_state" => "submit", "result_root" => "/srv#{job.result_root}", } - return job_messages if response["error"]? + return job_messages unless status end cluster_id = job_ids[0] @@ -117,31 +111,31 @@ class Sched # success: [{"job_id" => job_id, "message" => "", job_state => "submit"}] # failure: [{"job_id" => "0", "message" => err_msg, job_state => "submit"}] def submit_single_job(job) - queue = "#{job.queue}/#{job.subqueue}" + job_id = get_job_id(job.lab) - job_id = add_task(queue, job.lab) + status, msg = add_job(job, job_id) + job_id = "0" unless status return [{ - "job_id" => "0", - "message" => "add task queue sched/#{queue} failed", + "job_id" => job_id, + "message" => msg, "job_state" => "submit", - }] unless job_id - - response = add_job(job, job_id) - message = (response["error"]? ? response["error"]["root_cause"] : "") - - return [{ - "job_id" => job_id, - "message" => message.to_s, - "job_state" => "submit", "result_root" => "/srv#{job.result_root}", }] end - # return job_id - def add_task(queue, lab) - task_desc = JSON.parse(%({"domain": "compass-ci", "lab": "#{lab}"})) - response = @task_queue.add_task("sched/#{queue}", task_desc) - JSON.parse(response[1].to_json)["id"].to_s if response[0] == 200 + def add_task(job, job_id) + task_desc = JSON.parse(%({"domain": "compass-ci", "id": "#{job_id}"})) + key = "sched/#{job.queue}/ready/#{job.subqueue}/#{job_id}" + response = @etcd.put(key, task_desc) + if response + return true, "" + else + return false, "add the job to queue failed: job_id #{job_id}, queue #{key}" + end + end + + def get_job_id(lab) + "#{lab}.#{(a)redis.get_new_seqno()}" end def save_secrets(job, job_id) @@ -150,11 +144,19 @@ class Sched @redis.hash_set("id2secrets", job_id, job["secrets"]?.to_json) job.delete("secrets") end - # add job content to es and return a response + def add_job(job, job_id) + # deal the job fields at first save_secrets(job, job_id) job.update_id(job_id) - @es.set_job_content(job) + + # save the job to es + response = @es.set_job_content(job) + msg = (response["error"]? ? response["error"]["root_cause"] : "") + return false, msg.to_s if response["error"]? + + # set job_id to ready queue + add_task(job, job_id) end def get_cluster_config(cluster_file, lkp_initrd_user, os_arch) -- 2.23.0

1 0

[PATCH compass-ci 2/7] extract-stats: use etcd list-watch to handle extract-stats queue
by Cao Xueliang 29 Mar '21

29 Mar '21

before, we use taskqueue to handle extract-stats queue, after, we use etcd list-watch to handle extract-stats queue. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats.cr | 2 +- src/extract-stats/extract_stats.cr | 67 +++++++++++++++++++------ src/extract-stats/stats_worker.cr | 78 +++++++++--------------------- 3 files changed, 75 insertions(+), 72 deletions(-) diff --git a/src/extract-stats.cr b/src/extract-stats.cr index 2f4b97d..0908a91 100644 --- a/src/extract-stats.cr +++ b/src/extract-stats.cr @@ -5,4 +5,4 @@ require "./extract-stats/extract_stats" # results data post processing -ExtractStats.in_extract_stats +ExtractStats.consume_tasks diff --git a/src/extract-stats/extract_stats.cr b/src/extract-stats/extract_stats.cr index 45310ad..4ae3e71 100644 --- a/src/extract-stats/extract_stats.cr +++ b/src/extract-stats/extract_stats.cr @@ -1,30 +1,67 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +require "../lib/etcd_client" require "./constants" require "./stats_worker" module ExtractStats - # Consume scheduler queue - def self.in_extract_stats - back_fill_task - STATS_WORKER_COUNT.times do - Process.fork do - consume_task + def self.consume_tasks + channel = Channel(String).new + revision = self.consume_by_list(EXTRACT_STATS_QUEUE_PATH, channel) + self.consume_by_watch(EXTRACT_STATS_QUEUE_PATH, revision, channel) + end + + def self.consume_by_list(queue, channel) + tasks, revision = self.get_history_tasks(queue) + self.handle_history_tasks(tasks, channel) + + return revision + end + + def self.consume_by_watch(queue, revision, channel) + self.watch_queue(queue, revision, channel) + self.handle_events(channel) + end + + def self.get_history_tasks(queue) + tasks = [] of Etcd::Model::Kv + range = EtcdClient.new.range_prefix(queue) + revision = range.header.not_nil!.revision + tasks += range.kvs + + return tasks, revision + end + + def self.handle_history_tasks(tasks, channel) + while true + return if tasks.empty? + + task = tasks.delete_at(0) + spawn { StatsWorker.new.handle(task.key, channel) } + + sleep 1 + end + end + + def self.watch_queue(queue, revision, channel) + watcher = EtcdClient.new.watch_prefix(queue, start_revision: revision.to_i64, filters: [Etcd::Watch::Filter::NODELETE]) do |events| + events.each do |event| + puts event + channel.send(event.kv.key) end end - # keep main-process alive - sleep + spawn { watcher.start } + Fiber.yield end - def self.consume_task - worker = StatsWorker.new - worker.consume_sched_queue(EXTRACT_STATS_QUEUE_PATH) - end + def self.handle_events(channel) + while true + key = channel.receive + spawn { StatsWorker.new.handle(key, channel) } - def self.back_fill_task - worker = StatsWorker.new - worker.back_fill_task(EXTRACT_STATS_QUEUE_PATH) + sleep 1 + end end end diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 44c327d..5e0100e 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -require "../lib/taskqueue_api" +require "../lib/etcd_client" require "../scheduler/elasticsearch_client" require "../scheduler/redis_client" require "../scheduler/constants" @@ -11,40 +11,25 @@ require "./constants.cr" class StatsWorker def initialize @es = Elasticsearch::Client.new - @tq = TaskQueueAPI.new + @etcd = EtcdClient.new @rc = RegressionClient.new end - def consume_sched_queue(queue_path : String) - loop do - begin - job_id = get_job_id(queue_path) - if job_id # will consume the job by post-processing - job = @es.get_job_content(job_id) - result_root = job["result_root"]? - result_post_processing(job_id, result_root.to_s, queue_path) - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - end - rescue e - STDERR.puts e.message - error_message = e.message - - # incase of many error message when task-queue, ES does not work - sleep(10) - @tq = TaskQueueAPI.new if error_message && error_message.includes?("3060': Connection refused") - end - end - end - - # get job_id from task-queue - def get_job_id(queue_path : String) - response = @tq.consume_task(queue_path) - if response[0] == 200 - JSON.parse(response[1].to_json)["id"].to_s - else - # will sleep 60s if no task in task-queue - sleep(60) - nil + def handle(queue_path, channel) + begin + res = @etcd.range(queue_path) + return nil if res.count == 0 + + job_id = queue_path.split("/")[-1] + job = @es.get_job_content(job_id) + result_root = job["result_root"]? + result_post_processing(job_id, result_root.to_s, queue_path) + @etcd.delete(queue_path) + rescue e + channel.send(queue_path) + STDERR.puts e.message + # incase of many error message when task-queue, ES does not work + sleep(10) end end @@ -61,10 +46,7 @@ class StatsWorker def store_stats_es(result_root : String, job_id : String, queue_path : String) stats_path = "#{result_root}/stats.json" - unless File.exists?(stats_path) - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - raise "#{stats_path} file not exists." - end + return unless File.exists?(stats_path) stats = File.open(stats_path) do |file| JSON.parse(file) @@ -81,7 +63,6 @@ class StatsWorker { :index => "jobs", :type => "_doc", :id => job_id, - :refresh => "wait_for", :body => {:doc => update_content}, } ) @@ -90,9 +71,10 @@ class StatsWorker unless new_error_ids.empty? sample_error_id = new_error_ids.sample STDOUT.puts "send a delimiter task: job_id is #{job_id}" - @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => sample_error_id, - "job_id" => job_id, - "lab" => LAB}.to_json)) + queue = "#{DELIMITER_TASK_QUEUE}/#{job_id}" + value = {"error_id" => sample_error_id} + @etcd.put(queue, value) + msg = %({"job_id": "#{job_id}", "new_error_id": "#{sample_error_id}"}) system "echo '#{msg}'" end @@ -130,20 +112,4 @@ class StatsWorker end error_ids end - - def back_fill_task(queue_path) - redis_client = Redis::Client.new - # this queue may have leftover task_ids - queue_name = "queues/#{queue_path}/in_process" - begin - job_ids = redis_client.(a)client.zrange(queue_name, 0, -1) - return if job_ids.empty? - - job_ids.each do |job_id| - @tq.hand_over_task(queue_path, queue_path, job_id.to_s) - end - rescue e - STDERR.puts e.message - end - end end -- 2.23.0

1 0

[PATCH compass-ci 1/7] src/scheduler: add opt job in etcd
by Cao Xueliang 29 Mar '21

29 Mar '21

Before, we operate the job in redis. After, we operate the job in etcd. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/sched.cr | 3 +++ src/scheduler/opt_job_in_etcd.cr | 36 ++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) create mode 100644 src/scheduler/opt_job_in_etcd.cr diff --git a/src/lib/sched.cr b/src/lib/sched.cr index b10a16d..f74df9b 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -7,6 +7,7 @@ require "yaml" require "./mq" require "./job" require "./web_env" +require "./etcd_client" require "./block_helper" require "./taskqueue_api" require "./remote_git_client" @@ -23,6 +24,7 @@ require "../scheduler/request_cluster_state" require "../scheduler/update_job_parameter" require "../scheduler/create_job_cpio" require "../scheduler/download_file" +require "../scheduler/opt_job_in_etcd" class Sched property es @@ -37,6 +39,7 @@ class Sched @redis = Redis::Client.instance @mq = MQClient.instance @task_queue = TaskQueueAPI.new + @etcd = EtcdClient.new @rgc = RemoteGitClient.new @env = env @log = env.log.as(JSONLogger) diff --git a/src/scheduler/opt_job_in_etcd.cr b/src/scheduler/opt_job_in_etcd.cr new file mode 100644 index 0000000..f38433a --- /dev/null +++ b/src/scheduler/opt_job_in_etcd.cr @@ -0,0 +1,36 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Sched + def update_id2job(job_content) + id = job_content["id"].to_s + job = get_id2job(id) + job.update(job_content) + @etcd.update("sched/id2job/#{id}", job.dump_to_json) + end + + def set_id2job(job : Job) + @etcd.put("sched/id2job/#{job.id}", job.dump_to_json) + end + + def get_id2job(id) + response = @etcd.range("sched/id2job/#{id}") + raise "get job from etcd failed. id: #{id}" unless response.count == 1 + Job.new(JSON.parse(response.kvs[0].value.not_nil!), id) + end + + def delete_id2job(id) + @etcd.delete("sched/id2job/#{id}") + end + + def update_tbox_wtmp(testbox, wtmp_hash) + @etcd.update("sched/tbox_wtmp/#{testbox}", wtmp_hash) + end + + def move_process2stats(job : Job) + f_queue = "sched/#{job.queue}/in_process/#{job.subqueue}/#{job.id}" + t_queue = "extract_stats/#{job.id}" + value = { "id" => "#{job.id}" } + @etcd.move(f_queue, t_queue, value) + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/4-docker/buildall: build and start all in parallel to speedup
by Liu Yinsi 29 Mar '21

29 Mar '21

before: function build_depends() are parallel, but do_one_build(), start_depends(), do_one_run() are not parallel. after: function build_depends(), do_one_build(), start_depends(), do_one_run() all are parallel. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 3cd5719..3611158 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -24,7 +24,7 @@ do_one_build() { local container=$1 local container_name=$(basename $container) - lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock + lockfile-create -q --use-pid --retry 100 --lock-name "build_${container_name}".lock mkdir $tmpdir/$container_name 2>/dev/null && ( @@ -33,7 +33,7 @@ do_one_build() [ "$?" == 0 ] && log_info "finish build $container" [ -x install ] && ./install ) - lockfile-remove --lock-name "$container_name".lock + lockfile-remove --lock-name "build_${container_name}".lock } start_depends() @@ -42,8 +42,9 @@ start_depends() for dep in $(cat $container/start-depends 2> /dev/null) do - start_depends $CONTAINER_PATH/$dep + start_depends $CONTAINER_PATH/$dep & done + wait do_one_run $container } @@ -52,6 +53,7 @@ do_one_run() { local container=$1 local container_name=$(basename $container) + lockfile-create -q --use-pid --retry 100 --lock-name "start_${container_name}".lock mkdir $tmpdir/start_$container_name 2>/dev/null && ( @@ -64,14 +66,31 @@ do_one_run() [ -x start ] && ./start [ "$container_name" == 'lkp-initrd' ] && ./run ) + lockfile-remove --lock-name "start_${container_name}".lock +} + +build_all() +{ + for dir in $CONTAINER_PATH/*/ + do + build_depends $dir & + done + wait } +start_all() +{ + for dir in $CONTAINER_PATH/*/ + do + start_depends $dir & + done + wait +} + + tmpdir=$(mktemp -d) -for dir in $CONTAINER_PATH/*/ -do - [ "$action" != "start-only" ] && build_depends $dir - start_depends $dir -done +[ "$action" == "start-only" ] || build_all +start_all rm -fr $tmpdir -- 2.23.0

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: build and start all in parallel to speedup
by Liu Yinsi 29 Mar '21

29 Mar '21

before: function build_depends() are parallel, but do_one_build(), start_depends(), do_one_run() are not parallel. after: function build_depends(), do_one_build(), start_depends(), do_one_run() all are parallel. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 3cd5719..f6b9b65 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -24,7 +24,7 @@ do_one_build() { local container=$1 local container_name=$(basename $container) - lockfile-create -q --use-pid --retry 100 --lock-name "$container_name".lock + lockfile-create -q --use-pid --retry 100 --lock-name "build_${container_name}".lock mkdir $tmpdir/$container_name 2>/dev/null && ( @@ -33,7 +33,7 @@ do_one_build() [ "$?" == 0 ] && log_info "finish build $container" [ -x install ] && ./install ) - lockfile-remove --lock-name "$container_name".lock + lockfile-remove --lock-name "build_${container_name}".lock } start_depends() @@ -42,8 +42,9 @@ start_depends() for dep in $(cat $container/start-depends 2> /dev/null) do - start_depends $CONTAINER_PATH/$dep + start_depends $CONTAINER_PATH/$dep & done + wait do_one_run $container } @@ -52,6 +53,7 @@ do_one_run() { local container=$1 local container_name=$(basename $container) + lockfile-create -q --use-pid --retry 100 --lock-name "start_${container_name}".lock mkdir $tmpdir/start_$container_name 2>/dev/null && ( @@ -64,14 +66,31 @@ do_one_run() [ -x start ] && ./start [ "$container_name" == 'lkp-initrd' ] && ./run ) + lockfile-remove --lock-name "start_${container_name}".lock +} + +build_all() +{ + for dir in $CONTAINER_PATH/*/ + do + build_depends $dir & + done + wait } +start_all() +{ + for dir in $CONTAINER_PATH/*/ + do + start_depends $dir & + done + wait +} + + tmpdir=$(mktemp -d) -for dir in $CONTAINER_PATH/*/ -do - [ "$action" != "start-only" ] && build_depends $dir - start_depends $dir -done +[ "$action" != "start-only" ] && build_all +start_all rm -fr $tmpdir -- 2.23.0

3 3

[PATCH compass-ci] remove gitchche env
by Liu Yinsi 29 Mar '21

29 Mar '21

refer patch: container/gitcache: delete gitcache docker that is no longer in use Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/crystal-shards/build-depends | 1 - sparrow/3-code/dev-env | 4 ---- 2 files changed, 5 deletions(-) diff --git a/container/crystal-shards/build-depends b/container/crystal-shards/build-depends index 6eed7ec..6627b6f 100755 --- a/container/crystal-shards/build-depends +++ b/container/crystal-shards/build-depends @@ -1,2 +1 @@ crystal-base -gitcache diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f96972e..c42fba5 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -78,8 +78,6 @@ SCHED_PORT: $sched_port LKP_SERVER: $server_ip GIT_SERVER: $GIT_SERVER OS_HTTP_HOST: $OS_HTTP_HOST -GITCACHE_HOST: $gitcache_host -GITCACHE_PORT: $gitcache_port SRV_HTTP_RESULT_HOST: $SRV_HTTP_RESULT_HOST SRV_HTTP_OS_HOST: $SRV_HTTP_OS_HOST SRV_HTTP_GIT_HOST: $SRV_HTTP_GIT_HOST @@ -125,8 +123,6 @@ mkdir -p /etc/compass-ci/scheduler cat > /etc/compass-ci/scheduler/local-testbox-env.yaml <<-EOF LKP_SERVER: GIT_SERVER: -GITCACHE_HOST: -GITCACHE_PORT: SRV_HTTP_RESULT_HOST: SRV_HTTP_RESULT_PORT: SRV_HTTP_OS_HOST: -- 2.23.0

1 0

[PATCH v1 compass-ci] container: fix library libgc.so.1: No such file or directory
by Lu Weitao 29 Mar '21

29 Mar '21

use the compile env as the runtime env in extract-stats and taskqueue container Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/extract-stats/Dockerfile | 2 +- container/taskqueue/Dockerfile | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/extract-stats/Dockerfile b/container/extract-stats/Dockerfile index 3a8c8f3..08abc8c 100644 --- a/container/extract-stats/Dockerfile +++ b/container/extract-stats/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:scheduler-dev RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ diff --git a/container/taskqueue/Dockerfile b/container/taskqueue/Dockerfile index 28ac925..2d5b10a 100644 --- a/container/taskqueue/Dockerfile +++ b/container/taskqueue/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:scheduler-dev RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories -- 2.23.0

1 0

[PATCH compass-ci 2/2] lib/mail-post/email_limit_queue: use ENV instead of static value
by Luan Shengde 29 Mar '21

29 Mar '21

use ENV variable instead of static value. enable user-defined new value without edit the code. just restart the service. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post/email_limit_queue.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/mail-post/email_limit_queue.rb b/lib/mail-post/email_limit_queue.rb index c7e3954..62e9634 100755 --- a/lib/mail-post/email_limit_queue.rb +++ b/lib/mail-post/email_limit_queue.rb @@ -54,9 +54,9 @@ class EmailRateLimit end def change_queue(email) - return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + return unless (@redis.hget 'email_in_limit', email).to_i >= ENV['EMAIL_LIMIT_COUNT'].to_i @redis.hdel 'email_in_limit', email - @redis.hset 'email_out_limit', email, 10 + @redis.hset 'email_out_limit', email, ENV['EMAIL_LIMIT_COUNT'] end end -- 2.23.0

1 0

[PATCH compass-ci] container/gitcache: delete gitcache docker that is no longer in use
by Cui Yili 28 Mar '21

28 Mar '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/gitcache/Dockerfile | 15 --------------- container/gitcache/build | 10 ---------- container/gitcache/start | 20 -------------------- 3 files changed, 45 deletions(-) delete mode 100644 container/gitcache/Dockerfile delete mode 100755 container/gitcache/build delete mode 100755 container/gitcache/start diff --git a/container/gitcache/Dockerfile b/container/gitcache/Dockerfile deleted file mode 100644 index d92a141..0000000 --- a/container/gitcache/Dockerfile +++ /dev/null @@ -1,15 +0,0 @@ -# Origin: https://github.com/git-cloner/gitcache -# Copyright (C) 2019-2020 Eric D.little51 -# SPDX-License-Identifier: GPL-3.0 - -FROM golang:alpine - -ENV GO111MODULE on -ENV GOPROXY https://goproxy.cn - - -RUN apk add --no-cache git && \ - git clone https://github.com/git-cloner/gitcache /gitcache && \ - cd /gitcache && go build - -ENTRYPOINT ["/gitcache/gitcache"] diff --git a/container/gitcache/build b/container/gitcache/build deleted file mode 100755 index d7f3523..0000000 --- a/container/gitcache/build +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/bash -# Origin: https://github.com/git-cloner/gitcache -# Copyright (C) 2019-2020 Eric D.little51 -# SPDX-License-Identifier: GPL-3.0 - -. ../defconfig.sh - -docker_skip_rebuild "gitcache" - -docker build -t gitcache . diff --git a/container/gitcache/start b/container/gitcache/start deleted file mode 100755 index 0e1de1b..0000000 --- a/container/gitcache/start +++ /dev/null @@ -1,20 +0,0 @@ -#!/bin/bash -# Origin: https://github.com/git-cloner/gitcache -# Copyright (C) 2019-2020 Eric D.little51 -# SPDX-License-Identifier: GPL-3.0 - -. $CCI_SRC/container/defconfig.sh - -docker_rm gitcache - -cmd=( - docker run -dt - --name gitcache - --restart=always - -p 5000:5000 - -v /srv/cache/gitcache/:/var/gitcache/ - -v /etc/localtime:/etc/localtime:ro - gitcache -) - -"${cmd[@]}" -- 2.23.0

2 1

[PATCH compass-ci] container: add etcd container
by Cao Xueliang 28 Mar '21

28 Mar '21

add etcd container Dockerfile, build and start script. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/etcd/Dockerfile | 16 ++++++++++++++++ container/etcd/build | 12 ++++++++++++ container/etcd/start | 28 ++++++++++++++++++++++++++++ 3 files changed, 56 insertions(+) create mode 100644 container/etcd/Dockerfile create mode 100755 container/etcd/build create mode 100755 container/etcd/start diff --git a/container/etcd/Dockerfile b/container/etcd/Dockerfile new file mode 100644 index 0000000..3654fc4 --- /dev/null +++ b/container/etcd/Dockerfile @@ -0,0 +1,16 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.13 + +MAINTAINER Cao Xueliang <caoxl78320(a)163.com> + +ARG VERSION +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && apk add ca-certificates && \ + wget https://github.com/etcd-io/etcd/releases/download/${VERSION}/etcd-${VERSION… && \ + tar zxvf etcd-${VERSION}-linux-arm64.tar.gz && \ + mv etcd-${VERSION}-linux-arm64/etcd* /bin/ && \ + rm -Rf etcd-${VERSION}-linux-arm64* /var/cache/apk/* + +VOLUME /data diff --git a/container/etcd/build b/container/etcd/build new file mode 100755 index 0000000..6ae25dd --- /dev/null +++ b/container/etcd/build @@ -0,0 +1,12 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ../defconfig.sh + +version=v3.4.15 +docker_skip_rebuild "etcd-${version}:alpine" + +docker build -t etcd-${version}:alpine --build-arg VERSION=${version} . + + diff --git a/container/etcd/start b/container/etcd/start new file mode 100755 index 0000000..7657b86 --- /dev/null +++ b/container/etcd/start @@ -0,0 +1,28 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +docker_rm etcd-server + +cmd=( + docker run + -it + -p 2379:2379 + -p 2380:2380 + -p 4001:4001 + -p 7001:7001 + -v /srv/etcd/dir:/data + -e ETCD_UNSUPPORTED_ARCH=arm64 + --name etcd-server + etcd-v3.4.15:alpine + /bin/etcd + --name node-1 + --data-dir /data + --initial-cluster-state new + --listen-client-urls http://0.0.0.0:2379 + --advertise-client-urls http://0.0.0.0:2379 +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] 0-package/os/openEuler: update lockfile rpm download URL
by Liu Yinsi 27 Mar '21

27 Mar '21

The old URL is invalid. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/openEuler | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 61a37c4..9cb7b04 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -28,8 +28,8 @@ pkgs=( yum install -y --skip-broken "${pkgs[@]}" -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… +wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… +wget http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm rm -rf *.rpm -- 2.23.0

1 0

[PATCH compass-ci] container/web-backend: fix SyntaxError)
by Lu Weitao 27 Mar '21

27 Mar '21

[Why] Traceback (most recent call last): 2: from ./web-backend:9:in `<main>' 1: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require' /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require': /home/lwt/compass-ci/src/lib/web_backend.rb:681: syntax error, unexpected end-of-input, expecting keyword_end (SyntaxError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 1 + src/lib/web_backend.rb | 1 + 2 files changed, 2 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index c7d1ec6..1172160 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -170,6 +170,7 @@ end # } get '/get_jobs_summary' do group_jobs_stats(params) +end # GET /get_job_error?suite=virttest&tbox_group=vm-2p8g # Response diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 7f26b9c..932d362 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -614,6 +614,7 @@ def group_jobs_stats(params) return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end # ------------------------------------------------------------------------------------------- # job error table like: -- 2.23.0

1 0

[PATCH v2 compass-ci] netdata: modify the service port
by Yu Chuan 27 Mar '21

27 Mar '21

[Why] According our latest port policy, modify the service port: - netdata: 20012 => 20013 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/netdata/start | 4 +++- sparrow/3-code/dev-env | 1 + 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/container/netdata/start b/container/netdata/start index 18d225b8fcae..93eabb45dba7 100755 --- a/container/netdata/start +++ b/container/netdata/start @@ -4,13 +4,15 @@ . $CCI_SRC/container/defconfig.sh +load_cci_defaults + docker_rm netdata cmd=( docker run -d --restart=always --name netdata - -p 20012:19999 + -p ${NETDATA_PORT:=20013}:19999 -v /proc:/host/proc:ro -v /sys:/host/sys:ro -v /etc/localtime:/etc/localtime:ro diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f96972eb102d..209c6663238f 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -50,6 +50,7 @@ server_name=$(hostname | cut -f1 -d.) : ${REDIS_PORT:=6379} : ${MAILDIR:=/srv/cci/Maildir/.compass-ci} : ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} +: ${NETDATA_PORT:=20013} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF -- 2.23.0

1 0

[PATCH compass-ci] netdata: modify the service port
by Yu Chuan 27 Mar '21

27 Mar '21

[Why] According our latest port policy, modify the service port: - netdata: 20012 => 20013 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/netdata/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/netdata/start b/container/netdata/start index 18d225b8fcae..4b2d8962204e 100755 --- a/container/netdata/start +++ b/container/netdata/start @@ -10,7 +10,7 @@ cmd=( docker run -d --restart=always --name netdata - -p 20012:19999 + -p 20013:19999 -v /proc:/host/proc:ro -v /sys:/host/sys:ro -v /etc/localtime:/etc/localtime:ro -- 2.23.0

2 2

[PATCH lkp-tests] sbin/submit: fix error parsing parameter
by Wu Zhende 27 Mar '21

27 Mar '21

[Input] submit a="11 22 33" job.yaml [Current key-value] {"a" => "1122 33"} [Correct key-value] {"a" => "11 22 33"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index 5a87d781a..d2d9bec76 100755 --- a/sbin/submit +++ b/sbin/submit @@ -79,7 +79,7 @@ options.parse!(ARGV) seen_job_yaml = false ARGV.delete_if do |arg| if arg.index '=' - k, v = arg.sub(' ', '').split('=', 2) + k, v = arg.split('=', 2) if seen_job_yaml opt_set_key_value[k] = v else -- 2.23.0

2 1

[PATCH compass-ci] container: fix library libgc.so.1: No such file or directory
by Cao Xueliang 27 Mar '21

27 Mar '21

use the compile env as the runtime env in monitoring and lifecycle container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/lifecycle/Dockerfile | 4 ++-- container/monitoring/Dockerfile | 6 +----- 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/container/lifecycle/Dockerfile b/container/lifecycle/Dockerfile index aa3e737..d8f8234 100644 --- a/container/lifecycle/Dockerfile +++ b/container/lifecycle/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:scheduler-dev MAINTAINER Wu Zhende <wuzhende666(a)163.com> @@ -13,7 +13,7 @@ RUN apk update && \ RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ - gem install rest-client activesupport git json yaml + gem install yaml ENV LIFECYCLE_RUNTIME_DIR /c/cci/lifecycle diff --git a/container/monitoring/Dockerfile b/container/monitoring/Dockerfile index 0a5dc22..80d2ec9 100644 --- a/container/monitoring/Dockerfile +++ b/container/monitoring/Dockerfile @@ -1,4 +1,4 @@ -FROM alpine:3.11 +FROM alpine:scheduler-dev RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories @@ -9,10 +9,6 @@ RUN apk add --no-cache 'ruby-dev' \ RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ - gem install rest-client && \ - gem install activesupport && \ - gem install git && \ - gem install json && \ gem install yaml -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code/dev-env: rename defaults vars
by Liu Yinsi 27 Mar '21

27 Mar '21

before: : {$A:=$server_ip}; $A=$A after: : {$a:=$server_ip}; $A=$a [why] when submit a job to run a testbox, sched will get variables(from /etc/compass-ci/scheduler), and export them in a running testbox env, but for selftest, it will deploy a new Compass CI, no need to use variables from the host(crystal or z9), it should be the testbox own server ip, so change defaults variables name. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f96972e..f03fcb8 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -10,23 +10,23 @@ server_name=$(hostname | cut -f1 -d.) : ${ES_HOST:=$server_ip} : ${ES_PORT:=9200} : ${OS_HTTP_HOST:=$server_ip} -: ${SEND_MAIL_HOST:=$server_ip} +: ${send_mail_host:=$server_ip} : ${SEND_MAIL_PORT:=10001} -: ${SRV_HTTP_RESULT_HOST:=$server_ip} -: ${SRV_HTTP_OS_HOST:=$server_ip} -: ${SRV_HTTP_GIT_HOST:=$server_ip} -: ${SRV_HTTP_CCI_HOST:=$server_ip} +: ${srv_http_result_host:=$server_ip} +: ${srv_http_os_host:=$server_ip} +: ${srv_http_git_host:=$server_ip} +: ${srv_http_cci_host:=$server_ip} : ${SRV_HTTP_RESULT_PORT:=20007} : ${SRV_HTTP_OS_PORT:=20009} : ${SRV_HTTP_GIT_PORT:=20010} : ${SRV_HTTP_CCI_PORT:=20011} : ${LOGGING_ES_HOST:=$server_ip} : ${LOGGING_ES_PORT:=9202} -: ${INITRD_HTTP_HOST:=$server_ip} +: ${initrd_http_host:=$server_ip} : ${INITRD_HTTP_PORT:=8800} -: ${ASSIST_RESULT_HOST:=$server_ip} +: ${assist_result_host:=$server_ip} : ${ASSIST_RESULT_PORT:=8102} -: ${RESULT_WEBDAV_HOST:=$server_ip} +: ${result_webdav_host:=$server_ip} : ${RESULT_WEBDAV_PORT:=3080} : ${MASTER_FLUENTD_HOST:=$server_ip} : ${MASTER_FLUENTD_PORT:=24224} @@ -34,7 +34,7 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} -: ${GIT_SERVER:=$server_ip} +: ${git_server:=$server_ip} : ${SCHED_HTTPS_HOST:=$server_ip} : ${SCHED_HTTPS_PORT:=20002} : ${DOCKER_PROXY_HOST:=$server_ip} @@ -76,28 +76,28 @@ ES_PORT: $ES_PORT SCHED_HOST: $sched_host SCHED_PORT: $sched_port LKP_SERVER: $server_ip -GIT_SERVER: $GIT_SERVER +GIT_SERVER: $git_server OS_HTTP_HOST: $OS_HTTP_HOST GITCACHE_HOST: $gitcache_host GITCACHE_PORT: $gitcache_port -SRV_HTTP_RESULT_HOST: $SRV_HTTP_RESULT_HOST -SRV_HTTP_OS_HOST: $SRV_HTTP_OS_HOST -SRV_HTTP_GIT_HOST: $SRV_HTTP_GIT_HOST -SRV_HTTP_CCI_HOST: $SRV_HTTP_CCI_HOST +SRV_HTTP_RESULT_HOST: $srv_http_result_host +SRV_HTTP_OS_HOST: $srv_http_os_host +SRV_HTTP_GIT_HOST: $srv_http_git_host +SRV_HTTP_CCI_HOST: $srv_http_cci_host SRV_HTTP_RESULT_PORT: $SRV_HTTP_RESULT_PORT SRV_HTTP_OS_PORT: $SRV_HTTP_OS_PORT SRV_HTTP_GIT_PORT: $SRV_HTTP_GIT_PORT SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT TASKQUEUE_HOST: $sched_host -SEND_MAIL_HOST: $SEND_MAIL_HOST +SEND_MAIL_HOST: $send_mail_host SEND_MAIL_PORT: $SEND_MAIL_PORT LOGGING_ES_HOST: $LOGGING_ES_HOST LOGGING_ES_PORT: $LOGGING_ES_PORT -INITRD_HTTP_HOST: $INITRD_HTTP_HOST +INITRD_HTTP_HOST: $initrd_http_host INITRD_HTTP_PORT: $INITRD_HTTP_PORT -ASSIST_RESULT_HOST: $ASSIST_RESULT_HOST +ASSIST_RESULT_HOST: $assist_result_host ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT -RESULT_WEBDAV_HOST: $RESULT_WEBDAV_HOST +RESULT_WEBDAV_HOST: $result_webdav_host RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST MASTER_FLUENTD_PORT: $MASTER_FLUENTD_PORT -- 2.23.0

3 3

[PATCH compass-ci] src/lib: add default prefix for move queue
by Cao Xueliang 26 Mar '21

26 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/etcd_client.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/etcd_client.cr b/src/lib/etcd_client.cr index 637b738..7891eb3 100644 --- a/src/lib/etcd_client.cr +++ b/src/lib/etcd_client.cr @@ -39,6 +39,8 @@ class EtcdClient end def move(f_queue, t_queue, value) + f_queue = "#{BASE}/#{f_queue}" unless f_queue.starts_with?(BASE) + t_queue = "#{BASE}/#{t_queue}" unless t_queue.starts_with?(BASE) @etcd.kv.move(f_queue, t_queue, value) end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/0-package/openEuler: update wget URL
by Liu Yinsi 26 Mar '21

26 Mar '21

The ole URL doesn't work, so update valid RUL Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/openEuler | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 61a37c4..0b0348d 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -28,8 +28,8 @@ pkgs=( yum install -y --skip-broken "${pkgs[@]}" -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… -wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… +wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… +wget http://rpmfind.net/linux/openmandriva/cooker/repository/$(arch)/main/releas… rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm rm -rf *.rpm -- 2.23.0

1 1

[PATCH compass-ci] src/lib: add etcd client
by Cao Xueliang 26 Mar '21

26 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/constants.cr | 8 +++++++ src/lib/etcd_client.cr | 51 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) create mode 100644 src/lib/constants.cr create mode 100644 src/lib/etcd_client.cr diff --git a/src/lib/constants.cr b/src/lib/constants.cr new file mode 100644 index 0000000..d6a96d7 --- /dev/null +++ b/src/lib/constants.cr @@ -0,0 +1,8 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ or GPL-2.0 +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +BASE = "/queues" + +ETCD_HOST = "172.17.0.1" +ETCD_PORT = 2379 +ETCD_VERSION = "v3beta" diff --git a/src/lib/etcd_client.cr b/src/lib/etcd_client.cr new file mode 100644 index 0000000..637b738 --- /dev/null +++ b/src/lib/etcd_client.cr @@ -0,0 +1,51 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ or GPL-2.0 +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "etcd" + +require "./constants" + +class EtcdClient + def initialize + host = (ENV.has_key?("ETCD_HOST") ? ENV["ETCD_HOST"] : ETCD_HOST) + port = (ENV.has_key?("ETCD_PORT") ? ENV["ETCD_PORT"].to_i32 : ETCD_PORT) + version = (ENV.has_key?("ETCD_VERSION") ? ENV["ETCD_VERSION"] : ETCD_VERSION) + @etcd = Etcd.client(host, port, version) + end + + def put(queue, content) + queue = "#{BASE}/#{queue}" unless queue.starts_with?(BASE) + @etcd.kv.put_not_exists(queue, content) + end + + def delete(queue) + queue = "#{BASE}/#{queue}" unless queue.starts_with?(BASE) + @etcd.kv.delete(queue) + end + + def range(queue) + queue = "#{BASE}/#{queue}" unless queue.starts_with?(BASE) + @etcd.kv.range(queue) + end + + def range_prefix(prefix) + prefix = "#{BASE}/#{prefix}" unless prefix.starts_with?(BASE) + @etcd.kv.range_prefix(prefix) + end + + def update(queue, value) + queue = "#{BASE}/#{queue}" unless queue.starts_with?(BASE) + @etcd.kv.put(queue, value) + end + + def move(f_queue, t_queue, value) + @etcd.kv.move(f_queue, t_queue, value) + end + + def watch_prefix(prefix, **opts, &block : Array(Etcd::Model::WatchEvent) -> Void) + prefix = "#{BASE}/#{prefix}" unless prefix.starts_with?(BASE) + @etcd.watch.watch_prefix(prefix, **opts, &block) + end + +end + -- 2.23.0

1 0

[PATCH compass-ci] lib/dump_stat.rb: fix name error
by Lu Weitao 26 Mar '21

26 Mar '21

[Why] /home/baijing/compass-ci/lib/dump_stat.rb:171:in `number?': undefined local variable or method `monitor' for DumpStat:Module (NameError) Did you mean? @monitor Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/dump_stat.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/dump_stat.rb b/lib/dump_stat.rb index 1c0695e..de0168e 100644 --- a/lib/dump_stat.rb +++ b/lib/dump_stat.rb @@ -55,7 +55,7 @@ module DumpStat if value.is_a?(String) value = check_string_value(k, value, @monitor) next unless value - return nil unless number?(value, @invalid_records) + return nil unless number?(value, @invalid_records, @record_index, @monitor) value = value.index('.') ? value.to_f : value.to_i elsif value.is_a?(Array) @@ -64,7 +64,7 @@ module DumpStat value[i] = check_string_value(k, value[i], @monitor) next unless value[i] - return nil unless number?(value[i], @invalid_records) + return nil unless number?(value[i], @invalid_records, @record_index, @monitor) value[i] = value[i].index('.') ? value[i].to_f : value[i].to_i valid_stats_verification(k, value[i]) @@ -165,7 +165,7 @@ def check_string_value(key, value, monitor) end # only number is valid -def number?(value, invalid_records) +def number?(value, invalid_records, record_index, monitor) unless value.numeric? invalid_records.push record_index warn_stat "invalid stats value: #{value}", monitor -- 2.23.0

1 0

[PATCH compass-ci 3/3] container/scheduler: fix library libgc.so.1: No such file or directory
by Cao Xueliang 26 Mar '21

26 Mar '21

use the compile env as the runtime env Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/scheduler/Dockerfile | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/container/scheduler/Dockerfile b/container/scheduler/Dockerfile index 7fb8e35..229a9c7 100644 --- a/container/scheduler/Dockerfile +++ b/container/scheduler/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:scheduler-dev RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories @@ -12,10 +12,6 @@ RUN apk add --no-cache 'ruby-dev' \ RUN umask 002 && \ gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ - gem install rest-client && \ - gem install activesupport && \ - gem install git && \ - gem install json && \ gem install yaml -- 2.23.0

1 1

[PATCH compass-ci 2/3] container/crystal-shards: update crystal version from 0.33 to 0.35
by Cao Xueliang 26 Mar '21

26 Mar '21

add shard-amqp.yml to shard.yml, delete shard-amqp.yml add crystal-etcd to shard.yml Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/crystal-shards/Dockerfile | 3 --- container/crystal-shards/shard-amqp.yml | 7 ------- container/crystal-shards/shard.yml | 9 ++++++--- 3 files changed, 6 insertions(+), 13 deletions(-) delete mode 100644 container/crystal-shards/shard-amqp.yml diff --git a/container/crystal-shards/Dockerfile b/container/crystal-shards/Dockerfile index 15fe523..d3e321f 100644 --- a/container/crystal-shards/Dockerfile +++ b/container/crystal-shards/Dockerfile @@ -17,8 +17,5 @@ sed -i '99s/arguments\[:id]/arguments\[:id]?/' /usr/share/crystal/app/lib/elast sed -i 's/, Utils.__listify(arguments\[:q].as(String))/ /' /usr/share/crystal/app/lib/elasticsearch-crystal/src/elasticsearch/api/actions/search.cr;\ sed -i '205a \ params.clear' /usr/share/crystal/app/lib/elasticsearch-crystal/src/elasticsearch/api/actions/search.cr; -COPY shard-amqp.yml /usr/share/crystal/app/shard.yml -RUN shards - CMD ["bash"] diff --git a/container/crystal-shards/shard-amqp.yml b/container/crystal-shards/shard-amqp.yml deleted file mode 100644 index aea87cc..0000000 --- a/container/crystal-shards/shard-amqp.yml +++ /dev/null @@ -1,7 +0,0 @@ -name: monitor -version: 0.1.0 - -dependencies: - amqp-client: - github: cloudamqp/amqp-client.cr - version: 0.5.14 diff --git a/container/crystal-shards/shard.yml b/container/crystal-shards/shard.yml index ca8f72e..3466bc5 100644 --- a/container/crystal-shards/shard.yml +++ b/container/crystal-shards/shard.yml @@ -8,7 +8,7 @@ targets: scheduler: main: src/scheduler.cr -crystal: 0.33.0 +crystal: 0.35.0 license: Mulan PSL v2 @@ -17,10 +17,8 @@ dependencies: github: kemalcr/kemal redis: github: stefanwille/crystal-redis - version: ~> 2.5.3 elasticsearch-crystal: github: paktek123/elasticsearch-crystal - version: ~> 0.14 any_merge: github: icyleaf/any_merge branch: master @@ -32,3 +30,8 @@ dependencies: github: z64/rate_limiter singleton: github: waterlink/singleton.cr + etcd: + github: PythonEngineer007/crystal-etcd + branch: master + amqp-client: + github: cloudamqp/amqp-client.cr -- 2.23.0

1 0

[PATCH compass-ci 1/3] container/crystal-base: update alpine version from 3.11 to 3.13
by Cao Xueliang 26 Mar '21

26 Mar '21

need install llvm10 in alpine:3.13. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/crystal-base/Dockerfile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/crystal-base/Dockerfile b/container/crystal-base/Dockerfile index 03fe5e6..633a2ec 100644 --- a/container/crystal-base/Dockerfile +++ b/container/crystal-base/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:3.13 MAINTAINER chief <taxcom(a)tom.com> @@ -10,7 +10,7 @@ ONBUILD RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.hu ONBUILD RUN apk update ONBUILD RUN apk add bash zsh git rsync curl crystal shards tzdata sudo \ yaml-dev openssl-dev ncurses-static g++ make cmake \ - zlib-dev zlib-static openssl-libs-static llvm5 llvm5-dev \ - llvm5-libs llvm5-static musl-dev xz-dev libxml2-dev \ + zlib-dev zlib-static openssl-libs-static llvm10 llvm10-dev \ + llvm10-libs llvm10-static musl-dev xz-dev libxml2-dev \ vim ruby-dev -- 2.23.0

1 0

[PATCH compass-ci] container/result-webdav: add /srv/rpm share dir
by Cui Yili 26 Mar '21

26 Mar '21

Signed-off-by: Cui Yili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index 2852f24..993f24c 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -21,6 +21,7 @@ cmd=( -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ + -v /srv/rpm/:/srv/rpm/ -v /srv/cci/build-config/:/srv/cci/build-config/ result-webdav ) -- 2.23.0

1 0

[PATCH lab-z9] add testboxes host-file
by Zhang Dewan 26 Mar '21

26 Mar '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- hosts/taishan200-2280-2s48p-256g--a63 | 6 +++--- hosts/taishan200-2280-2s48p-256g--a64 | 6 +++--- hosts/taishan200-2280-2s48p-256g--a65 | 6 +++--- hosts/taishan200-2280-2s64p-256g--a110 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a112 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a113 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a114 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a116 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a118 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a119 | 19 ++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a124 | 22 +++++++++++++++++++++ hosts/taishan200-2280-2s64p-256g--a39 | 27 ++++++++++++++++++++++++++ 12 files changed, 191 insertions(+), 9 deletions(-) create mode 100644 hosts/taishan200-2280-2s64p-256g--a110 create mode 100644 hosts/taishan200-2280-2s64p-256g--a112 create mode 100644 hosts/taishan200-2280-2s64p-256g--a113 create mode 100644 hosts/taishan200-2280-2s64p-256g--a114 create mode 100644 hosts/taishan200-2280-2s64p-256g--a116 create mode 100644 hosts/taishan200-2280-2s64p-256g--a118 create mode 100644 hosts/taishan200-2280-2s64p-256g--a119 create mode 100644 hosts/taishan200-2280-2s64p-256g--a124 create mode 100644 hosts/taishan200-2280-2s64p-256g--a39 diff --git a/hosts/taishan200-2280-2s48p-256g--a63 b/hosts/taishan200-2280-2s48p-256g--a63 index a8e6688..0b3c938 100644 --- a/hosts/taishan200-2280-2s48p-256g--a63 +++ b/hosts/taishan200-2280-2s48p-256g--a63 @@ -1,6 +1,3 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 memory: 256G nr_hdd_partitions: 4 hdd_partitions: @@ -17,5 +14,8 @@ mac_addr: - 44:67:47:c9:e5:49 - 44:67:47:c9:e5:4a - 44:67:47:c9:e5:4b +arch: aarch64 +nr_node: 4 +nr_cpu: 96 model_name: Kunpeng-920 ipmi_ip: 9.3.4.4 diff --git a/hosts/taishan200-2280-2s48p-256g--a64 b/hosts/taishan200-2280-2s48p-256g--a64 index f8aa111..7cd0c48 100644 --- a/hosts/taishan200-2280-2s48p-256g--a64 +++ b/hosts/taishan200-2280-2s48p-256g--a64 @@ -1,6 +1,3 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 memory: 256G nr_hdd_partitions: 4 hdd_partitions: @@ -17,5 +14,8 @@ mac_addr: - 44:67:47:c9:ea:b7 - 44:67:47:c9:ea:b8 - 44:67:47:c9:ea:b9 +arch: aarch64 +nr_node: 4 +nr_cpu: 96 model_name: Kunpeng-920 ipmi_ip: 9.3.4.5 diff --git a/hosts/taishan200-2280-2s48p-256g--a65 b/hosts/taishan200-2280-2s48p-256g--a65 index 341303c..88441cc 100644 --- a/hosts/taishan200-2280-2s48p-256g--a65 +++ b/hosts/taishan200-2280-2s48p-256g--a65 @@ -1,6 +1,3 @@ -arch: aarch64 -nr_node: 4 -nr_cpu: 96 memory: 256G nr_hdd_partitions: 4 hdd_partitions: @@ -17,5 +14,8 @@ mac_addr: - 44:67:47:c9:e9:95 - 44:67:47:c9:e9:96 - 44:67:47:c9:e9:97 +arch: aarch64 +nr_node: 4 +nr_cpu: 96 model_name: Kunpeng-920 ipmi_ip: 9.3.4.6 diff --git a/hosts/taishan200-2280-2s64p-256g--a110 b/hosts/taishan200-2280-2s64p-256g--a110 new file mode 100644 index 0000000..50625bc --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a110 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b6079 + - /dev/disk/by-id/scsi-350000399886b69bd +mac_addr: + - 44:67:47:e9:79:3d + - 44:67:47:e9:79:3e + - 44:67:47:e9:79:3f + - 44:67:47:e9:79:40 + - 44:a1:91:9f:9b:8a + - 44:a1:91:9f:9b:8b + - 44:a1:91:9f:9b:8c + - 44:a1:91:9f:9b:8d +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.3 diff --git a/hosts/taishan200-2280-2s64p-256g--a112 b/hosts/taishan200-2280-2s64p-256g--a112 new file mode 100644 index 0000000..30c0f80 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a112 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399888173c1 + - /dev/disk/by-id/scsi-350000399888169c9 +mac_addr: + - 44:a1:91:9d:f3:3a + - 44:a1:91:9d:f3:3b + - 44:a1:91:9d:f3:3c + - 44:a1:91:9d:f3:3d + - 44:a1:91:e5:85:11 + - 44:a1:91:e5:85:12 + - 44:a1:91:e5:85:13 + - 44:a1:91:e5:85:14 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.5 diff --git a/hosts/taishan200-2280-2s64p-256g--a113 b/hosts/taishan200-2280-2s64p-256g--a113 new file mode 100644 index 0000000..826b18d --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a113 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039988700409 + - /dev/disk/by-id/scsi-350000399886bfb25 +mac_addr: + - 44:67:47:85:ed:b7 + - 44:67:47:85:ed:b8 + - 44:67:47:85:ed:b9 + - 44:67:47:85:ed:ba + - cc:05:77:ee:b3:1f + - cc:05:77:ee:b3:20 + - cc:05:77:ee:b3:21 + - cc:05:77:ee:b3:22 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.6 diff --git a/hosts/taishan200-2280-2s64p-256g--a114 b/hosts/taishan200-2280-2s64p-256g--a114 new file mode 100644 index 0000000..f3a4d68 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a114 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399785bf105 + - /dev/disk/by-id/scsi-350000399785bf189 +mac_addr: + - 44:a1:91:9d:d5:44 + - 44:a1:91:9d:d5:45 + - 44:a1:91:9d:d5:46 + - 44:a1:91:9d:d5:47 + - 44:a1:91:e5:6f:01 + - 44:a1:91:e5:6f:02 + - 44:a1:91:e5:6f:03 + - 44:a1:91:e5:6f:04 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.7 diff --git a/hosts/taishan200-2280-2s64p-256g--a116 b/hosts/taishan200-2280-2s64p-256g--a116 new file mode 100644 index 0000000..5ca6cd4 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a116 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b6fad + - /dev/disk/by-id/scsi-350000399886b6ba9 +mac_addr: + - 44:67:47:85:f0:af + - 44:67:47:85:f0:b0 + - 44:67:47:85:f0:b1 + - 44:67:47:85:f0:b2 + - cc:05:77:ee:b0:ef + - cc:05:77:ee:b0:f0 + - cc:05:77:ee:b0:f1 + - cc:05:77:ee:b0:f2 +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.9 diff --git a/hosts/taishan200-2280-2s64p-256g--a118 b/hosts/taishan200-2280-2s64p-256g--a118 new file mode 100644 index 0000000..9a93e03 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a118 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-35000039988707e91 + - /dev/disk/by-id/scsi-350000399887080e5 +mac_addr: + - 44:67:47:e9:78:cf + - 44:67:47:e9:78:d0 + - 44:67:47:e9:78:d1 + - 44:67:47:e9:78:d2 + - 44:a1:91:9f:dd:aa + - 44:a1:91:9f:dd:ab + - 44:a1:91:9f:dd:ac + - 44:a1:91:9f:dd:ad +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.11 diff --git a/hosts/taishan200-2280-2s64p-256g--a119 b/hosts/taishan200-2280-2s64p-256g--a119 new file mode 100644 index 0000000..57da863 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a119 @@ -0,0 +1,19 @@ +memory: 128G +nr_hdd_partitions: 2 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399886b6be9 + - /dev/disk/by-id/scsi-350000399886b5dbd +mac_addr: + - 44:67:47:e9:79:65 + - 44:67:47:e9:79:66 + - 44:67:47:e9:79:67 + - 44:67:47:e9:79:68 + - 44:a1:91:9f:9b:4a + - 44:a1:91:9f:9b:4b + - 44:a1:91:9f:9b:4c + - 44:a1:91:9f:9b:4d +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.11.12 diff --git a/hosts/taishan200-2280-2s64p-256g--a124 b/hosts/taishan200-2280-2s64p-256g--a124 new file mode 100644 index 0000000..7e7bb42 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a124 @@ -0,0 +1,22 @@ +memory: 256G +nr_ssd_partitions: 1 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB44344 +mac_addr: + - 44:67:47:e9:76:9f + - 44:67:47:e9:76:a0 + - 44:67:47:e9:76:a1 + - 44:67:47:e9:76:a2 + - ac:b3:b5:1e:d5:3b + - ac:b3:b5:1e:d5:3c + - ac:b3:b5:1e:d5:3d + - ac:b3:b5:1e:d5:3e + - ac:b3:b5:1e:d6:6b + - ac:b3:b5:1e:d6:6c + - ac:b3:b5:1e:d6:6d + - ac:b3:b5:1e:d6:6e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.12.5 diff --git a/hosts/taishan200-2280-2s64p-256g--a39 b/hosts/taishan200-2280-2s64p-256g--a39 new file mode 100644 index 0000000..97d99d9 --- /dev/null +++ b/hosts/taishan200-2280-2s64p-256g--a39 @@ -0,0 +1,27 @@ +memory: 256G +nr_hdd_partitions: 1 +nr_ssd_partitions: 1 +hdd_partitions: + - /dev/disk/by-id/scsi-350000399981347d5 +ssd_partitions: + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part1 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part2 + - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43895-part3 +mac_addr: + - 84:46:fe:89:b9:98 + - 84:46:fe:89:b9:99 + - 84:46:fe:89:b9:9a + - 84:46:fe:89:b9:9b + - ac:b3:b5:1e:db:2b + - ac:b3:b5:1e:db:2c + - ac:b3:b5:1e:db:2d + - ac:b3:b5:1e:db:2e + - ac:b3:b5:1e:d8:3b + - ac:b3:b5:1e:d8:3c + - ac:b3:b5:1e:d8:3d + - ac:b3:b5:1e:d8:3e +arch: aarch64 +nr_node: 4 +nr_cpu: 128 +model_name: Kunpeng-920 +ipmi_ip: 9.3.2.4 -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: change upload dir
by Wang Yong 26 Mar '21

26 Mar '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index c1ca223..5767f61 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -293,7 +293,7 @@ class Job elsif @hash["cci-depends"]? package_dir = ",/initrd/deps/#{common_dir}/#{@hash["cci-depends"]["benchmark"]}" elsif @hash["rpmbuild-pkg"]? - package_dir = ",/initrd/rpmbuild-pkg/#{common_dir}" + package_dir = ",/rpm/upload/#{common_dir}" elsif @hash["build-pkg"]? if @hash["pkgbuild_repo"].to_s =~ /(packages|community)\/\// package_name = @hash["pkgbuild_repo"].to_s.split("/")[-2] -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: change upload dir
by Wang Yong 26 Mar '21

26 Mar '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 90905b220..7a101b6a0 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -11,7 +11,7 @@ [ -n "$upstream_repo" ] || die "upstream_repo is empty" -dest_dir="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}" +dest_dir="/rpm/upload/${os}-${os_version}/${compat_os}" from_git() { -- 2.23.0

1 0

[PATCH compass-ci] doc: fix some spell errors
by Li Yuanchao 26 Mar '21

26 Mar '21

blacklight => backlight Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.en.md | 2 +- doc/manual/test-oss-project.zh.md | 4 ++-- .../{fork_blacklight.png => fork_backlight.png} | Bin 3 files changed, 3 insertions(+), 3 deletions(-) rename doc/pictures/{fork_blacklight.png => fork_backlight.png} (100%) diff --git a/doc/manual/test-oss-project.en.md b/doc/manual/test-oss-project.en.md index a3c0781..6b6d9ca 100644 --- a/doc/manual/test-oss-project.en.md +++ b/doc/manual/test-oss-project.en.md @@ -8,7 +8,7 @@ Perform the following steps to add the information of the code repository to be 1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. -![](./../pictures/fork_blacklight.png) +![](./../pictures/fork_backlight.png) 2. Run the following command to create a file path named with the repository name and its initial letter: diff --git a/doc/manual/test-oss-project.zh.md b/doc/manual/test-oss-project.zh.md index b455c32..413066d 100644 --- a/doc/manual/test-oss-project.zh.md +++ b/doc/manual/test-oss-project.zh.md @@ -6,9 +6,9 @@ 执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 -1. Fork upstream-repos 仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 +1. Fork upstream-repos 仓库并 git clone 到本地，本文以 backlight 仓库（https://github.com/baskerville/backlight ）为例说明。 -![](./../pictures/fork_blacklight.png) +![](./../pictures/fork_backlight.png) 2. 执行以下命令，以首字母和仓库名创建文件路径。 diff --git a/doc/pictures/fork_blacklight.png b/doc/pictures/fork_backlight.png similarity index 100% rename from doc/pictures/fork_blacklight.png rename to doc/pictures/fork_backlight.png -- 2.23.0

1 0

[PATCH lkp-tests] lib/email.sh: rename $MAIL_HOST
by Liu Yinsi 25 Mar '21

25 Mar '21

MAIL_HOST => SEND_MAIL_HOST [why] because MAIL_HOST is duplicate variable name, $MAIL_HOST is $SEND_MAIL_HOST. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/email.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/email.sh b/lib/email.sh index bca4ab435..4f1055033 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -87,5 +87,5 @@ send_email() ${subject}_content - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-10001}/send_mail_text" -d "$email_content" + curl -XPOST "${SEND_MAIL_HOST:-$LKP_SERVER}:${SEND_MAIL_PORT:-10001}/send_mail_text" -d "$email_content" } -- 2.23.0

1 0

[PATCH v3 compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..d61f11e 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,6 +34,21 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=20002} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MONITOR_HOST:=$server_ip} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -53,6 +68,8 @@ EOF mkdir -p /etc/compass-ci/service cat > /etc/compass-ci/service/service-env.yaml <<EOF +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT ES_HOST: $ES_HOST ES_PORT: $ES_PORT SCHED_HOST: $sched_host @@ -88,6 +105,18 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MONITOR_HOST: $MONITOR_HOST +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

2 2

[PATCH v4 compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..f96972e 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,6 +34,22 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=20002} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MONITOR_HOST:=$server_ip} +: ${MONITOR_PORT:=20001} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -53,6 +69,8 @@ EOF mkdir -p /etc/compass-ci/service cat > /etc/compass-ci/service/service-env.yaml <<EOF +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT ES_HOST: $ES_HOST ES_PORT: $ES_PORT SCHED_HOST: $sched_host @@ -88,6 +106,19 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MONITOR_HOST: $MONITOR_HOST +MONITOR_PORT: $MONITOR_PORT +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor.rb: update SRV_HTTP_RESULT_PORT
by Liu Yinsi 25 Mar '21

25 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/email.sh | 2 +- lib/monitor.rb | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/email.sh b/lib/email.sh index 63d5a51e2..bca4ab435 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -8,7 +8,7 @@ Subject: [NOTIFY Compass-CI] Test job failed: $id Dear $my_name: Sorry to inform you that your test job is failed, you can click the following link to view details. - http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT:-11300}$result_root + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… Regards Compass-CI diff --git a/lib/monitor.rb b/lib/monitor.rb index f14e131e3..bf7ef69ba 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -100,9 +100,9 @@ class Monitor @result_roots.each do |res| res.to_s.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' - srv_http_port = job['SRV_HTTP_PORT'] || '11300' - url = "http://#{srv_http_host}:#{srv_http_port}#{res}" + srv_http_result_host = job['SRV_HTTP_RESULT_HOST'] || 'api.compass-ci.openeuler.org' + srv_http_result_port = job['SRV_HTTP_RESULT_PORT'] || '20007' + url = "http://#{srv_http_result_host}:#{srv_http_result_port}#{res}" system "lftp -c mirror #{url} >/dev/null 2>&1" end end -- 2.23.0

1 0

help star
by Wu Fengguang 25 Mar '21

25 Mar '21

Yu Chuan, Wu Zhende is our help star again, thanks Yu Chuan, Wu Zhende! wfg@crystal ~/todo/bin% ./help-star ../people/*/credit/2021-03-25.md 15.2 Yu Chuan 15.1 Wu Zhende 11.2 Cao Xueliang 11.0 Wei Jihui 8.8 Liu Yinsi 7.4 Luan Shengde 7.0 Wang Yong 6.4 Lu Weitao 5.6 Wu Fengguang 5.4 Wang Chenglong 5.1 Zhang Dewan 5.0 Li Yuanchao 4.2 Liu Shaofei 4.1 Lin Jiaxin 2.9 Li Ping 1.8 Du Kaitian 1.6 Xu Xijian 1.2 Bai Jing 0.9 Cui Yili 0.0 Zhang Yale

1 0

git count
by Wu Fengguang 25 Mar '21

25 Mar '21

Wei Jihui is our code star, thanks Wei Jihui! wfg@crystal ~/todo/bin% ./git-count 6 Wei Jihui 5 Li Yuanchao 4 Wu Zhende 4 Yu Chuan 4 Luan Shengde 4 Lin Jiaxin 3 Cao Xueliang 3 Liu Shaofei 3 Liu Yinsi 3 Li Ping 2 Bai Jing 2 Xu Xijian 2 Lu Weitao 2 Cui Yili 2 Wang Chenglong 2 Wang Yong 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale 0 Zhang Dewan

1 0

review star
by Wu Fengguang 25 Mar '21

25 Mar '21

Luan Shengde is our review star, thanks Luan Shengde! wfg@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 9 4 19 9 Wu Fengguang 19 9 18 7 Luan Shengde 49 6 6 6 Wei Jihui 21 5 5 6 Liu Shaofei 39 2 2 5 Cao Xueliang 14 4 4 3 Lin Jiaxin 22 3 5 2 Li Yuanchao 3 0 0 2 Xu Xijian 8 3 5 2 Bai Jing 20 2 2 1 Wang Chenglong 17 3 5 1 Wang Yong 9 1 0 1 Lu Weitao 7 2 3 1 Cui Yili 3 0 0 0 Zhang Dewan 4 1 1 0 Yu Chuan 14 0 0 0 Wu Zhende 11 1 1 0 Li Ping 9 0 0 0 Liu Yinsi

1 0

[PATCH v5 compass-ci] container/web-backend: add API /get_table_error
by Lu Weitao 25 Mar '21

25 Mar '21

[Example] curl -X GET 'localhost:10002/get_job_error?group_id=wcl_ansible-openeuler-03-10' return: { "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, "attributes":["job_id","error_id","error_message","link_to_result"], "objects":[ { "job_id": "crystal.1354921", "error_id": "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-xxx.fail", "error_message": "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", "link_to_result": "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…", ate/crystal.1354921" }, { "job_id": "crystal.1354929", ... }, ... ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 18 +++++++++ src/lib/web_backend.rb | 65 +++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index adb91e0..c7d1ec6 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -170,4 +170,22 @@ end # } get '/get_jobs_summary' do group_jobs_stats(params) + +# GET /get_job_error?suite=virttest&tbox_group=vm-2p8g +# Response +# { +# "filter": {"group_id":"wcl_ansible-openeuler-03-10"}, +# "attributes": ["job_id", "error_id", "error_message", "link_to_result"], +# "objects": [ +# { +# "job_id": "crystal.1354921", +# "error_id": "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-failed.fail", +# "error_message": "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", +# "link_to_result": "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…" +# }, +# ... +# ] +# } +get '/get_job_error' do + get_job_error(params) end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 40471e0..7f26b9c 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/params_group.rb" require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -613,4 +614,68 @@ def group_jobs_stats(params) return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] + +# ------------------------------------------------------------------------------------------- +# job error table like: +# job_id error_id error_message link to result +# ------------------------------------------------------------------------------------- +# crystal.630608 "stderr.xxx" "messag:xxxx" https://$host:$port/$result_root +# ... +# ------------------------------------------------------------------------------------------- + +def get_job_error(params) + begin + body = job_error_body(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get error table error'] + end + + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def job_error_body(params) + error_objects = get_error_objects(params) + { + filter: params, + attributes: ['job_id', 'error_id', 'error_message', 'link_to_result'], + objects: error_objects, + }.to_json +end + +def get_error_objects(filter_items) + error_objs = [] + + job_list = get_job_list(filter_items) + job_list.each do |job| + error_obj = get_error_from_job(job) + error_objs << error_obj unless error_obj.empty? + end + + error_objs +end + +def get_job_list(items) + es = ESQuery.new(ES_HOST, ES_PORT) + query_results = es.multi_field_query(items) + + extract_jobs_list(query_results['hits']['hits']) +end + +# get all error_id from one job +def get_error_from_job(job) + job_error_obj = {} + job['stats'].each do |metric, value| + next unless metric.end_with?('.message') + + result_host = ENV['SRV_HTTP_RESULT_HOST'] || SRV_HTTP_RESULT_HOST + result_port = ENV['SRV_HTTP_RESULT_PORT'] || SRV_HTTP_RESULT_PORT + error_id = metric.sub('.message', '.fail') + job_error_obj['id'] = job['id'] + job_error_obj['error_id'] = error_id + job_error_obj['error_message'] = value + job_error_obj['link_to_result'] = value + end + + job_error_obj end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..b2c1814 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,12 +34,33 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=11306} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${LKP_CGI_PORT:=3000} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MAIL_HOST:=$server_ip} +: ${MAIL_PORT:=10001} +: ${MONITOR_HOST:=$server_ip} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} + mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port lab: $lab +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT EOF [[ -f /etc/compass-ci/defaults/upstream-config ]] || { @@ -88,6 +109,21 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +LKP_CGI_PORT: $LKP_CGI_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MAIL_HOST: $MAIL_HOST +MAIL_PORT: $MAIL_PORT +MONITOR_HOST: $MONITOR_HOST +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..005e29e 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,12 +34,32 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=11306} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${LKP_CGI_PORT:=3000} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MAIL_HOST:=$server_ip} +: ${MAIL_PORT:=10001} +: ${MONITOR_HOST:=$server_ip} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port lab: $lab +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT EOF [[ -f /etc/compass-ci/defaults/upstream-config ]] || { @@ -88,6 +108,21 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +LKP_CGI_PORT: $LKP_CGI_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MAIL_HOST: $MAIL_HOST +MAIL_PORT: $MAIL_PORT +MONITOR_HOST: $MONITOR_HOST +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/build-pkg: update filtering rule
by Lin Jiaxin 25 Mar '21

25 Mar '21

error_ids that are filtered out: /tmp/lkp/clementine-git/src/Clementine/3rdparty/qsqlite/qsql_sqlite.cpp: In function ‘QSqlError qMakeError(sqlite3*, const QString&, QSqlError::ErrorType, int)’: make: *** [makefile.linux:68: makefile.dependencies] Error 1 go: finding github.com/pingcap/errors v0.11.5-0.20201126102027-b0a155152ca3 Cyclomatic Complexity 8 ./include/trace/events/error_report.h:trace_error_report_end etc. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- stats/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/build-pkg b/stats/build-pkg index 90f27656..764594b7 100755 --- a/stats/build-pkg +++ b/stats/build-pkg @@ -20,7 +20,7 @@ while (line = STDIN.gets) seqno = '' end next unless in_stderr - next unless line.downcase =~ /error|warning/ + next unless line.downcase =~ /(error|warning):[^:]/ error_ids << build_pkg_error_id(line) end -- 2.23.0

3 2

[PATCH lkp-tests] stats/ansible_test: add module name in error id
by Wang Chenglong 25 Mar '21

25 Mar '21

before: error.Failed-to-install-some-of-the-specified-packages: 1 after: error.[msg].Failed-to-install-some-of-the-specified-packages: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index d6acad1ab..3d3b0b0cb 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -2,18 +2,18 @@ require 'json' -def output_error(error_msg) +def output_error(error_msg, error_key) return if error_msg.nil? if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.#{error_id}: 1" + puts "error.[#{error_key}].#{error_id}: 1" puts "error.#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.#{error_id}: 1" + puts "error.[#{error_key}].#{error_id}: 1" puts "error.#{error_id}.message: #{@ansible_failed_info}" end end @@ -33,8 +33,8 @@ while (line = STDIN.gets) ansible_failed_json = JSON.parse @ansible_failed_info - output_error ansible_failed_json['msg'] - output_error ansible_failed_json['message'] - output_error ansible_failed_json['cmd'] - output_error ansible_failed_json['failures'] + output_error ansible_failed_json['msg'],'msg' + output_error ansible_failed_json['message'],'message' + output_error ansible_failed_json['cmd'],'cmd' + output_error ansible_failed_json['failures'],'failures' end -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/rpmbuild-pkg: add build way from srpm
by Wang Yong 25 Mar '21

25 Mar '21

original build RPMs from git repo, now support building from SRPM Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 67 +++++++++++++++++++++++++++++++++++----------- 1 file changed, 51 insertions(+), 16 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 256e836bf..90905b220 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,6 +1,8 @@ #!/bin/bash # - upstream_repo # - compat_os +# - repo_name +# - repo_addr . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh @@ -8,9 +10,30 @@ : "${compat_os:=budding-openeuler}" [ -n "$upstream_repo" ] || die "upstream_repo is empty" -package_name=${upstream_repo##*/} -rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/${os_arch}/Packages" -src_rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/source/Packages" + +dest_dir="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}" + +from_git() +{ + package_name=${upstream_repo##*/} + rpm_dest="${dest_dir}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/source/Packages" + + init_workspace + download_upstream_repo +} + +from_srpm() +{ + [ -n "$repo_name" ] || die "repo_name is empty" + [ -n "$repo_addr" ] || die "repo_addr is empty" + + rpm_dest="${dest_dir}/${repo_name}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/${repo_name}/source/Packages" + + install_srpm +} + init_workspace() { @@ -20,30 +43,43 @@ init_workspace() download_upstream_repo() { - git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" - cd $package_name || exit + local url="git://${GIT_SERVER}/openeuler/${upstream_repo}" + + git clone "$url" || die "clone git repo ${package_name} failed: $url" + cd "$package_name" || exit filelist=$(git ls-files) + for pkgfile in ${filelist[@]} do local dir="SOURCES" - echo $pkgfile | egrep "\.spec$" && dir="SPECS" + + echo "$pkgfile" | grep -E "\\.spec$" && dir="SPECS" mv "$pkgfile" "${HOME}/rpmbuild/${dir}/" done } -build_rpm() +install_srpm() { - local spec_file=${HOME}/rpmbuild/SPECS/$package_name.spec - - # HTTP is proxy cache friendly - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_file` + rpm -i "${repo_addr}/${upstream_repo}" >/dev/null || die "failed to install source rpm: ${repo_addr}/${upstream_repo}" +} +build_rpm() +{ + local spec_dir="${HOME}/rpmbuild/SPECS" + [ -n "$package_name" ] && + { + # HTTP is proxy cache friendly + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/${package_name}.spec")" + } # Install build depends - yum-builddep -y $spec_file || exit + yum-builddep -y "$spec_dir"/*.spec || die "failed to solve dependencies" # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE - spectool -g -R $spec_file || exit + [ -n "$package_name" ] && + { + spectool -g -R "$spec_dir/${package_name}.spec" || die "failed to download source file" + } # Building rpm or srpm packages - rpmbuild -ba $spec_file || exit + rpmbuild -ba "$spec_dir"/*.spec || die "failed to build rpms" } show_rpm_files() @@ -71,7 +107,6 @@ upload_rpm_pkg() done } -init_workspace -download_upstream_repo +[ -n "$repo_name" ] && from_srpm || from_git build_rpm upload_rpm_pkg -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/rpmbuild-pkg: add build way from srpm
by Wang Yong 25 Mar '21

25 Mar '21

original build RPMs from git repo, now support building from SRPM Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 73 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 16 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 256e836bf..282d473fd 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,6 +1,8 @@ #!/bin/bash # - upstream_repo # - compat_os +# - repo_name +# - repo_addr . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh @@ -8,9 +10,38 @@ : "${compat_os:=budding-openeuler}" [ -n "$upstream_repo" ] || die "upstream_repo is empty" -package_name=${upstream_repo##*/} -rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/${os_arch}/Packages" -src_rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/source/Packages" + +dest_dir="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}" + +from_git() +{ + package_name=${upstream_repo##*/} + rpm_dest="${dest_dir}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/source/Packages" + + init_workspace + download_upstream_repo +} + +from_srpm() +{ + [ -n "$repo_name" ] || die "repo_name is empty" + [ -n "$repo_addr" ] || die "repo_addr is empty" + + rpm_dest="${dest_dir}/${repo_name}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/${repo_name}/source/Packages" + + install_srpm +} + +check_flow() +{ + if [ -n "$repo_name" ]; then + from_srpm + else + from_git + fi +} init_workspace() { @@ -20,30 +51,41 @@ init_workspace() download_upstream_repo() { - git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" - cd $package_name || exit + git clone "git://${GIT_SERVER}/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://${GIT_SERVER}/openeuler/${upstream_repo}" + cd "$package_name" || exit filelist=$(git ls-files) + for pkgfile in ${filelist[@]} do local dir="SOURCES" - echo $pkgfile | egrep "\.spec$" && dir="SPECS" + + echo "$pkgfile" | grep -E "\\.spec$" && dir="SPECS" mv "$pkgfile" "${HOME}/rpmbuild/${dir}/" done } -build_rpm() +install_srpm() { - local spec_file=${HOME}/rpmbuild/SPECS/$package_name.spec - - # HTTP is proxy cache friendly - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_file` + rpm -i "${repo_addr}/${upstream_repo}" >/dev/null || die "failed to install source rpm: ${repo_addr}/${upstream_repo}" +} +build_rpm() +{ + local spec_dir="${HOME}/rpmbuild/SPECS" + [ -n "$package_name" ] && + { + # HTTP is proxy cache friendly + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/${package_name}.spec")" + } # Install build depends - yum-builddep -y $spec_file || exit + yum-builddep -y "$spec_dir"/*.spec || die "failed to solve dependencies" # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE - spectool -g -R $spec_file || exit + [ -n "$package_name" ] && + { + spectool -g -R "$spec_dir/${package_name}.spec" || die "failed to download source file" + } # Building rpm or srpm packages - rpmbuild -ba $spec_file || exit + rpmbuild -ba "$spec_dir"/*.spec || die "failed to build rpms" } show_rpm_files() @@ -71,7 +113,6 @@ upload_rpm_pkg() done } -init_workspace -download_upstream_repo +check_flow build_rpm upload_rpm_pkg -- 2.23.0

3 6

[PATCH v5 compass-ci 5/5] doc/manual: update test-oss-project.en.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.en.md | 152 ++++++++++++++++-------------- 1 file changed, 81 insertions(+), 71 deletions(-) diff --git a/doc/manual/test-oss-project.en.md b/doc/manual/test-oss-project.en.md index 1b551b5..a3c0781 100644 --- a/doc/manual/test-oss-project.en.md +++ b/doc/manual/test-oss-project.en.md @@ -1,71 +1,81 @@ -# How DO I Use the Compass-CI Platform to Test Open Source Projects? - -This document describes how to use the Compass-CI platform to test open source projects. - -### Adding the URL of the Repository to Be Tested to the upstream-repos Repository - -Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: - -1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. - -![](./../pictures/fork_blacklight.png) - -2. Run the following command to create a file path named with the repository name and its initial letter: - - ``` - mkdir -p b/backlight - ``` - -3. Run the following command to create the **backlight** file in the directory: - - ``` - cd b/backlight - touch backlight - ``` - -4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: - - ``` - vim backlight - ``` - - The format is as follows: - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - > ![](./../public_sys-resources/icon-notice.gif) **Note** - > - > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. - -5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. - -### Submitting the Test Task to the Compass-CI Platform - -1. Prepare a test case. - - You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) - - * Use the test cases that have been adapted in the repository. - - Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. - - * Compile a test case and add it to the repository. - - For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. - -2. Configure the **auto\_submit.yaml** file and submit the test task. - - You only need to add the following configuration information to the **sbin/auto\_submit.yaml** file in the **compass-ci** repository: - - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - - Submit the modified **auto\_submit.yaml** file to the **compass-ci** repository using Pull Request. Then you can use the Compass-CI platform to test your project. - - For details about how to set parameters in the **auto\_submit.yaml** file, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. +# How Do I Use the Compass-CI Platform to Test Open Source Projects? + +This document describes how to use the Compass-CI platform to test open source projects. + +### Adding the URL of the Repository to Be Tested to the upstream-repos Repository + +Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: + +1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. + +![](./../pictures/fork_blacklight.png) + +2. Run the following command to create a file path named with the repository name and its initial letter: + + ``` + mkdir -p b/backlight + ``` + +3. Run the following command to create the **backlight** file in the directory: + + ``` + cd b/backlight + touch backlight + ``` + +4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: + + ``` + vim backlight + ``` + + The format is as follows: + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + > ![](./../public_sys-resources/icon-notice.gif) **Note** + > + > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. + +5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. + +### Submitting the Test Task to the Compass-CI Platform + +1. Prepare a test case. + + You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) + + * Use the test cases that have been adapted in the repository. + + Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. + + * Compile a test case and add it to the repository. + + For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. + +2. Configure **DEFAULTS** files in **upstream-repos** repository and submit the test task. + + You only need to add **DEFAULTS** file in the directory of file **backlight** we referred to above, and configure it like: + + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + + Submit the modified **DEFAULTS** file to the **upstream-repos** repository using Pull Request. Then you can use the Compass-CI platform to test your project. + + For details about how to configure DEFAULTS files, see https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md. + + For meaning and effect of parameters in the command, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. -- 2.23.0

1 0

[PATCH v5 compass-ci 4/5] doc/manual: update test-oss-project.zh.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.zh.md | 146 ++++++++++++++++-------------- 1 file changed, 78 insertions(+), 68 deletions(-) diff --git a/doc/manual/test-oss-project.zh.md b/doc/manual/test-oss-project.zh.md index aa2cd15..b455c32 100644 --- a/doc/manual/test-oss-project.zh.md +++ b/doc/manual/test-oss-project.zh.md @@ -1,68 +1,78 @@ -# 使用 compass-ci 平台测试开源项目 - -本文介绍了使用 compass-ci 平台测试开源项目的操作。 - -### 添加待测试仓库 URL 到 upstream-repos 仓库 - -执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 - -1. Fork 要测试的代码仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 - -![](./../pictures/fork_blacklight.png) - -2. 执行以下命令，以首字母和仓库名创建文件路径。 - - ``` - mkdir -p b/backlight - ``` - -3. 执行以下命令，在该目录下新建同名文件 backlight。 - ``` - cd b/backlight - touch backlight - ``` - -4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 - - ``` - vim backlight - ``` - 内容格式为 - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - >![](./../public_sys-resources/icon-notice.gif) **注意：** - > - >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 - -5. 通过 Pull Request 命令将新增的 backlight 文件提交到 blacklight 仓库。 - - -### 提交测试任务到 compass-ci 平台 - -1. 准备测试用例 - - 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 - - * 使用仓库中已经适配好的测试用例 - 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： - iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 - - * 编写测试用例并添加到仓库 - - 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) - -2. 配置 auto_submit.yaml 文件，提交测试任务 - - 你只需要在 compass-ci 仓库下面的 sbin/auto_submit.yaml 文件中添加配置信息，如： - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - 通过 Pull Request 的方式将修改好的 auto_submit.yaml 文件提交到 compass-ci 仓库，就可以使用 compass-ci 测试你的项目了。 - - auto_submit.yaml 文件的参数配置请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 +# 使用 compass-ci 平台测试开源项目 + +本文介绍了使用 compass-ci 平台测试开源项目的操作。 + +### 添加待测试仓库 URL 到 upstream-repos 仓库 + +执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 + +1. Fork upstream-repos 仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 + +![](./../pictures/fork_blacklight.png) + +2. 执行以下命令，以首字母和仓库名创建文件路径。 + + ``` + mkdir -p b/backlight + ``` + +3. 执行以下命令，在该目录下新建同名文件 backlight。 + ``` + cd b/backlight + touch backlight + ``` + +4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 + + ``` + vim backlight + ``` + 内容格式为 + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + >![](./../public_sys-resources/icon-notice.gif) **注意：** + > + >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 + +5. 通过 Pull Request 命令将新增的 backlight 文件提交到 upstream-repos 仓库。 + + +### 提交测试任务到 compass-ci 平台 + +1. 准备测试用例 + + 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 + + * 使用仓库中已经适配好的测试用例 + 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： + iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 + + * 编写测试用例并添加到仓库 + + 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) + +2. 配置 upstream-repos仓库中的DEFAULTS文件，提交测试任务 + + 你只需要在上述backlight文件所在目录增加 DEFAULTS 文件并添加配置信息，如： + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + 通过 Pull Request 的方式将修改好的 DEFAULTS 文件提交到 upstream-repos 仓库，就可以使用 compass-ci 测试你的项目了。 + + 详细配置方式请参考 https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md 。 + + 命令参数意义及作用请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 -- 2.23.0

1 0

[PATCH v5 compass-ci 3/5] lib/git_mirror.rb: change the way to calculate priority for repo
by Li Yuanchao 24 Mar '21

24 Mar '21

Change the way to calculate priority for repo, to make active repo be updated more times and more effective. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 50a6c75..5ba9154 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -115,7 +115,6 @@ class MirrorMain def initialize @feedback_queue = Queue.new @fork_stat = {} - @priority = 0 @priority_queue = PriorityQueue.new @git_info = {} @defaults = {} @@ -221,11 +220,9 @@ class MirrorMain def push_git_queue return if @git_queue.size >= 1 - fork_key = @priority_queue.delete_min_return_key + fork_key, old_pri = @priority_queue.delete_min do_push(fork_key) - priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 - @priority_queue.push fork_key, priority_set - @priority += 1 + @priority_queue.push fork_key, get_repo_priority(fork_key, old_pri) end def main_loop @@ -251,8 +248,7 @@ class MirrorMain @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo], belong) fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end def compare_refs(cur_refs, old_refs) @@ -388,8 +384,7 @@ class MirrorMain @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true, 'belong' => belong } fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end end @@ -474,6 +469,8 @@ end # main thread class MirrorMain + WEEK_SECONDS = 604800 + def merge_defaults(object_key, object, belong) return object if object_key == belong @@ -541,4 +538,25 @@ class MirrorMain return true end + + def get_repo_priority(git_repo, old_pri) + old_pri ||= 0 + mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}" + mirror_dir = "#{mirror_dir}.git" unless @git_info[git_repo]['is_submodule'] + + return old_pri + Math.cbrt(WEEK_SECONDS) unless File.directory?(mirror_dir) + + return cal_priority(mirror_dir, old_pri) + end + + def cal_priority(mirror_dir, old_pri) + last_commit_time = %x(git -C #{mirror_dir} log --pretty=format:"%ct" -1 2>/dev/null).to_i + return old_pri + Math.cbrt(WEEK_SECONDS) if last_commit_time.zero? + + t = Time.now.to_i + interval = t - last_commit_time + return old_pri + Math.cbrt(WEEK_SECONDS) if interval <= 0 + + return old_pri + Math.cbrt(interval) + end end -- 2.23.0

1 0

[PATCH v5 compass-ci 2/5] src/lib/web_backend.rb: implement for group jobs stats count
by Li Yuanchao 24 Mar '21

24 Mar '21

Query jobs by conditions such as group id or suite, and count the number of passed cases and failed cases. output is like: { "kezhiming": { "nr_all": $nr_all, "nr_pass": $nr_pass, "nr_fail": $nr_fail }, "chenqun": { "nr_all": $nr_all, "nr_pass": $nr_pass, "nr_fail": $nr_fail }, ... } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 66 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..40471e0 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -121,6 +121,8 @@ end def get_dimension_conditions(params) dimension = params.key?(:dimension) ? [params.delete(:dimension)] : [] + dimension = params.key?(:GROUP_BY) ? [params.delete(:GROUP_BY)] : [] if dimension.empty? + conditions = {} FIELDS.each do |f| v = params[f] @@ -548,3 +550,67 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def single_count(stats) + fail_count = 0 + pass_count = 0 + single_nr_fail = 0 + single_nr_pass = 0 + stats.each do |stat, value| + fail_count += 1 if stat.match(/\.fail$/i) + pass_count += 1 if stat.match(/\.pass$/i) + single_nr_fail = value if stat.match(/\.nr_fail$/i) + single_nr_pass = value if stat.match(/\.nr_pass$/i) + end + fail_count = single_nr_fail.zero? ? fail_count : single_nr_fail + pass_count = single_nr_pass.zero? ? pass_count : single_nr_pass + [fail_count, pass_count, fail_count + pass_count] +end + +def count_stats(job_list) + nr_stats = { 'nr_fail' => 0, 'nr_pass' => 0, 'nr_all' => 0 } + job_list.each do |job| + next unless job['_source']['stats'] + + fail_count, pass_count, all_count = single_count(job['_source']['stats']) + nr_stats['nr_fail'] += fail_count + nr_stats['nr_pass'] += pass_count + nr_stats['nr_all'] += all_count + end + nr_stats +end + +def get_jobs_stats_count(dimension, must, size, from) + dimension_list = get_dimension_list(dimension) + stats_count = {} + dimension_list.each do |dim| + job_list = query_dimension(dimension[0], dim, must, size, from) + stats_count[dim] = count_stats(job_list) + end + stats_count.to_json +end + +def get_stats_by_dimension(conditions, dimension, must, size, from) + must += build_multi_field_subquery_body(conditions) + count_query = { query: { bool: { must: must } } } + total = es_count(count_query) + return {} if total < 1 + + get_jobs_stats_count(dimension, must, size, from) +end + +def get_jobs_stats(params) + dimension, conditions = get_dimension_conditions(params) + must = get_es_must(params) + get_stats_by_dimension(conditions, dimension, must, 1000, 0) +end + +def group_jobs_stats(params) + begin + body = get_jobs_stats(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH v5 compass-ci 1/5] container/web-backend: add api for group jobs stats count
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..adb91e0 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,28 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_jobs_summary?suite=iperf&GROUP_BY=my_name +# must: +# - query_conditions +# - suite / group_id / ... +# - GROUP_BY +# - group_id / my_email / my_name +# +# Response like: +# - { +# "kezhiming": { +# "nr_all": $nr_all, +# "nr_pass": $nr_pass, +# "nr_fail: $nr_fail +# }, +# "chenqun": { +# "nr_all": $nr_all, +# "nr_pass": $nr_pass, +# "nr_fail": $nr_fail +# }, +# ... +# } +get '/get_jobs_summary' do + group_jobs_stats(params) +end -- 2.23.0

1 0

[PATCH v4 compass-ci 5/5] doc/manual: update test-oss-project.en.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.en.md | 152 ++++++++++++++++-------------- 1 file changed, 81 insertions(+), 71 deletions(-) diff --git a/doc/manual/test-oss-project.en.md b/doc/manual/test-oss-project.en.md index 1b551b5..7b72e2b 100644 --- a/doc/manual/test-oss-project.en.md +++ b/doc/manual/test-oss-project.en.md @@ -1,71 +1,81 @@ -# How DO I Use the Compass-CI Platform to Test Open Source Projects? - -This document describes how to use the Compass-CI platform to test open source projects. - -### Adding the URL of the Repository to Be Tested to the upstream-repos Repository - -Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: - -1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. - -![](./../pictures/fork_blacklight.png) - -2. Run the following command to create a file path named with the repository name and its initial letter: - - ``` - mkdir -p b/backlight - ``` - -3. Run the following command to create the **backlight** file in the directory: - - ``` - cd b/backlight - touch backlight - ``` - -4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: - - ``` - vim backlight - ``` - - The format is as follows: - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - > ![](./../public_sys-resources/icon-notice.gif) **Note** - > - > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. - -5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. - -### Submitting the Test Task to the Compass-CI Platform - -1. Prepare a test case. - - You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) - - * Use the test cases that have been adapted in the repository. - - Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. - - * Compile a test case and add it to the repository. - - For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. - -2. Configure the **auto\_submit.yaml** file and submit the test task. - - You only need to add the following configuration information to the **sbin/auto\_submit.yaml** file in the **compass-ci** repository: - - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - - Submit the modified **auto\_submit.yaml** file to the **compass-ci** repository using Pull Request. Then you can use the Compass-CI platform to test your project. - - For details about how to set parameters in the **auto\_submit.yaml** file, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. +# How DO I Use the Compass-CI Platform to Test Open Source Projects? + +This document describes how to use the Compass-CI platform to test open source projects. + +### Adding the URL of the Repository to Be Tested to the upstream-repos Repository + +Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: + +1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. + +![](./../pictures/fork_blacklight.png) + +2. Run the following command to create a file path named with the repository name and its initial letter: + + ``` + mkdir -p b/backlight + ``` + +3. Run the following command to create the **backlight** file in the directory: + + ``` + cd b/backlight + touch backlight + ``` + +4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: + + ``` + vim backlight + ``` + + The format is as follows: + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + > ![](./../public_sys-resources/icon-notice.gif) **Note** + > + > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. + +5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. + +### Submitting the Test Task to the Compass-CI Platform + +1. Prepare a test case. + + You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) + + * Use the test cases that have been adapted in the repository. + + Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. + + * Compile a test case and add it to the repository. + + For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. + +2. Configure **DEFAULTS** files in **upstream-repos** repository and submit the test task. + + You only need to add **DEFAULTS** file in the directory of file **backlight** we referred to above, and configure it like: + + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + + Submit the modified **DEFAULTS** file to the **upstream-repos** repository using Pull Request. Then you can use the Compass-CI platform to test your project. + + For details about how to configure DEFAULTS files, see https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md. + + For meaning and effect of parameters in the command, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. -- 2.23.0

2 2

[PATCH v2 compass-ci] container/srv-http/Dockerfile: build h5ai by builder
by Lu Weitao 24 Mar '21

24 Mar '21

[Why] use node:alpine as a Builder for build h5ai we can reduce install app such: nginx, node, npm to save deploy time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/Dockerfile | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index 6564730..b7d0677 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -1,25 +1,32 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.9 +FROM node:alpine as Builder + +MAINTAINER Lu Weitao <luweitaobe(a)163.com> + +RUN apk add git + +RUN git clone https://gitee.com/luweitao_y/h5ai.git /h5ai + +WORKDIR /h5ai + +RUN npm install && \ + npm run build + +FROM nginx:alpine ADD root / -RUN apk add --no-cache nginx php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo nodejs npm git \ - && mkdir /run/nginx/ +RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf RUN sed -i "s/user = nobody/user = nginx/g" /etc/php7/php-fpm.d/www.conf RUN sed -i "s/group = nobody/group = nginx/g" /etc/php7/php-fpm.d/www.conf -RUN git clone https://gitee.com/luweitao_y/h5ai.git /h5ai -WORKDIR /h5ai +COPY --from=Builder /h5ai/build/_h5ai /srv/_h5ai -# build h5ai package -RUN npm install \ - && npm run build \ - && cp -r ./build/_h5ai /srv \ - && chown -R nginx:nginx /srv/_h5ai +RUN chown -R nginx:nginx /srv/_h5ai ENTRYPOINT ["/sbin/entrypoint.sh"] -- 2.23.0

1 0

[PATCH compass-ci] container/srv-http/Dockerfile: build h5ai by builder
by Lu Weitao 24 Mar '21

24 Mar '21

[Why] use node:alpine as a Builder for build h5ai we can reduce install app such: nginx, node, npm to save deploy time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/Dockerfile | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index 6564730..81a8a45 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -1,25 +1,30 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.9 +FROM node:alpine as Builder + +RUN apk add git + +RUN git clone https://gitee.com/luweitao_y/h5ai.git /h5ai + +WORKDIR /h5ai + +RUN npm install && \ + npm run build + +FROM nginx:alpine ADD root / -RUN apk add --no-cache nginx php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo nodejs npm git \ - && mkdir /run/nginx/ +RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf RUN sed -i "s/user = nobody/user = nginx/g" /etc/php7/php-fpm.d/www.conf RUN sed -i "s/group = nobody/group = nginx/g" /etc/php7/php-fpm.d/www.conf -RUN git clone https://gitee.com/luweitao_y/h5ai.git /h5ai -WORKDIR /h5ai +COPY --from=Builder /h5ai/build/_h5ai /srv/_h5ai -# build h5ai package -RUN npm install \ - && npm run build \ - && cp -r ./build/_h5ai /srv \ - && chown -R nginx:nginx /srv/_h5ai +RUN chown -R nginx:nginx /srv/_h5ai ENTRYPOINT ["/sbin/entrypoint.sh"] -- 2.23.0

2 2

[PATCH v4 compass-ci 2/5] src/lib/web_backend.rb: implement for group jobs stats count
by Li Yuanchao 24 Mar '21

24 Mar '21

Query jobs by conditions such as group id or suite, and count the number of passed cases and failed cases. output is like: { "kezhiming": { "nr_all": $nr_all, "nr_pass": $nr_pass, "nr_fail": $nr_fail }, "chenqun": { "nr_all": $nr_all, "nr_pass": $nr_pass, "nr_fail": $nr_fail }, ... } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 68 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..04bccdb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -121,6 +121,8 @@ end def get_dimension_conditions(params) dimension = params.key?(:dimension) ? [params.delete(:dimension)] : [] + dimension = params.key?(:GROUP_BY) ? [params.delete(:GROUP_BY)] : [] if dimension.empty? + conditions = {} FIELDS.each do |f| v = params[f] @@ -548,3 +550,69 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def single_count(stats) + fail_count = 0 + pass_count = 0 + single_nr_fail = 0 + single_nr_pass = 0 + stats.each do |stat, value| + fail_count += 1 if stat.match(/\.fail$/i) + pass_count += 1 if stat.match(/\.pass$/i) + single_nr_fail = value if stat.match(/\.nr_fail$/i) + single_nr_pass = value if stat.match(/\.nr_pass$/i) + end + fail_count = single_nr_fail.zero? ? fail_count : single_nr_fail + pass_count = single_nr_pass.zero? ? pass_count : single_nr_pass + [fail_count, pass_count, fail_count + pass_count] +end + +def count_stats(job_list) + nr_fail = 0 + nr_pass = 0 + nr_all = 0 + job_list.each do |job| + next unless job['_source']['stats'] + + fail_count, pass_count, all_count = single_count(job['_source']['stats']) + nr_fail += fail_count + nr_pass += pass_count + nr_all += all_count + end + { 'nr_fail' => nr_fail, 'nr_pass' => nr_pass, 'nr_all' => nr_all } +end + +def get_jobs_stats_count(dimension, must, size, from) + dimension_list = get_dimension_list(dimension) + stats_count = {} + dimension_list.each do |dim| + job_list = query_dimension(dimension[0], dim, must, size, from) + stats_count[dim] = count_stats(job_list) + end + stats_count.to_json +end + +def get_stats_by_dimension(conditions, dimension, must, size, from) + must += build_multi_field_subquery_body(conditions) + count_query = { query: { bool: { must: must } } } + total = es_count(count_query) + return {} if total < 1 + + get_jobs_stats_count(dimension, must, size, from) +end + +def get_jobs_stats(params) + dimension, conditions = get_dimension_conditions(params) + must = get_es_must(params) + get_stats_by_dimension(conditions, dimension, must, 1000, 0) +end + +def group_jobs_stats(params) + begin + body = get_jobs_stats(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

2 2

[PATCH lkp-tests] tests/rpmbuild-pkg: add build way from srpm
by Wang Yong 24 Mar '21

24 Mar '21

original build RPMs from git repo, now support building from SRPM Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 73 ++++++++++++++++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 16 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 256e836bf..17169a31b 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,6 +1,8 @@ #!/bin/bash # - upstream_repo # - compat_os +# - repo_name +# - repo_addr . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh @@ -8,9 +10,38 @@ : "${compat_os:=budding-openeuler}" [ -n "$upstream_repo" ] || die "upstream_repo is empty" -package_name=${upstream_repo##*/} -rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/${os_arch}/Packages" -src_rpm_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}/source/Packages" + +dest_dir="/initrd/rpmbuild-pkg/${os}-${os_version}/${compat_os}" + +from_git() +{ + package_name=${upstream_repo##*/} + rpm_dest="${dest_dir}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/source/Packages" + + init_workspace + download_upstream_repo +} + +from_srpm() +{ + [ -n "$repo_name" ] || die "repo_name is empty" + [ -n "$repo_addr" ] || die "repo_addr is empty" + + rpm_dest="${dest_dir}/${repo_name}/${os_arch}/Packages" + src_rpm_dest="${dest_dir}/${repo_name}/source/Packages" + + install_srpm +} + +check_flow() +{ + if [ -n "$repo_name" ]; then + from_srpm + else + from_git + fi +} init_workspace() { @@ -20,30 +51,41 @@ init_workspace() download_upstream_repo() { - git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://$GIT_SERVER/openeuler/${upstream_repo}" - cd $package_name || exit + git clone "git://${GIT_SERVER}/openeuler/${upstream_repo}" || die "clone git repo ${package_name} failed: git://${GIT_SERVER}/openeuler/${upstream_repo}" + cd "$package_name" || exit filelist=$(git ls-files) + for pkgfile in ${filelist[@]} do local dir="SOURCES" - echo $pkgfile | egrep "\.spec$" && dir="SPECS" + + echo "$pkgfile" | grep -E "\\.spec$" && dir="SPECS" mv "$pkgfile" "${HOME}/rpmbuild/${dir}/" done } -build_rpm() +install_srpm() { - local spec_file=${HOME}/rpmbuild/SPECS/$package_name.spec - - # HTTP is proxy cache friendly - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' `grep http -rl $spec_file` + rpm -i "${repo_addr}/${upstream_repo}" >/dev/null || die "failed to install source rpm" +} +build_rpm() +{ + local spec_dir="${HOME}/rpmbuild/SPECS" + [ -n "$package_name" ] && + { + # HTTP is proxy cache friendly + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/${package_name}.spec")" + } # Install build depends - yum-builddep -y $spec_file || exit + yum-builddep -y "$spec_dir"/*.spec || die "failed to solve dependencies" # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE - spectool -g -R $spec_file || exit + [ -n "$package_name" ] && + { + spectool -g -R "$spec_dir/${package_name}.spec" || die "failed to download source file" + } # Building rpm or srpm packages - rpmbuild -ba $spec_file || exit + rpmbuild -ba "$spec_dir"/*.spec || die "failed to build rpms" } show_rpm_files() @@ -71,7 +113,6 @@ upload_rpm_pkg() done } -init_workspace -download_upstream_repo +check_flow build_rpm upload_rpm_pkg -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] provides: fix require error
by Wu Zhende 24 Mar '21

24 Mar '21

[Error] can't load such file "../lib/mq_client" [How] use require_relative Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/multi-qemu | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/multi-qemu b/providers/multi-qemu index d3f8406..01ba3d5 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -7,7 +7,7 @@ require 'fileutils' require 'optparse' require 'json' -require '../lib/mq_client' +require_relative "../lib/mq_client" opt = {} options = OptionParser.new do |opts| -- 2.23.0

2 1

[PATCH v3 compass-ci 1/2] service/sshr: change port
by Wu Zhende 24 Mar '21

24 Mar '21

Restarting the sshr service will disconnect the existing sshr connection and the machine that has been applied for can't be accessed. Therefore, we can't restart the sshr service. Instead, we start a new sshr service and use a new port. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 2 +- container/scheduler/start | 4 ++-- container/ssh-r/start | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index ee93e79..4ec84a2 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -38,7 +38,7 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT'] ||= 5051 defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' diff --git a/container/scheduler/start b/container/scheduler/start index e655dd4..8d800f9 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -31,8 +31,8 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 -defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT'] ||= 5051 +defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' diff --git a/container/ssh-r/start b/container/ssh-r/start index 6c7a373..68a63f2 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -16,7 +16,7 @@ cmd=( -e TCP_FORWARDING=true -d -p 21000-23999:21000-23999 - -p 5050:22 + -p 5051:22 -v /etc/localtime:/etc/localtime:ro -v /srv/pub/sshr/keys/:/etc/ssh/keys ssh-r:0.001 -- 2.23.0

1 0

[PATCH v2 compass-ci] container/mail-robot: update port
by Luan Shengde 24 Mar '21

24 Mar '21

services have assigned new port: assign-account: 29999 => 10000 send-internet-mail: 49000 => 10001 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 2 +- container/mail-robot/lib/apply-jumper-account.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 97c3989..798546a 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -12,7 +12,7 @@ require_relative 'assign-account-fail-eamil' require_relative 'apply-jumper-account' require_relative 'parse-apply-account-email' -SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_PORT = ENV['SEND_MAIL_PORT'] || 10001 # assign uuid/account for user # when mail-robot listened new email, and the email's subject diff --git a/container/mail-robot/lib/apply-jumper-account.rb b/container/mail-robot/lib/apply-jumper-account.rb index 73d2946..9043309 100755 --- a/container/mail-robot/lib/apply-jumper-account.rb +++ b/container/mail-robot/lib/apply-jumper-account.rb @@ -6,7 +6,7 @@ require 'json' JUMPER_HOST = ENV['JUMPER_HOST'] || 'api.compass-ci.openeuler.org' -JUMPER_PORT = ENV['JUMPER_PORT'] || 29999 +JUMPER_PORT = ENV['JUMPER_PORT'] || 10000 # used to apply account # be called after AssignAccount successfully parsed my_commit_url and my_ssh_pubkey -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] service/sshr: change port
by Wu Zhende 24 Mar '21

24 Mar '21

Restarting the sshr service will disconnect the existing sshr connection and the machine that has been applied for can't be accessed. Therefore, we can't restart the sshr service. Instead, we start a new sshr service and use a new port. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 2 +- container/scheduler/start | 4 ++-- container/ssh-r/start | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index ee93e79..4ec84a2 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -38,7 +38,7 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 +defaults['SSHR_PORT'] ||= 5051 defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' diff --git a/container/scheduler/start b/container/scheduler/start index e655dd4..8d800f9 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -31,8 +31,8 @@ names = Set.new %w[ ] defaults = relevant_defaults(names) -defaults['SSHR_PORT'] ||= 5050 -defaults['SSHR_PORT_BASE'] ||= 50000 +defaults['SSHR_PORT'] ||= 5051 +defaults['SSHR_PORT_BASE'] ||= 21000 defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' diff --git a/container/ssh-r/start b/container/ssh-r/start index 6c7a373..68a63f2 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -16,7 +16,7 @@ cmd=( -e TCP_FORWARDING=true -d -p 21000-23999:21000-23999 - -p 5050:22 + -p 5051:22 -v /etc/localtime:/etc/localtime:ro -v /srv/pub/sshr/keys/:/etc/ssh/keys ssh-r:0.001 -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] provides: fix require error
by Wu Zhende 24 Mar '21

24 Mar '21

[Error] can't load such file "../lib/mq_client" [How] use absolute path Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 4 +++- providers/multi-qemu | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 9d6854f..b372a11 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -3,12 +3,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +CCI_SRC = ENV['CCI_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + require 'open-uri' require 'json' require 'set' require 'fileutils' -require '../lib/mq_client' +require "#{CCI_SRC}/lib/mq_client" require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' diff --git a/providers/multi-qemu b/providers/multi-qemu index d3f8406..70675d1 100755 --- a/providers/multi-qemu +++ b/providers/multi-qemu @@ -3,11 +3,13 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +CCI_SRC = ENV['CCI_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + require 'fileutils' require 'optparse' require 'json' -require '../lib/mq_client' +require "#{CCI_SRC}/lib/mq_client" opt = {} options = OptionParser.new do |opts| -- 2.23.0

2 1

[PATCH v4 compass-ci 4/5] doc/manual: update test-oss-project.zh.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.zh.md | 146 ++++++++++++++++-------------- 1 file changed, 78 insertions(+), 68 deletions(-) diff --git a/doc/manual/test-oss-project.zh.md b/doc/manual/test-oss-project.zh.md index aa2cd15..b455c32 100644 --- a/doc/manual/test-oss-project.zh.md +++ b/doc/manual/test-oss-project.zh.md @@ -1,68 +1,78 @@ -# 使用 compass-ci 平台测试开源项目 - -本文介绍了使用 compass-ci 平台测试开源项目的操作。 - -### 添加待测试仓库 URL 到 upstream-repos 仓库 - -执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 - -1. Fork 要测试的代码仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 - -![](./../pictures/fork_blacklight.png) - -2. 执行以下命令，以首字母和仓库名创建文件路径。 - - ``` - mkdir -p b/backlight - ``` - -3. 执行以下命令，在该目录下新建同名文件 backlight。 - ``` - cd b/backlight - touch backlight - ``` - -4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 - - ``` - vim backlight - ``` - 内容格式为 - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - >![](./../public_sys-resources/icon-notice.gif) **注意：** - > - >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 - -5. 通过 Pull Request 命令将新增的 backlight 文件提交到 blacklight 仓库。 - - -### 提交测试任务到 compass-ci 平台 - -1. 准备测试用例 - - 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 - - * 使用仓库中已经适配好的测试用例 - 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： - iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 - - * 编写测试用例并添加到仓库 - - 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) - -2. 配置 auto_submit.yaml 文件，提交测试任务 - - 你只需要在 compass-ci 仓库下面的 sbin/auto_submit.yaml 文件中添加配置信息，如： - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - 通过 Pull Request 的方式将修改好的 auto_submit.yaml 文件提交到 compass-ci 仓库，就可以使用 compass-ci 测试你的项目了。 - - auto_submit.yaml 文件的参数配置请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 +# 使用 compass-ci 平台测试开源项目 + +本文介绍了使用 compass-ci 平台测试开源项目的操作。 + +### 添加待测试仓库 URL 到 upstream-repos 仓库 + +执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 + +1. Fork upstream-repos 仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 + +![](./../pictures/fork_blacklight.png) + +2. 执行以下命令，以首字母和仓库名创建文件路径。 + + ``` + mkdir -p b/backlight + ``` + +3. 执行以下命令，在该目录下新建同名文件 backlight。 + ``` + cd b/backlight + touch backlight + ``` + +4. 执行以下命令，将 backlight 仓库 url 信息写入 backlight 文件。 + + ``` + vim backlight + ``` + 内容格式为 + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + >![](./../public_sys-resources/icon-notice.gif) **注意：** + > + >可参考 upstream-repos 仓库中已有文件格式,请保持格式一致。 + +5. 通过 Pull Request 命令将新增的 backlight 文件提交到 upstream-repos 仓库。 + + +### 提交测试任务到 compass-ci 平台 + +1. 准备测试用例 + + 测试用例可以自己编写并添加到 lkp-tests 仓库，也可以直接使用 lkp-tests 仓库（https://gitee.com/wu_fengguang/lkp-tests ）的 jobs 目录下已有的测试用例。 + + * 使用仓库中已经适配好的测试用例 + 如果 lkp-tests 仓库中正好有你想要的测试用例，你可以直接使用。以 iperf.yaml 文件为例说明如下： + iperf.yaml 是一个已经适配好的测试用例，它位于 lkp-tests 仓库的 jobs 目录下，其中有一些基本的测试参数。 + + * 编写测试用例并添加到仓库 + + 请参考：[如何添加测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… ) + +2. 配置 upstream-repos仓库中的DEFAULTS文件，提交测试任务 + + 你只需要在上述backlight文件所在目录增加 DEFAULTS 文件并添加配置信息，如： + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + 通过 Pull Request 的方式将修改好的 DEFAULTS 文件提交到 upstream-repos 仓库，就可以使用 compass-ci 测试你的项目了。 + + 详细配置方式请参考 https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md 。 + + 命令参数意义及作用请参考 https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job 。 -- 2.23.0

1 0

[PATCH v4 compass-ci 3/5] lib/git_mirror.rb: change the way to calculate priority for repo
by Li Yuanchao 24 Mar '21

24 Mar '21

Change the way to calculate priority for repo, to make active repo be updated more times and more effective. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 50a6c75..5ba9154 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -115,7 +115,6 @@ class MirrorMain def initialize @feedback_queue = Queue.new @fork_stat = {} - @priority = 0 @priority_queue = PriorityQueue.new @git_info = {} @defaults = {} @@ -221,11 +220,9 @@ class MirrorMain def push_git_queue return if @git_queue.size >= 1 - fork_key = @priority_queue.delete_min_return_key + fork_key, old_pri = @priority_queue.delete_min do_push(fork_key) - priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 - @priority_queue.push fork_key, priority_set - @priority += 1 + @priority_queue.push fork_key, get_repo_priority(fork_key, old_pri) end def main_loop @@ -251,8 +248,7 @@ class MirrorMain @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo], belong) fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end def compare_refs(cur_refs, old_refs) @@ -388,8 +384,7 @@ class MirrorMain @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true, 'belong' => belong } fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end end @@ -474,6 +469,8 @@ end # main thread class MirrorMain + WEEK_SECONDS = 604800 + def merge_defaults(object_key, object, belong) return object if object_key == belong @@ -541,4 +538,25 @@ class MirrorMain return true end + + def get_repo_priority(git_repo, old_pri) + old_pri ||= 0 + mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}" + mirror_dir = "#{mirror_dir}.git" unless @git_info[git_repo]['is_submodule'] + + return old_pri + Math.cbrt(WEEK_SECONDS) unless File.directory?(mirror_dir) + + return cal_priority(mirror_dir, old_pri) + end + + def cal_priority(mirror_dir, old_pri) + last_commit_time = %x(git -C #{mirror_dir} log --pretty=format:"%ct" -1 2>/dev/null).to_i + return old_pri + Math.cbrt(WEEK_SECONDS) if last_commit_time.zero? + + t = Time.now.to_i + interval = t - last_commit_time + return old_pri + Math.cbrt(WEEK_SECONDS) if interval <= 0 + + return old_pri + Math.cbrt(interval) + end end -- 2.23.0

1 0

[PATCH v4 compass-ci 1/5] container/web-backend: add api for group jobs stats count
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..adb91e0 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,28 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_jobs_summary?suite=iperf&GROUP_BY=my_name +# must: +# - query_conditions +# - suite / group_id / ... +# - GROUP_BY +# - group_id / my_email / my_name +# +# Response like: +# - { +# "kezhiming": { +# "nr_all": $nr_all, +# "nr_pass": $nr_pass, +# "nr_fail: $nr_fail +# }, +# "chenqun": { +# "nr_all": $nr_all, +# "nr_pass": $nr_pass, +# "nr_fail": $nr_fail +# }, +# ... +# } +get '/get_jobs_summary' do + group_jobs_stats(params) +end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] container/web-backend: add api for group jobs stats count
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..49c07a7 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,28 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_jobs_summary?suite=iperf&GROUP_BY=my_name +# must: +# - query_conditions +# - suite / group_id / ... +# - GROUP_BY +# - group_id / my_email / my_name +# +# Response like: +# - { +# "kezhiming" : { +# "nr_all" : $nr_all, +# "nr_pass" : $nr_pass, +# "nr_fail" : $nr_fail +# }, +# "chenqun" : { +# "nr_all" : $nr_all, +# "nr_pass" : $nr_pass, +# "nr_fail" : $nr_fail +# }, +# ... +# } +get '/get_jobs_summary' do + group_jobs_stats(params) +end -- 2.23.0

2 2

[PATCH compass-ci] doc/manual: update test-oss-project.en.md
by Li Yuanchao 24 Mar '21

24 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.en.md | 152 ++++++++++++++++-------------- 1 file changed, 81 insertions(+), 71 deletions(-) diff --git a/doc/manual/test-oss-project.en.md b/doc/manual/test-oss-project.en.md index 1b551b5..7b72e2b 100644 --- a/doc/manual/test-oss-project.en.md +++ b/doc/manual/test-oss-project.en.md @@ -1,71 +1,81 @@ -# How DO I Use the Compass-CI Platform to Test Open Source Projects? - -This document describes how to use the Compass-CI platform to test open source projects. - -### Adding the URL of the Repository to Be Tested to the upstream-repos Repository - -Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: - -1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. - -![](./../pictures/fork_blacklight.png) - -2. Run the following command to create a file path named with the repository name and its initial letter: - - ``` - mkdir -p b/backlight - ``` - -3. Run the following command to create the **backlight** file in the directory: - - ``` - cd b/backlight - touch backlight - ``` - -4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: - - ``` - vim backlight - ``` - - The format is as follows: - - ``` - --- - url: - - https://github.com/baskerville/backlight - ``` - - > ![](./../public_sys-resources/icon-notice.gif) **Note** - > - > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. - -5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. - -### Submitting the Test Task to the Compass-CI Platform - -1. Prepare a test case. - - You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) - - * Use the test cases that have been adapted in the repository. - - Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. - - * Compile a test case and add it to the repository. - - For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. - -2. Configure the **auto\_submit.yaml** file and submit the test task. - - You only need to add the following configuration information to the **sbin/auto\_submit.yaml** file in the **compass-ci** repository: - - ``` - b/backlight/backlight: - - testbox=vm-2p8g os=openEuler os_version=20.03 os_mount=initramfs os_arch=aarch64 iperf.yaml - ``` - - Submit the modified **auto\_submit.yaml** file to the **compass-ci** repository using Pull Request. Then you can use the Compass-CI platform to test your project. - - For details about how to set parameters in the **auto\_submit.yaml** file, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. +# How DO I Use the Compass-CI Platform to Test Open Source Projects? + +This document describes how to use the Compass-CI platform to test open source projects. + +### Adding the URL of the Repository to Be Tested to the upstream-repos Repository + +Perform the following steps to add the information of the code repository to be tested to the **upstream-repos** repository (https://gitee.com/wu_fengguang/upstream-repos) in YAML format: + +1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. + +![](./../pictures/fork_blacklight.png) + +2. Run the following command to create a file path named with the repository name and its initial letter: + + ``` + mkdir -p b/backlight + ``` + +3. Run the following command to create the **backlight** file in the directory: + + ``` + cd b/backlight + touch backlight + ``` + +4. Run the following command to write the URL of the **backlight** repository to the **backlight** file: + + ``` + vim backlight + ``` + + The format is as follows: + + ``` + --- + url: + - https://github.com/baskerville/backlight + ``` + + > ![](./../public_sys-resources/icon-notice.gif) **Note** + > + > You can refer to the existing file format in the **upstream-repos** repository. Ensure that the formats are consistent. + +5. Run the **Pull Request** command to submit the new **backlight** file to the **upstream-repos** repository. + +### Submitting the Test Task to the Compass-CI Platform + +1. Prepare a test case. + + You can compile and add a test case to the **lkp-tests** repository, or directly use the existing test cases in the **jobs** directory of the **lkp-tests** repository (https://gitee.com/wu_fengguang/lkp-tests) + + * Use the test cases that have been adapted in the repository. + + Use the test cases in the **lkp-tests** repository that meet the requirements. The **iperf.yaml** file is used as an example. The **iperf.yaml** file is a test case that has been adapted. It is stored in the **jobs** directory of the **lkp-tests** repository, and contains some basic test parameters. + + * Compile a test case and add it to the repository. + + For details, see [How To Add Test Cases](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. + +2. Configure **DEFAULTS** files in **upstream-repos** repository and submit the test task. + + You only need to add **DEFAULTS** file in the directory of file **backlight** we referred to above, and configure it like: + + ``` + submit: + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 api-avx2neon.yaml + branches: + - master + - next + - command: testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 other-avx2neon.yaml + branches: + - branch_name_a + - branch_name_b + + ``` + + Submit the modified **DEFAULTS** file to the **upstream-repos** repository using Pull Request. Then you can use the Compass-CI platform to test your project. + + For details about how to configure DEFAULTS files, see https://gitee.com/wu_fengguang/upstream-repos/blob/master/README.md. + + For meaning and effect of parameters in the command, see https://gitee.com/wu_fengguang/compass-ci/tree/master/doc/job. -- 2.23.0

1 0