- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] fix: sched: submit idle jobs
by Ren Wen 29 Sep '20

29 Sep '20

1) submit all idles jobs once use '*' -> submit each idle job once until all done. 2) add two fields into job if job's filename is 'multi-qemu.yaml'. 3) add field 'idle_job' for idle jobs, value is always 'true'. Two extra fields are used for: monitoring logs of virtual machines started by physical machine. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 15f5c9e..6ca20d7 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -467,10 +467,18 @@ class Sched end def auto_submit_idle_job(tbox_group) - full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" - Jobfile::Operate.auto_submit_job( - full_path_patterns, - "testbox: #{tbox_group}") if Dir.glob(full_path_patterns).size > 0 + idle_job_path = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/" + + return nil unless Dir.exists?(idle_job_path) + jobs = Dir[idle_job_path + "*.yaml"] + jobs.each do |job| + extra_job_fields = ["idle_job=true"] + if Path[job].basename == "multi-qemu.yaml" + extra_job_fields += ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", + "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}"] + end + Jobfile::Operate.auto_submit_job(job, "testbox: #{tbox_group}", extra_job_fields) + end end private def add_kernel_console_param(arch_tmp) @@ -520,7 +528,7 @@ class Sched "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" else response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" end response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" -- 2.23.0

2 3

[PATCH compass-ci] fix: sched: submit idle jobs
by Ren Wen 29 Sep '20

29 Sep '20

1) submit all idles jobs once use '*' -> submit each idle job once until all done. 2) add two fields into job if job's name is multi-qemu. job's name 'multi-qemu' is case-insensitive matching now. 3) `.yaml` and `.yml` can both match. Two extra fields are used for: monitoring logs of virtual machines started by physical machine. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 15f5c9e..c27dd99 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -467,10 +467,20 @@ class Sched end def auto_submit_idle_job(tbox_group) - full_path_patterns = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/*.yaml" - Jobfile::Operate.auto_submit_job( - full_path_patterns, - "testbox: #{tbox_group}") if Dir.glob(full_path_patterns).size > 0 + idle_job_path = "#{ENV["CCI_REPOS"]}/lab-#{ENV["lab"]}/allot/idle/#{tbox_group}/" + + if Dir.exists?(idle_job_path) + jobs = Dir[idle_job_path + "*.y{a,}ml"] + extra_job_fields = ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", + "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}"] + + jobs.each do |job| + if Path[job].basename =~ /^multi-qemu\.ya?ml/i + Jobfile::Operate.auto_submit_job(job, "testbox: #{tbox_group}", extra_job_fields) + end + Jobfile::Operate.auto_submit_job(job, "testbox: #{tbox_group}") + end + end end private def add_kernel_console_param(arch_tmp) @@ -520,7 +530,7 @@ class Sched "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{job.os_dir}/vmlinuz")}" else response += "kernel #{os_http_prefix}" + - "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" + "#{JobHelper.service_path("#{SRV_OS}/#{job.os_dir}/vmlinuz")}" end response += " user=lkp" response += " job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" -- 2.23.0

3 10

[PATCH compass-ci] check if author's email address exist in upstream repos
by Li Ping 29 Sep '20

29 Sep '20

--- lib/git.rb | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/lib/git.rb b/lib/git.rb index a181ec9..4939c94 100644 --- a/lib/git.rb +++ b/lib/git.rb @@ -2,7 +2,7 @@ # wrap common git commands class GitCommit - def initialize(repo, commit) + def initialize(repo, commit = nil) @repo = repo @commit = commit @git_prefix = cd_repo_path @@ -28,32 +28,42 @@ class GitCommit def cd_repo_path find_repo = check_repo - return unless find_repo + message = "The repo is not found in the upstream-repo" + + raise StandardError, message unless find_repo "git -C #{find_repo}" end + def check_author_in_upstream_repos(email) + author_email = %x(#{@git_prefix} log | grep '^Author'| grep #{email}).chomp + + message = "The author's email address is not found in the upstream-repo" + + raise StandardError, message if author_email.empty? + end + def author_name - `#{@git_prefix} log -n1 --pretty=format:'%an' #{(a)commit}`.chomp + %x(#{git_prefix} log -n1 --pretty=format:'%an' #{(a)commit}).chomp end def author_email - `#{@git_prefix} log -n1 --pretty=format:'%ae' #{(a)commit}`.chomp + %x(#{@git_prefix} log -n1 --pretty=format:'%ae' #{(a)commit}).chomp end def subject - `#{@git_prefix} log -n1 --pretty=format:'%s' #{(a)commit}`.chomp + %x(#{@git_prefix} log -n1 --pretty=format:'%s' #{(a)commit}).chomp end def commit_time - `#{@git_prefix} log -n1 --pretty=format:'%ci' #{(a)commit}`.chomp + %x(#{@git_prefix} log -n1 --pretty=format:'%ci' #{(a)commit}).chomp end def url - `#{@git_prefix} remote -v`.split[1] + %x(#{@git_prefix} remote -v).split[1] end def diff - `#{@git_prefix} diff -1 #{@commit}~..#{@commit}`.chomp + %x(#{@git_prefix} diff -1 #{@commit}~..#{@commit}).chomp end end -- 2.23.0

1 0

[PATCH compass-ci] sparrow: ignore containers when deploy
by Liu Yinsi 29 Sep '20

29 Sep '20

scheduler-https, conserver, web-backend-nginx no need to deploy. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 230c6ac..7698c08 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -29,7 +29,10 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd $container - [ "$container_name" == 'sub-fluentd' ] && exit + [ "$container_name" == 'sub-fluentd' ] || \ + [ "$container_name" == 'conserver' ] || \ + [ "$container_name" == 'web-backend-nginx' ] || \ + [ "$container_name" == 'scheduler-https' ] && exit [ -x build ] && ./build [ -x install ] && ./install [ -x first-run ] && ./first-run -- 2.23.0

3 4

[PATCH v3 compass-ci 2/2] container/extract-stats: add 'LAB' in environment variables
by Lu Weitao 29 Sep '20

29 Sep '20

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/extract-stats/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/extract-stats/start b/container/extract-stats/start index 25bdd25..6bb9f10 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -15,6 +15,7 @@ names = Set.new %w[ FLUENTD_PORT MAIL_HOST MAIL_PORT + LAB ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] src/extract-stats: fix can not send delimiter task
by Lu Weitao 29 Sep '20

29 Sep '20

problem: stats_worker call add_task() of TaskQueueAPI to send delimiter task, but no task added in delimiter task queue. cause analysis: the latest TaskQueueAPI.add_task(service_queue_path, task), require {"lab" => $LAB} in task content, otherwise return error message: "409, need lab in task content" in fact, we miss "lab", so can not send task to task queue. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..fcba6af 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -76,7 +76,8 @@ class StatsWorker unless new_error_ids.empty? STDOUT.puts "send a delimiter task: job_id is #{job_id}" @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => new_error_ids.sample, - "job_id" => job_id}.to_json)) + "job_id" => job_id, + "lab" => LAB}.to_json)) end msg = %({"job_id": "#{job_id}", "job_state": "extract_finished"}) system "echo '#{msg}'" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] container/extract-stats prefix set lab in ENV
by Lu Weitao 29 Sep '20

29 Sep '20

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/extract-stats/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/extract-stats/start b/container/extract-stats/start index 25bdd25..6bb9f10 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -15,6 +15,7 @@ names = Set.new %w[ FLUENTD_PORT MAIL_HOST MAIL_PORT + LAB ] defaults = relevant_defaults(names) -- 2.23.0

2 5

[PATCH compass-ci 2/2] sparrow: run 3-code/os/debian
by Hu Xuejiao 29 Sep '20

29 Sep '20

Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/3-code/dev-env | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index ae267bd..80ee2cf 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -32,3 +32,6 @@ export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +source /etc/os-release + +. $(dirname ${BASH_SOURCE[0]})/os/${ID} -- 2.23.0

2 1

[PATCH compass-ci] sparrow: delete the same name install package
by Zhang Yu 29 Sep '20

29 Sep '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- sparrow/0-package/os/debian | 1 - 1 file changed, 1 deletion(-) diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 8301500..68423d7 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -13,7 +13,6 @@ pkgs=( docker.io ruby-full ruby-dev - bridge-utils qemu dmidecode lockfile-progs -- 2.23.0

2 2

[PATCH compass-ci 4/4] container: consider the situation that is no GITCACHE_HOST
by Hu Xuejiao 29 Sep '20

29 Sep '20

When manually ./build in VM, the GITCACHE_HOST variable is not found. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- container/crystal-shards/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/crystal-shards/Dockerfile b/container/crystal-shards/Dockerfile index 0c5713c..bdc5b66 100644 --- a/container/crystal-shards/Dockerfile +++ b/container/crystal-shards/Dockerfile @@ -7,7 +7,7 @@ MAINTAINER chief <taxcom(a)tom.com> ARG GITCACHE_HOST ARG GITCACHE_PORT -RUN [ -n "$GITCACHE_HOST" ] && echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig +RUN [ -z "$GITCACHE_HOST" ] || echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig copy shard.yml /usr/share/crystal/app/shard.yml -- 2.23.0

3 3

[PATCH compass-ci 1/4] sparrow: start the firewall in adapting openeuler
by Hu Xuejiao 29 Sep '20

29 Sep '20

When firewalld is inactive, return code of 'systemctl status firewalld' will be 3, and install-tiny has set -e, so this will cause this process exit. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/2-network/iptables | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..a9c6cf8 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -12,15 +12,19 @@ iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE -systemctl status firewalld | grep -q "running" && { +systemctl start firewalld - DOCKER0_IFACE=docker0 - DOCKER0_SUBNET=172.17.0.0/16 +[ "$(systemctl is-active firewalld)" == "active" ] || { + echo "firewalld start failed" + exit 0 +} - iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE - iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE +DOCKER0_IFACE=docker0 +DOCKER0_SUBNET=172.17.0.0/16 - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -} +iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE +iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE + +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -- 2.23.0

3 3

[PATCH compass-ci] lib/commit_status.rb: optimize jobs_list extract
by Lu Kaiyi 29 Sep '20

29 Sep '20

optimize jobs_list extract and reduce unnecessary reliance on params_group.rb Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/commit_status.rb | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/commit_status.rb b/lib/commit_status.rb index 19473f4..81aeb29 100644 --- a/lib/commit_status.rb +++ b/lib/commit_status.rb @@ -4,7 +4,6 @@ require_relative './constants' require_relative './es_query' -require_relative './params_group' def query_commit_status(job, error_id) items = { @@ -29,8 +28,8 @@ end def query_jobs_from_es(items) es = ESQuery.new(ES_HOST, ES_PORT) result = es.multi_field_query items - jobs = result['hits']['hits'] - jobs_list = extract_jobs_list(jobs) + jobs_list = result['hits']['hits'] + jobs_list.map! { |job| job['_source'] } return jobs_list end -- 2.23.0

1 0

[PATCH compass-ci 1/2] sparrow: add domain name resolution for Docker images
by Hu Xuejiao 29 Sep '20

29 Sep '20

When executing ./crystal-shards/build appear error: http://mirrors.huaweicloud.com/alpine/v3.11/main: temporary error(try again later) Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/3-code/os/debian | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 sparrow/3-code/os/debian diff --git a/sparrow/3-code/os/debian b/sparrow/3-code/os/debian new file mode 100644 index 0000000..171b979 --- /dev/null +++ b/sparrow/3-code/os/debian @@ -0,0 +1,4 @@ +#!/bin/bash +cat >> /etc/default/docker <<EOF +DOCKER_OPTS="--dns 114.114.114.114" +EOF -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: change tbox_group filter regex
by Zhang Yuhang 29 Sep '20

29 Sep '20

change to remove /-(\d+)$/ Exmaple: "taishan-xxx--xxx-12" remove "-12" Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 4544690..419ef86 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -41,16 +41,19 @@ def es_count(query) ES_CLIENT.count(index: 'jobs*', body: query)['count'] end -# "vm-hi1620-2p8g-212" remove "-212" -# "vm-hi1620-2p8g-zzz" remove "-zzz" -# "vm-git-bisect" don't remove "-bisect" +# vm-hi1620-2p4g-12 remove -12 +# taishan-xxx--xxx-12 remove -12 +# vm-hi1620-2p4g--0x remove --0x +# vm-hi1620-2p4g-abc remove -abc +# vm-git-bisect don't remove -bisect def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ - index = r.index('--') || r.rindex('-') - r = r[0, index] + remove_index = r =~ /-(\d+)$/ + if r[0, 3] == 'vm-' + remove_index = r =~ /(--.+$)|(-\d\w*-[a-zA-Z]+$)/ || remove_index end + r = r[0, remove_index] if remove_index result.add r end result.to_a @@ -95,7 +98,7 @@ def compare_candidates_body OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, - { os: 'debian', os_version: ['10', 'sid'] }, + { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], os_arch: %w[aarch64 x86_64], -- 2.23.0

2 2

[PATCH v2 compass-ci 1/2] src/extract-stats fix can not send delimiter task
by Lu Weitao 29 Sep '20

29 Sep '20

cause analysis: return 409 when add_task without "lab" in input parameters Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..fcba6af 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -76,7 +76,8 @@ class StatsWorker unless new_error_ids.empty? STDOUT.puts "send a delimiter task: job_id is #{job_id}" @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => new_error_ids.sample, - "job_id" => job_id}.to_json)) + "job_id" => job_id, + "lab" => LAB}.to_json)) end msg = %({"job_id": "#{job_id}", "job_state": "extract_finished"}) system "echo '#{msg}'" -- 2.23.0

2 2

[PATCH compass-ci] feat: helper: delete_task
by Tong Qunfeng 29 Sep '20

29 Sep '20

Usage: delete_task task_id[|task_ids file] case A (delete task_ids) - helper/redis_op_gc/consistency.rb -o : output manually delete task to man_del.id, and live too long task to too_long.id - then use delete_task man_del.id too_long.id : delete all this task case B (delete a signle task) - helper/1_showjob.sh task_id : show the job content in es. - delete_task task_id : delete this task Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- user-client/helper/redis_op_gc/delete_task.rb | 36 ++++++++++++++++ .../helper/redis_op_gc/force_delete.rb | 41 +++++++++++++++++++ 2 files changed, 77 insertions(+) create mode 100755 user-client/helper/redis_op_gc/delete_task.rb create mode 100644 user-client/helper/redis_op_gc/force_delete.rb diff --git a/user-client/helper/redis_op_gc/delete_task.rb b/user-client/helper/redis_op_gc/delete_task.rb new file mode 100755 index 0000000..4a67a2b --- /dev/null +++ b/user-client/helper/redis_op_gc/delete_task.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby + +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './force_delete' + +if ARGV.size < 1 + puts "Usage: #{__FILE__} task_id[|task_ids]" + puts " delete special task with [task_id]" + puts " or delete special task form a [task_ids] file" + exit +end + +ARGV.each do |id| + flag = FileTest::exists?(id) + if flag == true + task_ids = IO.readlines(id) + else + task_ids = [id] + end + + i = 0 + task_num = task_ids.size + while i < task_num + task_id = task_ids[i] + fd = ForceDelete.new(task_id) + fd.run + + i += 1 + set_progress(i, task_num) + end + + puts "" +end diff --git a/user-client/helper/redis_op_gc/force_delete.rb b/user-client/helper/redis_op_gc/force_delete.rb new file mode 100644 index 0000000..d8e973a --- /dev/null +++ b/user-client/helper/redis_op_gc/force_delete.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require_relative './basic_env' + +# force delete for job (with job_id) +# - taskqueue hash key: "queues/sched/*/reday?in_process?idle?uuid..." +# - taskqueue hash key: "queues/id2content" +# - scheduler hash key: "sched/id2job" +class ForceDelete + def initialize(id) + @task_id = id + end + + def run + fd4jobqueue + gc4taskqueue + gc4scheduler + end + + def fd4jobqueue + task_content = get_taskqueue_content4id(nil, @task_id) + if task_content + queue_name = task_content["queue"] + cmd = "#{CMD_BASE} queues/#{queue_name} , zrem #{@task_id}" + `#{cmd}`.chomp + end + end + + def gc4taskqueue + cmd = "#{CMD_BASE} queues/id2content , hdel #{@task_id}" + `#{cmd}`.chomp + end + + def gc4scheduler + cmd = "#{CMD_BASE} sched/id2job , hdel #{@task_id}" + `#{cmd}`.chomp + end +end -- 2.23.0

2 2

[PATCH v2 compass-ci] feat: helper: delete_task
by Tong Qunfeng 29 Sep '20

29 Sep '20

Usage: delete_task task_id[|task_ids file] case A (delete task_ids) - helper/redis_op_gc/consistency.rb -o : output manually delete task to man_del.id, and live too long task to too_long.id - then use delete_task man_del.id too_long.id : delete all this task case B (delete a signle task) - helper/1_showjob.sh task_id : show the job content in es. - delete_task task_id : delete this task Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- user-client/helper/redis_op_gc/delete_task.rb | 31 ++++++++++++++ .../helper/redis_op_gc/force_delete.rb | 41 +++++++++++++++++++ 2 files changed, 72 insertions(+) create mode 100755 user-client/helper/redis_op_gc/delete_task.rb create mode 100644 user-client/helper/redis_op_gc/force_delete.rb diff --git a/user-client/helper/redis_op_gc/delete_task.rb b/user-client/helper/redis_op_gc/delete_task.rb new file mode 100755 index 0000000..643d2b3 --- /dev/null +++ b/user-client/helper/redis_op_gc/delete_task.rb @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby + +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './force_delete' + +if ARGV.empty? + puts "Usage: #{__FILE__} task_id[|task_ids]" + puts ' delete special task with [task_id]' + puts ' or delete special task form a [task_ids] file' + exit +end + +task_ids = [] +ARGV.each do |id| + task_ids += IO.readlines(id) if File.exist?(id) + task_ids << [id] unless File.exist?(id) +end + +i = 0 +task_num = task_ids.size +while i < task_num + task_id = task_ids[i] + fd = ForceDelete.new(task_id) + fd.run + + i += 1 + set_progress(i, task_num) +end diff --git a/user-client/helper/redis_op_gc/force_delete.rb b/user-client/helper/redis_op_gc/force_delete.rb new file mode 100644 index 0000000..f5e52d2 --- /dev/null +++ b/user-client/helper/redis_op_gc/force_delete.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require_relative './basic_env' + +# force delete for job (with job_id) +# - taskqueue hash key: "queues/sched/*/reday?in_process?idle?uuid..." +# - taskqueue hash key: "queues/id2content" +# - scheduler hash key: "sched/id2job" +class ForceDelete + def initialize(id) + @task_id = id.chomp + end + + def run + fd4jobqueue + gc4taskqueue + gc4scheduler + end + + def fd4jobqueue + task_content = get_taskqueue_content4id(nil, @task_id) + return unless task_content + + queue_name = task_content['queue'] + cmd = "#{CMD_BASE} queues/#{queue_name} , zrem #{@task_id}" + `#{cmd}`.chomp + end + + def gc4taskqueue + cmd = "#{CMD_BASE} queues/id2content , hdel #{@task_id}" + `#{cmd}`.chomp + end + + def gc4scheduler + cmd = "#{CMD_BASE} sched/id2job , hdel #{@task_id}" + `#{cmd}`.chomp + end +end -- 2.23.0

1 0

[PATCH compass-ci 3/4] sparrow: add domain name resolution for Docker images
by Hu Xuejiao 28 Sep '20

28 Sep '20

When executing ./crystal-shards/build appear error: http://mirrors.huaweicloud.com/alpine/v3.11/main: temporary error(try aggin later) Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/3-code/dev-env | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 00d19c0..6bd7f21 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -31,3 +31,6 @@ export CCI_SRC=/c/compass-ci export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +cat >> /etc/default/docker <<EOF +DOCKER_OPTS="--dns 114.114.114.114" +EOF -- 2.23.0

2 1

[PATCH compass-ci] Change crystal-ci to compass-ci
by Zhang Yu 28 Sep '20

28 Sep '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/tutorial.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index d366580..d9f6158 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -1,4 +1,4 @@ -# Crystal-CI Introduction +# Compass-CI Introduction ## 简单说明 @@ -659,7 +659,7 @@ Compass-CI 集开发调测、测试服务、测试结果分析、辅助定位为 b) 将测试用例所需debian下的包名填入创建的文件中 echo "apache2" >> lkp-tests/distro/depends/test c) 生成lkp-aarch64.cgz - ./crystal-ci/container/lkp-initrd/run + ./compass-ci/container/lkp-initrd/run d) 提交测试用例 submit test.yaml e) 查看结果 -- 2.23.0

2 1

[PATCH compass-ci 2/4] sparrow: use useradd to add a lkp user instead of adduser
by Hu Xuejiao 28 Sep '20

28 Sep '20

Executing "adduser -u 1090 lkp" appear: New password: After usering "adduser ** --disabled-password **" appear: Full Name []: So use "adduser" to need too much user-information. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 1bb6861..05becdf 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -4,7 +4,7 @@ gem install git activesupport rest-client cucumber json faye-websocket elasticsearch -grep -q '^lkp:' /etc/passwd || adduser -u 1090 lkp +grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^team:' /etc/group || groupadd team grep -q '^committer:' /etc/group || groupadd -g 1999 committer -- 2.23.0

1 0

[PATCH v2 compass-ci] src/extract-stats prefix exception handing
by Lu Weitao 28 Sep '20

28 Sep '20

background: extract-stats process is at risk of be interrupted, due to other dependent services(task-queue, es_client) may not working properly solution: 1. use begin ... rescue to handle possible exceptions 2. skip and remove the task from task-queue if the job data does not have post-processing conditions 3. refactor for stats_worker.cr Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 60 ++++++++++++++++++------------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..edebf27 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -18,40 +18,50 @@ class StatsWorker def consume_sched_queue(queue_path : String) loop do begin - response = @tq.consume_task(queue_path) + job_id = get_job_id(queue_path).to_s + if job_id # consume the job by post-processing + job = @es.get_job_content(job_id) + result_root = job["result_root"]? + result_post_processing(job_id, result_root.to_s, queue_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + end rescue e STDERR.puts e.message - # incase of many error message when task-queue is busy + # incase of many error message when task-queue, ES does not work sleep(10) - next end + end + end - if response[0] == 200 - job_id = JSON.parse(response[1].to_json)["id"] - job = @es.get_job_content(job_id.to_s) - result_root = job["result_root"] - begin - # extract stats.json - system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" - # storage stats to job in es - store_stats_es(result_root.to_s, job_id.to_s) if result_root - # send mail to submitter for job results - system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" - rescue e - STDERR.puts e.message - next - end - - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - else - sleep(60) - end + # get job_id from task-queue + def get_job_id(queue_path : String) + response = @tq.consume_task(queue_path) + if response[0] == 200 + JSON.parse(response[1].to_json)["id"] + else + # will sleep 60s if no task in task-queue + sleep(60) + nil end end - def store_stats_es(result_root : String, job_id : String) + def result_post_processing(job_id : String, result_root : String, queue_path : String) + return nil unless result_root + + # extract stats.json + system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" + # storage stats to job in es + store_stats_es(result_root, job_id, queue_path) + # send mail to submitter for job results + system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" + end + + def store_stats_es(result_root : String, job_id : String, queue_path : String) stats_path = "#{result_root}/stats.json" - raise "#{stats_path} file not exists." unless File.exists?(stats_path) + unless File.exists?(stats_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + raise "#{stats_path} file not exists." + end stats = File.open(stats_path) do |file| JSON.parse(file) -- 2.23.0

1 0

[PATCH compass-ci] feat: scheduler: add optional fields into job
by Ren Wen 28 Sep '20

28 Sep '20

Only `multi-qemu` job needs `FLUENTD_SERVER_*`, so move them to the place where calls this function. When calls this function, an array can be passed, like: ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}",] these elements will add into job as job fields. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/jobfile_operate.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index df5af1b..0533346 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -200,14 +200,13 @@ module Jobfile::Operate return target_path end - def self.auto_submit_job(job_file, override_parameter) + def self.auto_submit_job(job_file, override_parameter, other_parameters = nil) job_fields = ["SCHED_HOST=#{ENV["SCHED_HOST"]}", "SCHED_PORT=#{ENV["SCHED_PORT"]}", - "FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", - "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}", "GIT_SERVER=#{ENV["GIT_SERVER"]}", "lab=#{ENV["lab"]}"] + job_fields.concat(other_parameters) if other_parameters cmd = "#{ENV["LKP_SRC"]}/sbin/submit " cmd += job_fields.join(" ") cmd += " -s '#{override_parameter}' #{job_file}" -- 2.23.0

1 0

[PATCH compass-ci] sparrow: container: fix git-mirror start error
by Liu Yinsi 28 Sep '20

28 Sep '20

1. git clone upstream-repo and export REPO_SRC to fix error: No such file or directory @ rb_sysopen - /c/upstream-repos 2. chown lkp:committer /srv/git to fix error: Permission denied @ dir_s_mkdir - /srv/git/b Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 1 + sparrow/3-code/dev-env | 1 + sparrow/3-code/git | 4 ++++ 3 files changed, 6 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 35456a8..f5c1646 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -6,6 +6,7 @@ chown -R lkp:lkp /srv/es chown lkp:lkp /srv/cache/netdata_cache chown lkp:lkp /srv/cache/netdata_lib chown lkp:lkp /srv/cci/serial/fluentd-pos +chown lkp:committer /srv/git chgrp team /srv/cci/serial/logs chmod 775 /srv/result chmod 775 /srv/es/logging-es diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 00d19c0..ae267bd 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -27,6 +27,7 @@ EOF cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci +export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF diff --git a/sparrow/3-code/git b/sparrow/3-code/git index b1d2c03..5588696 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -16,3 +16,7 @@ cd /c || exit [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git } + +[ -d "upstream-repos/.git" ] || { + git clone https://gitee.com/wu_fengguang/upstream-repos.git +} -- 2.23.0

1 0

[PATCH compass-ci] Makefile: add ctags
by Wu Fengguang 28 Sep '20

28 Sep '20

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- Makefile | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 Makefile diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..c7505e8 --- /dev/null +++ b/Makefile @@ -0,0 +1,2 @@ +ctags: + ctags -R --links=no --langmap=ruby:.rb.cr -- 2.23.0

1 1

[PATCH compass-ci] fix: system rebooting when executed 2-network/br0
by Xu Xijian 28 Sep '20

28 Sep '20

When 2-network/br0 executed, program will hang up fo a while and then go rebooting. The cause is iptables need time to take effect, echo 1 to /proc/sys/net/ipv4/ip_forward to make it taking effect immediately. Logs(openeuler 20.03): [ 161.565451] + 2-network/iptables [ 161.565453] [ 161.568088] ++ ip route get 1.2.3.4 [ 161.568090] ... [ 168.107379] LKP: rebooting Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- sparrow/2-network/iptables | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..0bf1235 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -11,6 +11,8 @@ BR0_SUBNET=172.18.0.0/16 iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE +# To make upon rules taking effect immediately +echo 1 > /proc/sys/net/ipv4/ip_forward systemctl status firewalld | grep -q "running" && { -- 2.23.0

2 1