September 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 4/4] container: consider the situation that is no GITCACHE_HOST
by Hu Xuejiao 29 Sep '20

29 Sep '20

When manually ./build in VM, the GITCACHE_HOST variable is not found. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- container/crystal-shards/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/crystal-shards/Dockerfile b/container/crystal-shards/Dockerfile index 0c5713c..bdc5b66 100644 --- a/container/crystal-shards/Dockerfile +++ b/container/crystal-shards/Dockerfile @@ -7,7 +7,7 @@ MAINTAINER chief <taxcom(a)tom.com> ARG GITCACHE_HOST ARG GITCACHE_PORT -RUN [ -n "$GITCACHE_HOST" ] && echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig +RUN [ -z "$GITCACHE_HOST" ] || echo -e "[url \"http://$GITCACHE_HOST:${GITCACHE_PORT:-5000}/\"]\n\tinsteadOf = https://" >> /etc/gitconfig copy shard.yml /usr/share/crystal/app/shard.yml -- 2.23.0

3 3

[PATCH compass-ci 1/4] sparrow: start the firewall in adapting openeuler
by Hu Xuejiao 29 Sep '20

29 Sep '20

When firewalld is inactive, return code of 'systemctl status firewalld' will be 3, and install-tiny has set -e, so this will cause this process exit. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/2-network/iptables | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/sparrow/2-network/iptables b/sparrow/2-network/iptables index 092c4ca..a9c6cf8 100755 --- a/sparrow/2-network/iptables +++ b/sparrow/2-network/iptables @@ -12,15 +12,19 @@ iptables -I FORWARD 1 -j ACCEPT iptables -t nat -A POSTROUTING -o "$PUB_IFACE" -s $BR0_SUBNET -j MASQUERADE iptables -t nat -A POSTROUTING -o $BR0_IFACE -d $BR0_SUBNET -j MASQUERADE -systemctl status firewalld | grep -q "running" && { +systemctl start firewalld - DOCKER0_IFACE=docker0 - DOCKER0_SUBNET=172.17.0.0/16 +[ "$(systemctl is-active firewalld)" == "active" ] || { + echo "firewalld start failed" + exit 0 +} - iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE - iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE +DOCKER0_IFACE=docker0 +DOCKER0_SUBNET=172.17.0.0/16 - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" - firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -} +iptables -t nat -A POSTROUTING -o $PUB_IFACE -s $DOCKER0_SUBNET -j MASQUERADE +iptables -t nat -A POSTROUTING -o $DOCKER0_IFACE -d $DOCKER0_SUBNET -j MASQUERADE + +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$DOCKER0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=$BR0_SUBNET accept" +firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address=0.0.0.0/32 accept" -- 2.23.0

3 3

[PATCH compass-ci] lib/commit_status.rb: optimize jobs_list extract
by Lu Kaiyi 29 Sep '20

29 Sep '20

optimize jobs_list extract and reduce unnecessary reliance on params_group.rb Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/commit_status.rb | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/lib/commit_status.rb b/lib/commit_status.rb index 19473f4..81aeb29 100644 --- a/lib/commit_status.rb +++ b/lib/commit_status.rb @@ -4,7 +4,6 @@ require_relative './constants' require_relative './es_query' -require_relative './params_group' def query_commit_status(job, error_id) items = { @@ -29,8 +28,8 @@ end def query_jobs_from_es(items) es = ESQuery.new(ES_HOST, ES_PORT) result = es.multi_field_query items - jobs = result['hits']['hits'] - jobs_list = extract_jobs_list(jobs) + jobs_list = result['hits']['hits'] + jobs_list.map! { |job| job['_source'] } return jobs_list end -- 2.23.0

1 0

[PATCH compass-ci 1/2] sparrow: add domain name resolution for Docker images
by Hu Xuejiao 29 Sep '20

29 Sep '20

When executing ./crystal-shards/build appear error: http://mirrors.huaweicloud.com/alpine/v3.11/main: temporary error(try again later) Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/3-code/os/debian | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 sparrow/3-code/os/debian diff --git a/sparrow/3-code/os/debian b/sparrow/3-code/os/debian new file mode 100644 index 0000000..171b979 --- /dev/null +++ b/sparrow/3-code/os/debian @@ -0,0 +1,4 @@ +#!/bin/bash +cat >> /etc/default/docker <<EOF +DOCKER_OPTS="--dns 114.114.114.114" +EOF -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: change tbox_group filter regex
by Zhang Yuhang 29 Sep '20

29 Sep '20

change to remove /-(\d+)$/ Exmaple: "taishan-xxx--xxx-12" remove "-12" Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 4544690..419ef86 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -41,16 +41,19 @@ def es_count(query) ES_CLIENT.count(index: 'jobs*', body: query)['count'] end -# "vm-hi1620-2p8g-212" remove "-212" -# "vm-hi1620-2p8g-zzz" remove "-zzz" -# "vm-git-bisect" don't remove "-bisect" +# vm-hi1620-2p4g-12 remove -12 +# taishan-xxx--xxx-12 remove -12 +# vm-hi1620-2p4g--0x remove --0x +# vm-hi1620-2p4g-abc remove -abc +# vm-git-bisect don't remove -bisect def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ - index = r.index('--') || r.rindex('-') - r = r[0, index] + remove_index = r =~ /-(\d+)$/ + if r[0, 3] == 'vm-' + remove_index = r =~ /(--.+$)|(-\d\w*-[a-zA-Z]+$)/ || remove_index end + r = r[0, remove_index] if remove_index result.add r end result.to_a @@ -95,7 +98,7 @@ def compare_candidates_body OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, - { os: 'debian', os_version: ['10', 'sid'] }, + { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], os_arch: %w[aarch64 x86_64], -- 2.23.0

2 2

[PATCH v2 compass-ci 1/2] src/extract-stats fix can not send delimiter task
by Lu Weitao 29 Sep '20

29 Sep '20

cause analysis: return 409 when add_task without "lab" in input parameters Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..fcba6af 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -76,7 +76,8 @@ class StatsWorker unless new_error_ids.empty? STDOUT.puts "send a delimiter task: job_id is #{job_id}" @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => new_error_ids.sample, - "job_id" => job_id}.to_json)) + "job_id" => job_id, + "lab" => LAB}.to_json)) end msg = %({"job_id": "#{job_id}", "job_state": "extract_finished"}) system "echo '#{msg}'" -- 2.23.0

2 2

[PATCH compass-ci] feat: helper: delete_task
by Tong Qunfeng 29 Sep '20

29 Sep '20

Usage: delete_task task_id[|task_ids file] case A (delete task_ids) - helper/redis_op_gc/consistency.rb -o : output manually delete task to man_del.id, and live too long task to too_long.id - then use delete_task man_del.id too_long.id : delete all this task case B (delete a signle task) - helper/1_showjob.sh task_id : show the job content in es. - delete_task task_id : delete this task Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- user-client/helper/redis_op_gc/delete_task.rb | 36 ++++++++++++++++ .../helper/redis_op_gc/force_delete.rb | 41 +++++++++++++++++++ 2 files changed, 77 insertions(+) create mode 100755 user-client/helper/redis_op_gc/delete_task.rb create mode 100644 user-client/helper/redis_op_gc/force_delete.rb diff --git a/user-client/helper/redis_op_gc/delete_task.rb b/user-client/helper/redis_op_gc/delete_task.rb new file mode 100755 index 0000000..4a67a2b --- /dev/null +++ b/user-client/helper/redis_op_gc/delete_task.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby + +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './force_delete' + +if ARGV.size < 1 + puts "Usage: #{__FILE__} task_id[|task_ids]" + puts " delete special task with [task_id]" + puts " or delete special task form a [task_ids] file" + exit +end + +ARGV.each do |id| + flag = FileTest::exists?(id) + if flag == true + task_ids = IO.readlines(id) + else + task_ids = [id] + end + + i = 0 + task_num = task_ids.size + while i < task_num + task_id = task_ids[i] + fd = ForceDelete.new(task_id) + fd.run + + i += 1 + set_progress(i, task_num) + end + + puts "" +end diff --git a/user-client/helper/redis_op_gc/force_delete.rb b/user-client/helper/redis_op_gc/force_delete.rb new file mode 100644 index 0000000..d8e973a --- /dev/null +++ b/user-client/helper/redis_op_gc/force_delete.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require_relative './basic_env' + +# force delete for job (with job_id) +# - taskqueue hash key: "queues/sched/*/reday?in_process?idle?uuid..." +# - taskqueue hash key: "queues/id2content" +# - scheduler hash key: "sched/id2job" +class ForceDelete + def initialize(id) + @task_id = id + end + + def run + fd4jobqueue + gc4taskqueue + gc4scheduler + end + + def fd4jobqueue + task_content = get_taskqueue_content4id(nil, @task_id) + if task_content + queue_name = task_content["queue"] + cmd = "#{CMD_BASE} queues/#{queue_name} , zrem #{@task_id}" + `#{cmd}`.chomp + end + end + + def gc4taskqueue + cmd = "#{CMD_BASE} queues/id2content , hdel #{@task_id}" + `#{cmd}`.chomp + end + + def gc4scheduler + cmd = "#{CMD_BASE} sched/id2job , hdel #{@task_id}" + `#{cmd}`.chomp + end +end -- 2.23.0

2 2

[PATCH v2 compass-ci] feat: helper: delete_task
by Tong Qunfeng 29 Sep '20

29 Sep '20

Usage: delete_task task_id[|task_ids file] case A (delete task_ids) - helper/redis_op_gc/consistency.rb -o : output manually delete task to man_del.id, and live too long task to too_long.id - then use delete_task man_del.id too_long.id : delete all this task case B (delete a signle task) - helper/1_showjob.sh task_id : show the job content in es. - delete_task task_id : delete this task Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- user-client/helper/redis_op_gc/delete_task.rb | 31 ++++++++++++++ .../helper/redis_op_gc/force_delete.rb | 41 +++++++++++++++++++ 2 files changed, 72 insertions(+) create mode 100755 user-client/helper/redis_op_gc/delete_task.rb create mode 100644 user-client/helper/redis_op_gc/force_delete.rb diff --git a/user-client/helper/redis_op_gc/delete_task.rb b/user-client/helper/redis_op_gc/delete_task.rb new file mode 100755 index 0000000..643d2b3 --- /dev/null +++ b/user-client/helper/redis_op_gc/delete_task.rb @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby + +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './force_delete' + +if ARGV.empty? + puts "Usage: #{__FILE__} task_id[|task_ids]" + puts ' delete special task with [task_id]' + puts ' or delete special task form a [task_ids] file' + exit +end + +task_ids = [] +ARGV.each do |id| + task_ids += IO.readlines(id) if File.exist?(id) + task_ids << [id] unless File.exist?(id) +end + +i = 0 +task_num = task_ids.size +while i < task_num + task_id = task_ids[i] + fd = ForceDelete.new(task_id) + fd.run + + i += 1 + set_progress(i, task_num) +end diff --git a/user-client/helper/redis_op_gc/force_delete.rb b/user-client/helper/redis_op_gc/force_delete.rb new file mode 100644 index 0000000..f5e52d2 --- /dev/null +++ b/user-client/helper/redis_op_gc/force_delete.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require_relative './basic_env' + +# force delete for job (with job_id) +# - taskqueue hash key: "queues/sched/*/reday?in_process?idle?uuid..." +# - taskqueue hash key: "queues/id2content" +# - scheduler hash key: "sched/id2job" +class ForceDelete + def initialize(id) + @task_id = id.chomp + end + + def run + fd4jobqueue + gc4taskqueue + gc4scheduler + end + + def fd4jobqueue + task_content = get_taskqueue_content4id(nil, @task_id) + return unless task_content + + queue_name = task_content['queue'] + cmd = "#{CMD_BASE} queues/#{queue_name} , zrem #{@task_id}" + `#{cmd}`.chomp + end + + def gc4taskqueue + cmd = "#{CMD_BASE} queues/id2content , hdel #{@task_id}" + `#{cmd}`.chomp + end + + def gc4scheduler + cmd = "#{CMD_BASE} sched/id2job , hdel #{@task_id}" + `#{cmd}`.chomp + end +end -- 2.23.0

1 0

[PATCH compass-ci 3/4] sparrow: add domain name resolution for Docker images
by Hu Xuejiao 28 Sep '20

28 Sep '20

When executing ./crystal-shards/build appear error: http://mirrors.huaweicloud.com/alpine/v3.11/main: temporary error(try aggin later) Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/3-code/dev-env | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 00d19c0..6bd7f21 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -31,3 +31,6 @@ export CCI_SRC=/c/compass-ci export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +cat >> /etc/default/docker <<EOF +DOCKER_OPTS="--dns 114.114.114.114" +EOF -- 2.23.0

2 1

[PATCH compass-ci] Change crystal-ci to compass-ci
by Zhang Yu 28 Sep '20

28 Sep '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/tutorial.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index d366580..d9f6158 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -1,4 +1,4 @@ -# Crystal-CI Introduction +# Compass-CI Introduction ## 简单说明 @@ -659,7 +659,7 @@ Compass-CI 集开发调测、测试服务、测试结果分析、辅助定位为 b) 将测试用例所需debian下的包名填入创建的文件中 echo "apache2" >> lkp-tests/distro/depends/test c) 生成lkp-aarch64.cgz - ./crystal-ci/container/lkp-initrd/run + ./compass-ci/container/lkp-initrd/run d) 提交测试用例 submit test.yaml e) 查看结果 -- 2.23.0

2 1