- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] src/lib/web_backend.rb: change tbox_group filter regex
by Zhang Yuhang 29 Sep '20

29 Sep '20

change to remove /-(\d+)$/ Exmaple: "taishan-xxx--xxx-12" remove "-12" Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 4544690..419ef86 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -41,16 +41,19 @@ def es_count(query) ES_CLIENT.count(index: 'jobs*', body: query)['count'] end -# "vm-hi1620-2p8g-212" remove "-212" -# "vm-hi1620-2p8g-zzz" remove "-zzz" -# "vm-git-bisect" don't remove "-bisect" +# vm-hi1620-2p4g-12 remove -12 +# taishan-xxx--xxx-12 remove -12 +# vm-hi1620-2p4g--0x remove --0x +# vm-hi1620-2p4g-abc remove -abc +# vm-git-bisect don't remove -bisect def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ - index = r.index('--') || r.rindex('-') - r = r[0, index] + remove_index = r =~ /-(\d+)$/ + if r[0, 3] == 'vm-' + remove_index = r =~ /(--.+$)|(-\d\w*-[a-zA-Z]+$)/ || remove_index end + r = r[0, remove_index] if remove_index result.add r end result.to_a @@ -95,7 +98,7 @@ def compare_candidates_body OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, - { os: 'debian', os_version: ['10', 'sid'] }, + { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], os_arch: %w[aarch64 x86_64], -- 2.23.0

2 2

[PATCH v2 compass-ci 1/2] src/extract-stats fix can not send delimiter task
by Lu Weitao 29 Sep '20

29 Sep '20

cause analysis: return 409 when add_task without "lab" in input parameters Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..fcba6af 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -76,7 +76,8 @@ class StatsWorker unless new_error_ids.empty? STDOUT.puts "send a delimiter task: job_id is #{job_id}" @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => new_error_ids.sample, - "job_id" => job_id}.to_json)) + "job_id" => job_id, + "lab" => LAB}.to_json)) end msg = %({"job_id": "#{job_id}", "job_state": "extract_finished"}) system "echo '#{msg}'" -- 2.23.0

2 2

[PATCH compass-ci] feat: helper: delete_task
by Tong Qunfeng 29 Sep '20

29 Sep '20

Usage: delete_task task_id[|task_ids file] case A (delete task_ids) - helper/redis_op_gc/consistency.rb -o : output manually delete task to man_del.id, and live too long task to too_long.id - then use delete_task man_del.id too_long.id : delete all this task case B (delete a signle task) - helper/1_showjob.sh task_id : show the job content in es. - delete_task task_id : delete this task Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- user-client/helper/redis_op_gc/delete_task.rb | 36 ++++++++++++++++ .../helper/redis_op_gc/force_delete.rb | 41 +++++++++++++++++++ 2 files changed, 77 insertions(+) create mode 100755 user-client/helper/redis_op_gc/delete_task.rb create mode 100644 user-client/helper/redis_op_gc/force_delete.rb diff --git a/user-client/helper/redis_op_gc/delete_task.rb b/user-client/helper/redis_op_gc/delete_task.rb new file mode 100755 index 0000000..4a67a2b --- /dev/null +++ b/user-client/helper/redis_op_gc/delete_task.rb @@ -0,0 +1,36 @@ +#!/usr/bin/env ruby + +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './force_delete' + +if ARGV.size < 1 + puts "Usage: #{__FILE__} task_id[|task_ids]" + puts " delete special task with [task_id]" + puts " or delete special task form a [task_ids] file" + exit +end + +ARGV.each do |id| + flag = FileTest::exists?(id) + if flag == true + task_ids = IO.readlines(id) + else + task_ids = [id] + end + + i = 0 + task_num = task_ids.size + while i < task_num + task_id = task_ids[i] + fd = ForceDelete.new(task_id) + fd.run + + i += 1 + set_progress(i, task_num) + end + + puts "" +end diff --git a/user-client/helper/redis_op_gc/force_delete.rb b/user-client/helper/redis_op_gc/force_delete.rb new file mode 100644 index 0000000..d8e973a --- /dev/null +++ b/user-client/helper/redis_op_gc/force_delete.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require_relative './basic_env' + +# force delete for job (with job_id) +# - taskqueue hash key: "queues/sched/*/reday?in_process?idle?uuid..." +# - taskqueue hash key: "queues/id2content" +# - scheduler hash key: "sched/id2job" +class ForceDelete + def initialize(id) + @task_id = id + end + + def run + fd4jobqueue + gc4taskqueue + gc4scheduler + end + + def fd4jobqueue + task_content = get_taskqueue_content4id(nil, @task_id) + if task_content + queue_name = task_content["queue"] + cmd = "#{CMD_BASE} queues/#{queue_name} , zrem #{@task_id}" + `#{cmd}`.chomp + end + end + + def gc4taskqueue + cmd = "#{CMD_BASE} queues/id2content , hdel #{@task_id}" + `#{cmd}`.chomp + end + + def gc4scheduler + cmd = "#{CMD_BASE} sched/id2job , hdel #{@task_id}" + `#{cmd}`.chomp + end +end -- 2.23.0

2 2

[PATCH v2 compass-ci] feat: helper: delete_task
by Tong Qunfeng 29 Sep '20

29 Sep '20

Usage: delete_task task_id[|task_ids file] case A (delete task_ids) - helper/redis_op_gc/consistency.rb -o : output manually delete task to man_del.id, and live too long task to too_long.id - then use delete_task man_del.id too_long.id : delete all this task case B (delete a signle task) - helper/1_showjob.sh task_id : show the job content in es. - delete_task task_id : delete this task Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- user-client/helper/redis_op_gc/delete_task.rb | 31 ++++++++++++++ .../helper/redis_op_gc/force_delete.rb | 41 +++++++++++++++++++ 2 files changed, 72 insertions(+) create mode 100755 user-client/helper/redis_op_gc/delete_task.rb create mode 100644 user-client/helper/redis_op_gc/force_delete.rb diff --git a/user-client/helper/redis_op_gc/delete_task.rb b/user-client/helper/redis_op_gc/delete_task.rb new file mode 100755 index 0000000..643d2b3 --- /dev/null +++ b/user-client/helper/redis_op_gc/delete_task.rb @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby + +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative './force_delete' + +if ARGV.empty? + puts "Usage: #{__FILE__} task_id[|task_ids]" + puts ' delete special task with [task_id]' + puts ' or delete special task form a [task_ids] file' + exit +end + +task_ids = [] +ARGV.each do |id| + task_ids += IO.readlines(id) if File.exist?(id) + task_ids << [id] unless File.exist?(id) +end + +i = 0 +task_num = task_ids.size +while i < task_num + task_id = task_ids[i] + fd = ForceDelete.new(task_id) + fd.run + + i += 1 + set_progress(i, task_num) +end diff --git a/user-client/helper/redis_op_gc/force_delete.rb b/user-client/helper/redis_op_gc/force_delete.rb new file mode 100644 index 0000000..f5e52d2 --- /dev/null +++ b/user-client/helper/redis_op_gc/force_delete.rb @@ -0,0 +1,41 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require_relative './basic_env' + +# force delete for job (with job_id) +# - taskqueue hash key: "queues/sched/*/reday?in_process?idle?uuid..." +# - taskqueue hash key: "queues/id2content" +# - scheduler hash key: "sched/id2job" +class ForceDelete + def initialize(id) + @task_id = id.chomp + end + + def run + fd4jobqueue + gc4taskqueue + gc4scheduler + end + + def fd4jobqueue + task_content = get_taskqueue_content4id(nil, @task_id) + return unless task_content + + queue_name = task_content['queue'] + cmd = "#{CMD_BASE} queues/#{queue_name} , zrem #{@task_id}" + `#{cmd}`.chomp + end + + def gc4taskqueue + cmd = "#{CMD_BASE} queues/id2content , hdel #{@task_id}" + `#{cmd}`.chomp + end + + def gc4scheduler + cmd = "#{CMD_BASE} sched/id2job , hdel #{@task_id}" + `#{cmd}`.chomp + end +end -- 2.23.0

1 0

[PATCH compass-ci 3/4] sparrow: add domain name resolution for Docker images
by Hu Xuejiao 28 Sep '20

28 Sep '20

When executing ./crystal-shards/build appear error: http://mirrors.huaweicloud.com/alpine/v3.11/main: temporary error(try aggin later) Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/3-code/dev-env | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 00d19c0..6bd7f21 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -31,3 +31,6 @@ export CCI_SRC=/c/compass-ci export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF +cat >> /etc/default/docker <<EOF +DOCKER_OPTS="--dns 114.114.114.114" +EOF -- 2.23.0

2 1

[PATCH compass-ci] Change crystal-ci to compass-ci
by Zhang Yu 28 Sep '20

28 Sep '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/tutorial.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/tutorial.md b/doc/tutorial.md index d366580..d9f6158 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -1,4 +1,4 @@ -# Crystal-CI Introduction +# Compass-CI Introduction ## 简单说明 @@ -659,7 +659,7 @@ Compass-CI 集开发调测、测试服务、测试结果分析、辅助定位为 b) 将测试用例所需debian下的包名填入创建的文件中 echo "apache2" >> lkp-tests/distro/depends/test c) 生成lkp-aarch64.cgz - ./crystal-ci/container/lkp-initrd/run + ./compass-ci/container/lkp-initrd/run d) 提交测试用例 submit test.yaml e) 查看结果 -- 2.23.0

2 1

[PATCH compass-ci 2/4] sparrow: use useradd to add a lkp user instead of adduser
by Hu Xuejiao 28 Sep '20

28 Sep '20

Executing "adduser -u 1090 lkp" appear: New password: After usering "adduser ** --disabled-password **" appear: Full Name []: So use "adduser" to need too much user-information. Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 1bb6861..05becdf 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -4,7 +4,7 @@ gem install git activesupport rest-client cucumber json faye-websocket elasticsearch -grep -q '^lkp:' /etc/passwd || adduser -u 1090 lkp +grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^team:' /etc/group || groupadd team grep -q '^committer:' /etc/group || groupadd -g 1999 committer -- 2.23.0

1 0

[PATCH v2 compass-ci] src/extract-stats prefix exception handing
by Lu Weitao 28 Sep '20

28 Sep '20

background: extract-stats process is at risk of be interrupted, due to other dependent services(task-queue, es_client) may not working properly solution: 1. use begin ... rescue to handle possible exceptions 2. skip and remove the task from task-queue if the job data does not have post-processing conditions 3. refactor for stats_worker.cr Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 60 ++++++++++++++++++------------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 3cff51d..edebf27 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -18,40 +18,50 @@ class StatsWorker def consume_sched_queue(queue_path : String) loop do begin - response = @tq.consume_task(queue_path) + job_id = get_job_id(queue_path).to_s + if job_id # consume the job by post-processing + job = @es.get_job_content(job_id) + result_root = job["result_root"]? + result_post_processing(job_id, result_root.to_s, queue_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + end rescue e STDERR.puts e.message - # incase of many error message when task-queue is busy + # incase of many error message when task-queue, ES does not work sleep(10) - next end + end + end - if response[0] == 200 - job_id = JSON.parse(response[1].to_json)["id"] - job = @es.get_job_content(job_id.to_s) - result_root = job["result_root"] - begin - # extract stats.json - system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" - # storage stats to job in es - store_stats_es(result_root.to_s, job_id.to_s) if result_root - # send mail to submitter for job results - system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" - rescue e - STDERR.puts e.message - next - end - - @tq.delete_task(queue_path + "/in_process", "#{job_id}") - else - sleep(60) - end + # get job_id from task-queue + def get_job_id(queue_path : String) + response = @tq.consume_task(queue_path) + if response[0] == 200 + JSON.parse(response[1].to_json)["id"] + else + # will sleep 60s if no task in task-queue + sleep(60) + nil end end - def store_stats_es(result_root : String, job_id : String) + def result_post_processing(job_id : String, result_root : String, queue_path : String) + return nil unless result_root + + # extract stats.json + system "#{ENV["CCI_SRC"]}/sbin/result2stats #{result_root}" + # storage stats to job in es + store_stats_es(result_root, job_id, queue_path) + # send mail to submitter for job results + system "#{ENV["CCI_SRC"]}/sbin/mail-job #{job_id}" + end + + def store_stats_es(result_root : String, job_id : String, queue_path : String) stats_path = "#{result_root}/stats.json" - raise "#{stats_path} file not exists." unless File.exists?(stats_path) + unless File.exists?(stats_path) + @tq.delete_task(queue_path + "/in_process", "#{job_id}") + raise "#{stats_path} file not exists." + end stats = File.open(stats_path) do |file| JSON.parse(file) -- 2.23.0

1 0

[PATCH compass-ci] feat: scheduler: add optional fields into job
by Ren Wen 28 Sep '20

28 Sep '20

Only `multi-qemu` job needs `FLUENTD_SERVER_*`, so move them to the place where calls this function. When calls this function, an array can be passed, like: ["FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}",] these elements will add into job as job fields. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/jobfile_operate.cr | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/src/scheduler/jobfile_operate.cr b/src/scheduler/jobfile_operate.cr index df5af1b..0533346 100644 --- a/src/scheduler/jobfile_operate.cr +++ b/src/scheduler/jobfile_operate.cr @@ -200,14 +200,13 @@ module Jobfile::Operate return target_path end - def self.auto_submit_job(job_file, override_parameter) + def self.auto_submit_job(job_file, override_parameter, other_parameters = nil) job_fields = ["SCHED_HOST=#{ENV["SCHED_HOST"]}", "SCHED_PORT=#{ENV["SCHED_PORT"]}", - "FLUENTD_SERVER_HOST=#{ENV["FLUENTD_SERVER_HOST"]}", - "FLUENTD_SERVER_PORT=#{ENV["FLUENTD_SERVER_PORT"]}", "GIT_SERVER=#{ENV["GIT_SERVER"]}", "lab=#{ENV["lab"]}"] + job_fields.concat(other_parameters) if other_parameters cmd = "#{ENV["LKP_SRC"]}/sbin/submit " cmd += job_fields.join(" ") cmd += " -s '#{override_parameter}' #{job_file}" -- 2.23.0

1 0

[PATCH compass-ci] sparrow: container: fix git-mirror start error
by Liu Yinsi 28 Sep '20

28 Sep '20

1. git clone upstream-repo and export REPO_SRC to fix error: No such file or directory @ rb_sysopen - /c/upstream-repos 2. chown lkp:committer /srv/git to fix error: Permission denied @ dir_s_mkdir - /srv/git/b Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 1 + sparrow/3-code/dev-env | 1 + sparrow/3-code/git | 4 ++++ 3 files changed, 6 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 35456a8..f5c1646 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -6,6 +6,7 @@ chown -R lkp:lkp /srv/es chown lkp:lkp /srv/cache/netdata_cache chown lkp:lkp /srv/cache/netdata_lib chown lkp:lkp /srv/cci/serial/fluentd-pos +chown lkp:committer /srv/git chgrp team /srv/cci/serial/logs chmod 775 /srv/result chmod 775 /srv/es/logging-es diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 00d19c0..ae267bd 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -27,6 +27,7 @@ EOF cat > /etc/profile.d/compass.sh <<'EOF' export LKP_SRC=/c/lkp-tests export CCI_SRC=/c/compass-ci +export REPO_SRC=/c/upstream-repos export PATH="$PATH:$CCI_SRC/sbin:$LKP_SRC/sbin:$LKP_SRC/bin" EOF diff --git a/sparrow/3-code/git b/sparrow/3-code/git index b1d2c03..5588696 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -16,3 +16,7 @@ cd /c || exit [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git } + +[ -d "upstream-repos/.git" ] || { + git clone https://gitee.com/wu_fengguang/upstream-repos.git +} -- 2.23.0

1 0