- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci 3/3] container/mail-robot: optimize parsing email
by Luan Shengde 24 Mar '21

24 Mar '21

optimize code: use gem resource ruby-git instead of system command Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index e6c35f7..ce01fec 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-… [View More]

1 0

[PATCH v2 compass-ci 2/3] container/mail-robot: add use ENV variable
by Luan Shengde 24 Mar '21

24 Mar '21

use ENV variable instead of full-path string enable modify the path without re-edit the code. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 2 +- container/mail-robot/start | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 17705ec..97c3989 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-… [View More]

1 0

[PATCH v2 compass-ci 1/3] container/mail-robot: add install ruby-git
by Luan Shengde 24 Mar '21

24 Mar '21

add install ruby-git in dockerfile Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile index e4b3e7c..41056a5 100644 --- a/container/mail-robot/Dockerfile +++ b/container/mail-robot/Dockerfile @@ -10,7 +10,7 @@ ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update &&… [View More]

1 0

[PATCH compass-ci] src/monitor: detect crash and send msg to mq queue.
by Li Ping 23 Mar '21

23 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/monitoring/filter.cr | 2 +- src/monitoring/parse_serial_logs.cr | 50 ++++++++++++++++++++++++++++- 2 files changed, 50 insertions(+), 2 deletions(-) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 39cfbb6..b663f11 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -56,7 +56,7 @@ class Filter msg = JSON.parse(msg.to_s).as_h? return unless msg - @sp.… [View More]save_dmesg_to_result_root(msg) + @sp.deal_serial_log(msg) @hash.keys.each do |query| if match_query(query.as_h, msg) send_msg(query, msg) diff --git a/src/monitoring/parse_serial_logs.cr b/src/monitoring/parse_serial_logs.cr index 0589523..9430d8c 100644 --- a/src/monitoring/parse_serial_logs.cr +++ b/src/monitoring/parse_serial_logs.cr @@ -4,6 +4,7 @@ require "../scheduler/elasticsearch_client" require "set" require "json" +require "../lib/mq" # This parses dmesg in a stream of serial log, finding a number of patterns # in various places of the dmesg and take actions accordingly. @@ -40,9 +41,15 @@ class SerialParser "Restarting system", ] + CRASH_PATTERNS = [ + "mount.nfs: Connection timed out", + "No space left on device", + ] + def initialize @host2head = Hash(String, Array(String)).new @host2rt = Hash(String, String).new + @mq = MQClient.instance end def host_in_msg(msg) @@ -59,6 +66,14 @@ class SerialParser end end + def detect_crash(msg, host) + message = msg["message"].to_s + CRASH_PATTERNS.each do |pattern| + matched = message.match(/.*(?<crash>#{pattern})/) + return matched.named_captures["crash"] unless matched.nil? + end + end + def delete_host(msg, host, signal) boundary_signal = detect_start_or_end(msg, host, signal) return unless boundary_signal @@ -67,10 +82,43 @@ class SerialParser @host2rt.delete(host) end - def save_dmesg_to_result_root(msg) + def mq_publish(msg, host) + crash_signal = detect_crash(msg, host) + return unless crash_signal + + job_id = "" + if @host2rt.has_key?(host) + job_id = File.basename(@host2rt[host]) + end + + mq_msg = { + "job_id" => job_id, + "testbox" => host, + "time" => msg["time"]? || Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800"), + "job_state" => "crash" + } + spawn mq_publish_check("job_mq", mq_msg.to_json) + end + + def mq_publish_check(queue, msg) + 3.times do + @mq.publish_confirm(queue, msg) + break + rescue e + res = @mq.reconnect + sleep 5 + end + end + + def deal_serial_log(msg) host = host_in_msg(msg) return unless host + mq_publish(msg, host) + save_dmesg_to_result_root(msg, host) + end + + def save_dmesg_to_result_root(msg, host) delete_host(msg, host, START_PATTERNS) check_save = check_save_dmesg(msg, host) -- 2.23.0 [View Less]

1 0

[PATCH compass-ci] src/lib: add on_job_crash
by Li Ping 23 Mar '21

23 Mar '21

[how] mq publish the crash event, lifecycle will deal the job event. when job_state is crash and the job is consumed, then close the job and reboot crash machine, if no job, just reboot crash machine. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/lifecycle.cr | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 4bd2ac0..8f34903 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -122,6 … [View More]

1 0

[PATCH lab-z9 3/3] hosts: taishan200-2280-2s48p-256g--a65
by Wei Jihui 23 Mar '21

23 Mar '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a65 | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a65 diff --git a/hosts/taishan200-2280-2s48p-256g--a65 b/hosts/taishan200-2280-2s48p-256g--a65 new file mode 100644 index 0000000..341303c --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a65 @@ -0,0 +1,21 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions:… [View More]

1 0

[PATCH lab-z9 2/3] hosts: add taishan200-2280-2s48p-256g--a64
by Wei Jihui 23 Mar '21

23 Mar '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a64 | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a64 diff --git a/hosts/taishan200-2280-2s48p-256g--a64 b/hosts/taishan200-2280-2s48p-256g--a64 new file mode 100644 index 0000000..f8aa111 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a64 @@ -0,0 +1,21 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions:… [View More]

1 0

[PATCH lab-z9 1/3] hosts: add taishan200-2280-2s48p-256g--a63
by Wei Jihui 23 Mar '21

23 Mar '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a63 | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-256g--a63 diff --git a/hosts/taishan200-2280-2s48p-256g--a63 b/hosts/taishan200-2280-2s48p-256g--a63 new file mode 100644 index 0000000..a8e6688 --- /dev/null +++ b/hosts/taishan200-2280-2s48p-256g--a63 @@ -0,0 +1,21 @@ +arch: aarch64 +nr_node: 4 +nr_cpu: 96 +memory: 256G +nr_hdd_partitions:… [View More]

1 0

[PATCH compass-ci] lib/git_mirror.rb: change the way to calculate priority for repo
by Li Yuanchao 23 Mar '21

23 Mar '21

Change the way to calculate priority for repo, to make active repo be updated more times and more effective. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 36 +++++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 50a6c75..5ba9154 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -115,7 +115,6 @@ class MirrorMain def initialize @feedback_queue = Queue.new … [View More] @fork_stat = {} - @priority = 0 @priority_queue = PriorityQueue.new @git_info = {} @defaults = {} @@ -221,11 +220,9 @@ class MirrorMain def push_git_queue return if @git_queue.size >= 1 - fork_key = @priority_queue.delete_min_return_key + fork_key, old_pri = @priority_queue.delete_min do_push(fork_key) - priority_set = @priority > @fork_stat[fork_key][:priority] ? (@priority - @fork_stat[fork_key][:priority]) : 1 - @priority_queue.push fork_key, priority_set - @priority += 1 + @priority_queue.push fork_key, get_repo_priority(fork_key, old_pri) end def main_loop @@ -251,8 +248,7 @@ class MirrorMain @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo], belong) fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end def compare_refs(cur_refs, old_refs) @@ -388,8 +384,7 @@ class MirrorMain @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true, 'belong' => belong } fork_stat_init(git_repo) - @priority_queue.push git_repo, @priority - @priority += 1 + @priority_queue.push git_repo, get_repo_priority(git_repo, 0) end end @@ -474,6 +469,8 @@ end # main thread class MirrorMain + WEEK_SECONDS = 604800 + def merge_defaults(object_key, object, belong) return object if object_key == belong @@ -541,4 +538,25 @@ class MirrorMain return true end + + def get_repo_priority(git_repo, old_pri) + old_pri ||= 0 + mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}" + mirror_dir = "#{mirror_dir}.git" unless @git_info[git_repo]['is_submodule'] + + return old_pri + Math.cbrt(WEEK_SECONDS) unless File.directory?(mirror_dir) + + return cal_priority(mirror_dir, old_pri) + end + + def cal_priority(mirror_dir, old_pri) + last_commit_time = %x(git -C #{mirror_dir} log --pretty=format:"%ct" -1 2>/dev/null).to_i + return old_pri + Math.cbrt(WEEK_SECONDS) if last_commit_time.zero? + + t = Time.now.to_i + interval = t - last_commit_time + return old_pri + Math.cbrt(WEEK_SECONDS) if interval <= 0 + + return old_pri + Math.cbrt(interval) + end end -- 2.23.0 [View Less]

1 0

[PATCH compass-ci 1/2] service/sshr: change port
by Wu Zhende 23 Mar '21

23 Mar '21

Restarting the sshr service will disconnect the existing sshr connection and the machine that has been applied for can't be accessed. Therefore, we can't restart the sshr service. Instead, we start a new sshr service and use a new port. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler-https/start | 2 +- container/scheduler/start | 4 ++-- container/ssh-r/start | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/… [View More]

2 1

[PATCH lkp-tests] stats/build-pkg: update filtering rule
by Lin Jiaxin 23 Mar '21

23 Mar '21

error_ids that are filtered out: /tmp/lkp/clementine-git/src/Clementine/3rdparty/qsqlite/qsql_sqlite.cpp: In function ‘QSqlError qMakeError(sqlite3*, const QString&, QSqlError::ErrorType, int)’: make: *** [makefile.linux:68: makefile.dependencies] Error 1 go: finding github.com/pingcap/errors v0.11.5-0.20201126102027-b0a155152ca3 Cyclomatic Complexity 8 ./include/trace/events/error_report.h:trace_error_report_end etc. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- stats/build-… [View More]

1 0

[PATCH compass-ci 3/3] sparrow/3-code/dev-env: add upstream config file to fix git-mirror error
by Liu Yinsi 23 Mar '21

23 Mar '21

[error] 2021-03-18 14:24:19.000000000 +0800 git-mirror: ERROR: No upstream-config file 2021-03-18 14:25:20.000000000 +0800 git-mirror: /c/compass-ci/lib/git_mirror.rb:171:in `load_fork_info': undefined method `[]' for nil:NilClass (NoMethodError) 2021-03-18 14:25:20.000000000 +0800 git-mirror: from /c/compass-ci/lib/git_mirror.rb:126:in `initialize' 2021-03-18 14:25:20.000000000 +0800 git-mirror: from ./git-mirror.rb:8:in `new' 2021-03-18 14:25:20.000000000 +0800 git-mirror: … [View More]

1 0

[PATCH compass-ci 2/3] sparrow/3-code/dev-env: set $REPO_SRC value
by Liu Yinsi 23 Mar '21

23 Mar '21

before: REPO_SRC=/c/upstream-repos after: REPO_SRC=/c/git-repos [why] can refer: [PATCH compass-ci] container/git-mirror: change mount dir "-v #{ENV['REPO_SRC']}:#{DOCKER_REPO}" in $CCI_SRC/container/git-mirror/start script,, so need to change $REPO_SRC value. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index e182b65..eb162f7 100755 --- a/… [View More]

1 0

[PATCH compass-ci 1/3] sparrow/1-storage: add work dir and permission for /c/git-repos
by Liu Yinsi 23 Mar '21

23 Mar '21

the URL of the repository to be tested will store in /c/git-repos, set lkp:committer permission for git-mirror. refer: [PATCH compass-ci] container/git-mirror: change mount dir Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 2 ++ sparrow/1-storage/tiny | 1 + 2 files changed, 3 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 26086d5..3d9bdfa 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-… [View More]

1 0

[PATCH lkp-tests] daemon/sshd: change sshr service port
by Wu Zhende 23 Mar '21

23 Mar '21

Restarting the sshr service will disconnect the existing sshr connection and the machine that has been applied for can't be accessed. Therefore, we can't restart the sshr service. Instead, we start a new sshr service and use a new port. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- daemon/sshd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/daemon/sshd b/daemon/sshd index 50a7677ab..e0616ea8b 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -13,7 +13,7 @@ … [View More]

1 0

[PATCH compass-ci] docker/docker.rb: fix failed to borrow and run multi-docker
by Lin Jiaxin 23 Mar '21

23 Mar '21

job id=z9.1471951 ==> /tmp/stderr <== /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require': cannot load such file -- ../lib/mq_client (LoadError) from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:59:in `require' from /c/compass-ci/providers/docker/docker.rb:11:in `<top (required)>' from /c/compass-ci/providers/multi-docker:7:in `require_relative' from /c/compass-ci/providers/multi-docker:7:in `<main>' Signed-off-… [View More]

1 0

[PATCH compass-ci 2/2] provides: fix require error
by Wu Zhende 23 Mar '21

23 Mar '21

[Error] can't load such file "../lib/mq_client" [How] use absolute path Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 4 +++- providers/multi-qemu | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 9d6854f..b372a11 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -3,12 +3,14 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All … [View More]

1 0

[PATCH v4 lkp-tests] tests/ansible_test: ansible test case
by Wang Chenglong 23 Mar '21

23 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 81 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 000000000..bf75b19b7 --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,81 @@ +#!/bin/bash +# - ansible_repo + +. $LKP_SRC/lib/debug.sh + +check_vars() +{ + ansible_host=localhost + [ -n "$ansible_repo" ] || … [View More]

3 6

[PATCH compass-ci 3/3] container/mail-robot: optimize git operations
by Luan Shengde 22 Mar '21

22 Mar '21

Use gem resource ruby-git instead of system command. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index e6c35f7..ce01fec 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-… [View More]

1 0

[PATCH compass-ci 2/3] add use ENV variable
by Luan Shengde 22 Mar '21

22 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 2 +- container/mail-robot/start | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 17705ec..97c3989 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -56,7 +56,7 @@ class ApplyAccount @es_port = ES_PORT … [View More]

1 0

[PATCH compass-ci 1/3] add install ruby-git
by Luan Shengde 22 Mar '21

22 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile index e4b3e7c..41056a5 100644 --- a/container/mail-robot/Dockerfile +++ b/container/mail-robot/Dockerfile @@ -10,7 +10,7 @@ ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ - apt-get install -y … [View More]

1 0

[PATCH v4 compass-ci] container/web-backend: add API /get_table_error
by Lu Weitao 22 Mar '21

22 Mar '21

[Example] curl -X GET 'localhost:10002/get_job_error?group_id=wcl_ansible-openeuler-03-10' return: { "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, "attributes":["job_id","error_id","error_message","link_to_result"], "objects":[ { "job_id": "crystal.1354921", "error_id": "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-xxx.fail", "error_message": "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for … [View More]httpd.service failed because a timeout was exceeded.}", "link_to_result": "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…", ate/crystal.1354921" }, { "job_id": "crystal.1354929", ... }, ... ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 19 +++++++++ src/lib/web_backend.rb | 67 +++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..91cebb5 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,22 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_job_error?suite=virttest&tbox_group=vm-2p8g +# Response +# { +# "filter": {"group_id":"wcl_ansible-openeuler-03-10"}, +# "attributes": ["job_id", "error_id", "error_message", "link_to_result"], +# "objects": [ +# { +# "job_id": "crystal.1354921", +# "error_id": "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-failed.fail", +# "error_message": "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", +# "link_to_result": "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…" +# }, +# ... +# ] +# } +get '/get_job_error' do + get_job_error(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..08bffcb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/params_group.rb" require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -548,3 +549,69 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + + +# ------------------------------------------------------------------------------------------- +# job error table like: +# job_id error_id error_message link to result +# ------------------------------------------------------------------------------------- +# crystal.630608 "stderr.xxx" "messag:xxxx" https://$host:$port/$result_root +# ... +# ------------------------------------------------------------------------------------------- + +def get_job_error(params) + begin + body = job_error_body(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get error table error'] + end + + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def job_error_body(params) + error_objects = get_error_objects(params) + { + filter: params, + attributes: ['job_id', 'error_id', 'error_message', 'link_to_result'], + objects: error_objects, + }.to_json +end + +def get_error_objects(filter_items) + error_objs = [] + + job_list = get_job_list(filter_items) + job_list.each do |job| + error_obj = get_error_from_job(job) + error_objs << error_obj unless error_obj.empty? + end + + error_objs +end + +def get_job_list(items) + es = ESQuery.new(ES_HOST, ES_PORT) + query_results = es.multi_field_query(items) + + extract_jobs_list(query_results['hits']['hits']) +end + +# get all error_id from one job +def get_error_from_job(job) + job_error_obj = {} + job['stats'].each do |metric, value| + next unless metric.end_with?('.message') + + result_host = ENV['SRV_HTTP_RESULT_HOST'] || SRV_HTTP_RESULT_HOST + result_port = ENV['SRV_HTTP_RESULT_PORT'] || SRV_HTTP_RESULT_PORT + error_id = metric.sub('.message', '.fail') + job_error_obj['id'] = job['id'] + job_error_obj['error_id'] = error_id + job_error_obj['error_message'] = value + job_error_obj['link_to_result'] = value + end + + job_error_obj +end -- 2.23.0 [View Less]

1 0

[PATCH v2 compass-ci] providers/qemu/kvm.sh: added QEMU multiple network card functionality
by Wang Chenglong 22 Mar '21

22 Mar '21

[How] Use the 'nr_nic' to set the number of network cards in yaml. nr_nic: 2 root@vm-2p8g ~# ifconfig enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.42.60 netmask 255.255.0.0 broadcast 172.18.255.255 inet6 fe80::843:24ff:fef2:ec17 prefixlen 64 scopeid 0x20<link> ether 0a:43:24:f2:ec:17 txqueuelen 1000 (Ethernet) RX packets 155492 bytes 1172898602 (1.0 GiB) RX errors 0 dropped 0 overruns 0 frame 0 … [View More]TX packets 75 bytes 11207 (10.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp0s2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.88.228 netmask 255.255.0.0 broadcast 172.18.255.255 inet6 fe80::820:70ff:fe34:d27b prefixlen 64 scopeid 0x20<link> ether 0a:20:70:34:d2:7b txqueuelen 1000 (Ethernet) RX packets 168 bytes 9079 (8.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 199208 bytes 32959935 (31.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- providers/qemu/kvm.sh | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index dcab81a..a1c0046 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -119,9 +119,34 @@ add_disk() done } +set_mac() +{ + job_id=$(awk -F'/' '/job_initrd_tmpfs/{print $(NF-1)}' $ipxe_script) + + if [ $(command -v es-find) ]; then + nr_nic=$(es-find id=$job_id | awk '/nr_nic/{print $2}' | tr -d ,) + else + echo "command not found: es-find. set nr_nic=1" + sleep 1 + fi + + mac_arr[1]=$mac + nr_nic=${nr_nic:-1} + + [ "$nr_nic" -ge "2" ] || return + for i in $(seq 2 $nr_nic) + do + mac=$(echo $hostname$i | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/0a-\1-\2-\3-\4-\5/') + mac_arr[$i]=$mac + done +} + set_nic() { - nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" + for i in $(seq 1 $nr_nic) + do + nic[$i]="-nic tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac_arr[$i]}" + done } set_device() @@ -190,7 +215,7 @@ individual_option() -machine virt-4.0,accel=kvm,gic-version=3 -cpu Kunpeng-920 -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-kvm) @@ -198,17 +223,17 @@ individual_option() -machine virt-4.0,accel=kvm,gic-version=3 -cpu Kunpeng-920 -bios $bios - -nic $nic + ${nic[@]} ) [ "$(arch)" == "x86_64" ] && arch_option=( -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-system-x86_64) arch_option=( -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-system-riscv64) @@ -234,6 +259,7 @@ set_options() { set_bios set_helper + set_mac set_nic set_device set_netdev -- 2.23.0 [View Less]

1 0

[PATCH compass-ci] container/srv-http: add srv-http-rpm
by Li Ping 22 Mar '21

22 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/srv-http/start-rpm | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/srv-http/start-rpm diff --git a/container/srv-http/start-rpm b/container/srv-http/start-rpm new file mode 100755 index 0000000..68563b9 --- /dev/null +++ b/container/srv-http/start-rpm @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights … [View More]

1 1

[PATCH v2 compass-ci] container/srv-http: add srv-http-rpm
by Li Ping 22 Mar '21

22 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/srv-http/start | 1 + container/srv-http/start-rpm | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+) create mode 100755 container/srv-http/start-rpm diff --git a/container/srv-http/start b/container/srv-http/start index f262d81..5de1273 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -7,4 +7,5 @@ ./start-initrd ./start-os ./start-git +./start-rpm ./start-cci diff --git a/container/srv-… [View More]

1 0

[PATCH compass-ci] container: modify the services port
by Yu Chuan 20 Mar '21

20 Mar '21

[Why] According our latest port policy, modify the services port: - rsync-server: 11387 => 20004 - netdata: 19999 => 20012 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/netdata/start | 2 +- container/rsync-server/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/netdata/start b/container/netdata/start index 58fb6755be94..18d225b8fcae 100755 --- a/container/netdata/start +++ b/container/netdata/start @@ -10,7 +10,7 @@ cmd=( … [View More]

1 0

[PATCH compass-ci] sparrow: modify the default system param
by Yu Chuan 20 Mar '21

20 Mar '21

[Why] According our latest port policy, modify the system param: net.ipv4.ip_local_port_range = 32000 65535 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 8942247ae4d1..9fdb4be94c38 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -14,7 +14,7 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=… [View More]

1 0

[PATCH compass-ci] job.cr: when init job from es no need to reload the service ports
by Cao Xueliang 20 Mar '21

20 Mar '21

move load service ports to the submit Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b96cf15..c1ca223 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -53,7 +53,6 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h - @hash.merge!(testbox_env) @es = Elasticsearch::Client.new @account_info = Hash(String, … [View More]

1 0

[PATCH v3 compass-ci 2/2] src/lib/web_backend.rb: implement for group jobs stats count
by Li Yuanchao 19 Mar '21

19 Mar '21

Query jobs by conditions such as group id or suite, and count the number of passed cases and failed cases. output is like: { "kezhiming" : { "nr_all" : $nr_all, "nr_pass" : $nr_pass, "nr_fail" : $nr_fail }, "chenqun" : { "nr_all" : $nr_all, "nr_pass" : $nr_pass, "nr_fail" : $nr_fail }, ... … [View More] } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 68 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..04bccdb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -121,6 +121,8 @@ end def get_dimension_conditions(params) dimension = params.key?(:dimension) ? [params.delete(:dimension)] : [] + dimension = params.key?(:GROUP_BY) ? [params.delete(:GROUP_BY)] : [] if dimension.empty? + conditions = {} FIELDS.each do |f| v = params[f] @@ -548,3 +550,69 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def single_count(stats) + fail_count = 0 + pass_count = 0 + single_nr_fail = 0 + single_nr_pass = 0 + stats.each do |stat, value| + fail_count += 1 if stat.match(/\.fail$/i) + pass_count += 1 if stat.match(/\.pass$/i) + single_nr_fail = value if stat.match(/\.nr_fail$/i) + single_nr_pass = value if stat.match(/\.nr_pass$/i) + end + fail_count = single_nr_fail.zero? ? fail_count : single_nr_fail + pass_count = single_nr_pass.zero? ? pass_count : single_nr_pass + [fail_count, pass_count, fail_count + pass_count] +end + +def count_stats(job_list) + nr_fail = 0 + nr_pass = 0 + nr_all = 0 + job_list.each do |job| + next unless job['_source']['stats'] + + fail_count, pass_count, all_count = single_count(job['_source']['stats']) + nr_fail += fail_count + nr_pass += pass_count + nr_all += all_count + end + { 'nr_fail' => nr_fail, 'nr_pass' => nr_pass, 'nr_all' => nr_all } +end + +def get_jobs_stats_count(dimension, must, size, from) + dimension_list = get_dimension_list(dimension) + stats_count = {} + dimension_list.each do |dim| + job_list = query_dimension(dimension[0], dim, must, size, from) + stats_count[dim] = count_stats(job_list) + end + stats_count.to_json +end + +def get_stats_by_dimension(conditions, dimension, must, size, from) + must += build_multi_field_subquery_body(conditions) + count_query = { query: { bool: { must: must } } } + total = es_count(count_query) + return {} if total < 1 + + get_jobs_stats_count(dimension, must, size, from) +end + +def get_jobs_stats(params) + dimension, conditions = get_dimension_conditions(params) + must = get_es_must(params) + get_stats_by_dimension(conditions, dimension, must, 1000, 0) +end + +def group_jobs_stats(params) + begin + body = get_jobs_stats(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0 [View Less]

1 0

[PATCH compass-ci] scheduler/elasticsearch_client.cr: fix search error
by Wu Zhende 19 Mar '21

19 Mar '21

Unhandled exception: Missing hash key: "hits" (KeyError) from /usr/lib/crystal/core/hash.cr:0:9 in '[]' from /usr/lib/crystal/core/json/any.cr:102:7 in '[]' from /c/cci/src/scheduler/elasticsearch_client.cr:85:5 in 'search' from /c/cci/src/lib/lifecycle.cr:109:5 in 'get_active_machines' from /c/cci/src/lib/lifecycle.cr:60:5 in 'init_from_es' from /c/cci/src/lifecycle.cr:14:3 in '__crystal_main' from /usr/lib/crystal/core/crystal/main.cr:97:5 in 'main_user_code' from /usr/lib/… [View More]

1 0

[PATCH v3 compass-ci] container/web-backend: add API /get_table_error
by Lu Weitao 19 Mar '21

19 Mar '21

[Example] curl -X GET 'localhost:30000/get_job_error?group_id=wcl_ansible-openeuler-03-10' return: { "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, "table_fields":["job_id","error_id","error_message","link to result"], "data_table":[ [ "crystal.1354921", "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-xxx.fail", "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a … [View More]timeout was exceeded.}", "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene… ate/crystal.1354921" ], [ "crystal.1354929", ... ] ... ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 19 +++++++++ src/lib/web_backend.rb | 68 +++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..284e7a3 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,22 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_job_error?suite=virttest&tbox_group=vm-2p8g +# Response +# { +# "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, +# "table_fields":["job_id","error_id","error_message","link to result"], +# "data_table":[ +# [ +# "crystal.1354921", +# "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-failed.fail", +# "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.} +# "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…" +# ], +# ... +# ] +# } +get '/get_job_error' do + get_job_error(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..48b7ef2 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/params_group.rb" require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -548,3 +549,70 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + + +# ------------------------------------------------------------------------------------------- +# job error table like: +# job_id error_id error_message link to result +# ------------------------------------------------------------------------------------- +# crystal.630608 "stderr.xxx" "messag:xxxx" https://$host:$port/$result_root +# ... +# ------------------------------------------------------------------------------------------- + +def get_job_error(params) + begin + body = job_error_body(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get error table error'] + end + + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def job_error_body(params) + error_table = get_error_table(params) + { + filter: params, + table_fields: ["job_id", "error_id", "error_message", "link to result"], + data_table: error_table, + }.to_json +end + +def get_error_table(filter_items) + error_table = [] + + job_list = get_job_list(filter_items) + job_list.each do |job| + error_table += get_error_from_job(job) + end + + error_table +end + +def get_job_list(items) + es = ESQuery.new(ES_HOST, ES_PORT) + query_results = es.multi_field_query(items) + + extract_jobs_list(query_results['hits']['hits']) +end + +# get all error_id from one job +def get_error_from_job(job) + job_error = [] + job['stats'].each do |metric, value| + next unless metric.end_with?('.message') + + result_host = ENV['SRV_HTTP_RESULT_HOST'] || SRV_HTTP_RESULT_HOST + result_port = ENV['SRV_HTTP_RESULT_PORT'] || SRV_HTTP_RESULT_PORT + error_id = metric.sub('.message', '.fail') + job_error << [ + job['id'], + error_id, + value, + "http://#{result_host}:#{result_port}#{job['result_root']}" + ] + end + + job_error +end -- 2.23.0 [View Less]

2 4

[PATCH lkp-tests] lib/monitor: add mirror_result action judgement
by Li Ping 19 Mar '21

19 Mar '21

[why] submit -m would trigger mirror_result, even if i have not use the option "-r" Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/monitor.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/monitor.rb b/lib/monitor.rb index 24191fa8d..f6dcef7fd 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -96,6 +96,8 @@ class Monitor end def mirror_result + return unless @action['mirror_result'] + @result_roots.each do |res| res.to_s.delete_prefix!('/srv') … [View More]

1 0

[PATCH v2 compass-ci 1/2] container/web-backend: add api for group jobs stats count
by Li Yuanchao 19 Mar '21

19 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..2971719 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,28 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_jobs_summary?suite=iperf&dimension=… [View More]

2 2

[PATCH compass-ci] dracut-initrd: add support of custom bootstrap-$(arch).cgz
by Yu Chuan 19 Mar '21

19 Mar '21

[Why] When user want run job with their own custom bootstrap instead of lkp-tests framework, they will generate a bootstrap-$(arch).cgz firstly, and scheduler will give the bootstrap-$(arch).cgz path to testbox, then testbox download the bootstrap-$(arch).cgz and use it as part of initrd. So we add a step to handle bootstrap-$(arch).cgz, extract all of it to $NEWROOT and boot from $NEWROOT. [Addition] The root dir in bootstrap-$(arch).cgz is /custom_bootstrap. Signed-off-by: Yu Chuan <… [View More]

1 0

[PATCH compass-ci] scheduler: add custom_bootstrap logic
by Cao Xueliang 19 Mar '21

19 Mar '21

If the custom_bootstrap field in job yaml, need runtime field too, then just return the custom_bootstrap.cgz to the testbox. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 40aaa62..b96cf15 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -514,6 +514,15 @@ class Job private def common_initrds temp_initrds = [] of String + if @hash.has_key?("… [View More]

1 0

[PATCH v2 compass-ci] providers/qemu/kvm.sh: added QEMU multiple network card functionality
by Wang Chenglong 19 Mar '21

19 Mar '21

[How] Use the 'nr_nic' to set the number of network cards in yaml. nr_nic: 2 root@vm-2p8g ~# ifconfig enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.42.60 netmask 255.255.0.0 broadcast 172.18.255.255 inet6 fe80::843:24ff:fef2:ec17 prefixlen 64 scopeid 0x20<link> ether 0a:43:24:f2:ec:17 txqueuelen 1000 (Ethernet) RX packets 155492 bytes 1172898602 (1.0 GiB) RX errors 0 dropped 0 overruns 0 frame 0 … [View More]TX packets 75 bytes 11207 (10.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp0s2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.88.228 netmask 255.255.0.0 broadcast 172.18.255.255 inet6 fe80::820:70ff:fe34:d27b prefixlen 64 scopeid 0x20<link> ether 0a:20:70:34:d2:7b txqueuelen 1000 (Ethernet) RX packets 168 bytes 9079 (8.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 199208 bytes 32959935 (31.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- providers/qemu/kvm.sh | 36 +++++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index dcab81a..a1c0046 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -119,9 +119,34 @@ add_disk() done } +set_mac() +{ + job_id=$(awk -F'/' '/job_initrd_tmpfs/{print $(NF-1)}' $ipxe_script) + + if [ $(command -v es-find) ]; then + nr_nic=$(es-find id=$job_id | awk '/nr_nic/{print $2}' | tr -d ,) + else + echo "command not found: es-find. set nr_nic=1" + sleep 1 + fi + + mac_arr[1]=$mac + nr_nic=${nr_nic:-1} + + [ "$nr_nic" -ge "2" ] || return + for i in $(seq 2 $nr_nic) + do + mac=$(echo $hostname$i | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/0a-\1-\2-\3-\4-\5/') + mac_arr[$i]=$mac + done +} + set_nic() { - nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" + for i in $(seq 1 $nr_nic) + do + nic[$i]="-nic tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac_arr[$i]}" + done } set_device() @@ -190,7 +215,7 @@ individual_option() -machine virt-4.0,accel=kvm,gic-version=3 -cpu Kunpeng-920 -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-kvm) @@ -198,17 +223,17 @@ individual_option() -machine virt-4.0,accel=kvm,gic-version=3 -cpu Kunpeng-920 -bios $bios - -nic $nic + ${nic[@]} ) [ "$(arch)" == "x86_64" ] && arch_option=( -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-system-x86_64) arch_option=( -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-system-riscv64) @@ -234,6 +259,7 @@ set_options() { set_bios set_helper + set_mac set_nic set_device set_netdev -- 2.23.0 [View Less]

1 0

[PATCH compass-ci] providers/qemu/kvm.sh: added QEMU multiple network card functionality
by Wang Chenglong 19 Mar '21

19 Mar '21

[How] Use the 'nr_nic' to set the number of network cards in yaml. nr_nic: 2 root@vm-2p8g ~# ifconfig enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.42.60 netmask 255.255.0.0 broadcast 172.18.255.255 inet6 fe80::843:24ff:fef2:ec17 prefixlen 64 scopeid 0x20<link> ether 0a:43:24:f2:ec:17 txqueuelen 1000 (Ethernet) RX packets 155492 bytes 1172898602 (1.0 GiB) RX errors 0 dropped 0 overruns 0 frame 0 … [View More]TX packets 75 bytes 11207 (10.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp0s2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.88.228 netmask 255.255.0.0 broadcast 172.18.255.255 inet6 fe80::820:70ff:fe34:d27b prefixlen 64 scopeid 0x20<link> ether 0a:20:70:34:d2:7b txqueuelen 1000 (Ethernet) RX packets 168 bytes 9079 (8.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 199208 bytes 32959935 (31.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- providers/qemu/kvm.sh | 37 ++++++++++++++++++++++++++++++++----- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index dcab81a..bbd5cd5 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -119,9 +119,35 @@ add_disk() done } +set_mac() +{ + job_id=$(cat ipxe_script|awk -F'/' '/job_initrd_tmpfs/{print $(NF-1)}') + + if [ $(command -v es-find) ]; then + nr_nic=$(es-find id=$job_id | awk '/nr_nic/{print $2}' | tr -d ,) + else + echo "command not found: es-find. set nr_nic=1" + sleep 1 + fi + + mac_arr[1]=$mac + nr_nic=${nr_nic:-1} + + if [ "$nr_nic" -ge "2" ]; then + for i in $(seq 2 $nr_nic) + do + mac=$(echo $hostname$i | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/0a-\1-\2-\3-\4-\5/') + mac_arr[$i]=$mac + done + fi +} + set_nic() { - nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" + for i in $(seq 1 $nr_nic) + do + nic[$i]="-nic tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac_arr[$i]}" + done } set_device() @@ -190,7 +216,7 @@ individual_option() -machine virt-4.0,accel=kvm,gic-version=3 -cpu Kunpeng-920 -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-kvm) @@ -198,17 +224,17 @@ individual_option() -machine virt-4.0,accel=kvm,gic-version=3 -cpu Kunpeng-920 -bios $bios - -nic $nic + ${nic[@]} ) [ "$(arch)" == "x86_64" ] && arch_option=( -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-system-x86_64) arch_option=( -bios $bios - -nic $nic + ${nic[@]} ) ;; qemu-system-riscv64) @@ -234,6 +260,7 @@ set_options() { set_bios set_helper + set_mac set_nic set_device set_netdev -- 2.23.0 [View Less]

3 4

[PATCH v2 compass-ci] sbin/es-jobs: add -e to get fail result
by Wu Zhende 19 Mar '21

19 Mar '21

es-jobs -e group_id=xxxx error.result.path=/srv/result/openeuler_docker/xxxx: ["Error: unable to find a match"] Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 24 +++++++++++++++++++++++- sbin/es-jobs | 7 ++++++- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index b0d0fb2..e4e9530 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -12,13 +12,14 @@ require_relative './es_query' # deal jobs search from es … [View More]class ESJobs - def initialize(es_query, my_refine = [], fields = [], stats_filter = []) + def initialize(es_query, my_refine = [], fields = [], stats_filter = [], error_result = false) @es_query = es_query @jobs = query_jobs_from_es @refine = my_refine @fields = fields @stats_filter = stats_filter @refine_jobs = [] + @error_result = error_result set_jobs_summary end @@ -98,6 +99,7 @@ class ESJobs 'max.stats' => {}, 'min.stats' => {} } + result['error.result'] = {} if @error_result metrics.each { |metric| result['raw.stats'][metric] = [] } result end @@ -119,7 +121,27 @@ class ESJobs result['kvcount']["#{keyword}=#{job[keyword]}"] += 1 result['raw.id']["[#{keyword}=#{job[keyword]}]"] ||= [] result['raw.id']["[#{keyword}=#{job[keyword]}]"] << job['id'] + + next unless @error_result && job[keyword] == "failed" + + get_failed_result(result, job) + end + end + + def get_failed_result(result, job) + path = "/srv#{job['result_root']}/#{job['suite']}" + res = find_error_lines(path) + result['error.result']["path=#{path}"] = res + end + + def find_error_lines(path) + res = [] + return res unless File.exist?(path) + + File.open(path).each do |line| + res << line.to_s if line.match(/error:/i) end + res end def stats_count(result) diff --git a/sbin/es-jobs b/sbin/es-jobs index ff0d540..3804dc9 100755 --- a/sbin/es-jobs +++ b/sbin/es-jobs @@ -22,6 +22,7 @@ end opt_refine = [-1] opt_fields = [] opt_stats_filter = [] +opt_error_result = false opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-jobs [options] search_key1=val1[,val2..] ..' @@ -45,11 +46,15 @@ opt_parser = OptionParser.new do |opts| opts.on('-s fields', '--stats-filter fields', 'return data contains fields in stats') do |fields| opt_stats_filter = fields.split(',') end + + opts.on('-e', '--error.result', 'obtain the error information about job failure') do + opt_error_result = true + end end opt_parser.parse!(ARGV) items = parse_argv raise 'Please enter a query' if items.empty? -es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter) +es_jobs = ESJobs.new(items, opt_refine, opt_fields, opt_stats_filter, opt_error_result) es_jobs.output -- 2.23.0 [View Less]

1 0

review star
by Wu Fengguang 18 Mar '21

18 Mar '21

Yu Chuan is our review star again, thanks YuChuan! @crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 29 4 10 12 Yu Chuan 76 7 9 10 Wei Jihui 71 2 2 8 Cao Xueliang 24 10 14 7 Xu … [View More]

1 0

git count
by Wu Fengguang 18 Mar '21

18 Mar '21

Jihui,Zhende is our code star, thanks Jihui,Zhende! weijihui@crystal ~/todo/bin% ./git-count 8 Wei Jihui 8 Wu Zhende 7 Liu Shaofei 7 Lu Weitao 7 Luan Shengde 6 Bai Jing 6 Xu Xijian 6 Liu Yinsi 6 Li Yuanchao 5 Cao Xueliang 5 Yu Chuan 3 Wang Chenglong 3 Lin Jiaxin 3 Li Ping 1 Cui Yili 1 Wang Yong 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale 0 Zhang Dewan

1 0

help star
by Wu Fengguang 18 Mar '21

18 Mar '21

Zhende is our help star, thanks Zhende! bijing@crystal ~/todo/bin% ./help-star 14.8 Wu Fengguang 13.2 Wu Zhende 13.0 Wei Jihui 12.7 Yu Chuan 10.9 Cao Xueliang 10.5 Wang Yong 8.0 Luan Shengde 5.5 Lu Weitao 5.3 Li Yuanchao 4.8 Liu Yinsi 4.8 Xu Xijian 4.7 Bai Jing 4.6 Cui Yili 3.9 Li Ping 3.7 Lin Jiaxin 3.2 Liu Shaofei 2.6 Zhang Dewan 2.6 Wang Chenglong 1.3 Du Kaitian 0.0 Zhang Yale

1 0

[PATCH compass-ci] src/*.cr: fix ameba warning
by Wang Yong 18 Mar '21

18 Mar '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- src/lib/sched.cr | 2 +- src/scheduler/testbox_env.cr | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index ac677de..b10a16d 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -224,7 +224,7 @@ class Sched @mq.publish_confirm(queue, msg) break rescue e - res = @mq.reconnect + @mq.reconnect sleep 5 end end diff --git a/… [View More]

1 0

[PATCH lkp-tests] tests/deploy-cci: remove -e
by Liu Yinsi 18 Mar '21

18 Mar '21

[why] -e will cause process exit when return code is not 0, only use when debugging, so remove it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/deploy-cci | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 6f97d7151..102cc751c 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -1,4 +1,4 @@ -#!/bin/sh -e +#!/bin/sh # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All … [View More]

1 0

[PATCH lkp-tests] sbin/makepkg: cpio add --quiet to do not print the number of blocks copied
by Lin Jiaxin 18 Mar '21

18 Mar '21

avoid unnecessary stderr like this: ==> /tmp/stderr <== 43 blocks Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/makepkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/makepkg b/sbin/makepkg index cb051641..9352557a 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -2273,7 +2273,7 @@ extend_pkgver_cgz_name() { create_cpio() { find . | grep -Fxv -e ./bin -e ./sbin -e ./lib -e ./lib64 -e ./etc/init.d | - cpio -o -H newc --owner=root.root | + cpio … [View More]

1 0

[PATCH compass-ci 3/3] providers/qemu/pxe.sh: added QEMU multiple network card functionality
by Wang Chenglong 18 Mar '21

18 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- providers/qemu/pxe.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/pxe.sh b/providers/qemu/pxe.sh index 3ab3c2b..b6c4d6d 100755 --- a/providers/qemu/pxe.sh +++ b/providers/qemu/pxe.sh @@ -31,7 +31,7 @@ kvm=( -device virtio-gpu-pci -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -rtc base=localtime - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -… [View More]

1 1

[PATCH compass-ci 2/3] providers/qemu/kvm.sh: added QEMU multiple network card functionality
by Wang Chenglong 18 Mar '21

18 Mar '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- providers/qemu/kvm.sh | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index dcab81a..3b220ef 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -23,8 +23,9 @@ check_logfile() write_logfile() { ipxe_script=ipxe_script - curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + curl http:… [View More]

1 1

[PATCH compass-ci 1/3] providers/qemu.sh: added QEMU multiple network card functionality
by Wang Chenglong 18 Mar '21

18 Mar '21

[How] Use the 'nr_nic' to set the number of network cards in yaml. nr_nic: 2 root@vm-2p8g ~# ifconfig enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.18.115.162 netmask 255.255.0.0 broadcast 172.18.255.255 inet6 fe80::835:ffff:fedf:384e prefixlen 64 scopeid 0x20<link> ether 0a:35:ff:df:38:4e txqueuelen 1000 (Ethernet) RX packets 586 bytes 180087 (175.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX … [View More]packets 80 bytes 12579 (12.2 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp0s2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::868:97ff:fe92:59bb prefixlen 64 scopeid 0x20<link> ether 0a:68:97:92:59:bb txqueuelen 1000 (Ethernet) RX packets 527 bytes 173034 (168.9 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 20 bytes 4760 (4.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- providers/qemu.sh | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/providers/qemu.sh b/providers/qemu.sh index 2fa5771..24211bc 100755 --- a/providers/qemu.sh +++ b/providers/qemu.sh @@ -2,6 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # - hostname +# - nr_nic . $LKP_SRC/lib/yaml.sh . $CCI_SRC/container/defconfig.sh @@ -10,26 +11,40 @@ load_cci_defaults : ${hostname:="vm-1p1g-1"} : ${queues:="vm-1p1g.$(arch)"} +: ${nr_nic:=1} # unicast prefix: x2, x6, xA, xE -export mac=$(echo $hostname | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/0a-\1-\2-\3-\4-\5/') + +set_mac() +{ + export nr_nic=$(expr $nr_nic - 1) + for i in $(seq 0 $nr_nic) + do + mac=$(echo $hostname$i | md5sum | sed 's/^$..$$..$$..$$..$$..$.*$/0a-\1-\2-\3-\4-\5/') + mac_arr[$i]=$mac + done +} +set_mac + + echo hostname: $hostname -echo mac: $mac -echo $mac > mac -echo "arp -n | grep ${mac//-/:}" > ip.sh +echo mac: ${mac_arr[*]} +echo ${mac_arr[*]} > mac +echo "arp -n | grep ${mac_arr[0]//-/:}" > ip.sh chmod +x ip.sh + set_host_info() { # use "," replace " " local api_queues=$(echo $queues | sed -r 's/ +/,/g') - curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac}" + curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host_mac?hostname=${hostname}&mac=${mac_arr[0]}" curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/set_host2queues?host=${hostname}&queues=${api_queues}" } set_host_info del_host_info() { - curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/del_host_mac?mac=${mac}" > /dev/null 2>&1 + curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/del_host_mac?mac=${mac_arr[0]}" > /dev/null 2>&1 curl -X PUT "http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/del_host2queues?host=${hostname}" > /dev/null 2>&1 } -- 2.23.0 [View Less]

1 1

[PATCH compass-ci] container/mail-robot: update new port
by Luan Shengde 18 Mar '21

18 Mar '21

assigned new port for service: assign-account: 29999 => 10000 send-internet-mail: 49000 => 10001 Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 2 +- container/mail-robot/lib/apply-jumper-account.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index 17705ec..1de250b 100755 --- a/container/mail-robot/lib/… [View More]

2 1

[PATCH compass-ci] sparrow: add /srv/rabbitmq for rabbitmq container
by Cao Xueliang 18 Mar '21

18 Mar '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/1-storage/permission | 2 ++ sparrow/1-storage/tiny | 1 + 2 files changed, 3 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 0713d7e..26086d5 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -10,6 +10,7 @@ chmod_dir=( /srv/cci/libvirt-xml /srv/cci/serial/fluentd-pos /srv/cci/Maildir + /srv/rabbitmq ) chmod 775 "${chmod_dir[@]}" @@… [View More]

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: fix spelling error
by Liu Yinsi 18 Mar '21

18 Mar '21

before: initrd-lkp after: lkp-initrd Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index a7ace65..3cd5719 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -62,7 +62,7 @@ do_one_run() } [ -x first-run ] && ./first-run [ -x start ] && ./start - [ "$container_name" == 'initrd-lkp' ] &&… [View More]

1 0