- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/srv-http: add srv-http-rpm
by Li Ping 22 Mar '21

22 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/srv-http/start-rpm | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/srv-http/start-rpm diff --git a/container/srv-http/start-rpm b/container/srv-http/start-rpm new file mode 100755 index 0000000..68563b9 --- /dev/null +++ b/container/srv-http/start-rpm @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-rpm + +cmd=( + docker run + --restart=always + --name srv-http-rpm + -p 20004:11300 + -v /srv/rpm/pub:/srv/rpm/pub:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" -- 2.23.0

1 1

[PATCH v2 compass-ci] container/srv-http: add srv-http-rpm
by Li Ping 22 Mar '21

22 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/srv-http/start | 1 + container/srv-http/start-rpm | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+) create mode 100755 container/srv-http/start-rpm diff --git a/container/srv-http/start b/container/srv-http/start index f262d81..5de1273 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -7,4 +7,5 @@ ./start-initrd ./start-os ./start-git +./start-rpm ./start-cci diff --git a/container/srv-http/start-rpm b/container/srv-http/start-rpm new file mode 100755 index 0000000..73e5162 --- /dev/null +++ b/container/srv-http/start-rpm @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-rpm + +cmd=( + docker run + --restart=always + --name srv-http-rpm + -p 20012:11300 + -v /srv/rpm/pub:/srv/rpm/pub:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] container: modify the services port
by Yu Chuan 20 Mar '21

20 Mar '21

[Why] According our latest port policy, modify the services port: - rsync-server: 11387 => 20004 - netdata: 19999 => 20012 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/netdata/start | 2 +- container/rsync-server/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/netdata/start b/container/netdata/start index 58fb6755be94..18d225b8fcae 100755 --- a/container/netdata/start +++ b/container/netdata/start @@ -10,7 +10,7 @@ cmd=( docker run -d --restart=always --name netdata - -p 19999:19999 + -p 20012:19999 -v /proc:/host/proc:ro -v /sys:/host/sys:ro -v /etc/localtime:/etc/localtime:ro diff --git a/container/rsync-server/start b/container/rsync-server/start index 334f5c2c4a69..25a04f3cef6b 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -13,7 +13,7 @@ cmd=( -v /srv/result/upload:/srv/result/upload -v /srv/os/install/ks:/srv/os/install/ks -v /etc/localtime:/etc/localtime:ro - -p 11387:11387 + -p 20004:11387 rsync_server:latest ) -- 2.23.0

1 0

[PATCH compass-ci] sparrow: modify the default system param
by Yu Chuan 20 Mar '21

20 Mar '21

[Why] According our latest port policy, modify the system param: net.ipv4.ip_local_port_range = 32000 65535 Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/0-package/common | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 8942247ae4d1..9fdb4be94c38 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -14,7 +14,7 @@ cat >> /etc/sysctl.conf <<EOF net.ipv4.ip_forward=1 net.ipv6.bindv6only=1 vm.max_map_count=262144 -net.ipv4.ip_local_port_range = 1024 48999 +net.ipv4.ip_local_port_range = 32000 65535 EOF sysctl -p -- 2.23.0

1 0

[PATCH compass-ci] job.cr: when init job from es no need to reload the service ports
by Cao Xueliang 20 Mar '21

20 Mar '21

move load service ports to the submit Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index b96cf15..c1ca223 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -53,7 +53,6 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h - @hash.merge!(testbox_env) @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new @log = JSONLogger.new @@ -150,6 +149,7 @@ class Job check_account_info() check_run_time() set_defaults() + @hash.merge!(testbox_env) end private def set_defaults -- 2.23.0

1 0

[PATCH v3 compass-ci 2/2] src/lib/web_backend.rb: implement for group jobs stats count
by Li Yuanchao 19 Mar '21

19 Mar '21

Query jobs by conditions such as group id or suite, and count the number of passed cases and failed cases. output is like: { "kezhiming" : { "nr_all" : $nr_all, "nr_pass" : $nr_pass, "nr_fail" : $nr_fail }, "chenqun" : { "nr_all" : $nr_all, "nr_pass" : $nr_pass, "nr_fail" : $nr_fail }, ... } Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 68 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..04bccdb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -121,6 +121,8 @@ end def get_dimension_conditions(params) dimension = params.key?(:dimension) ? [params.delete(:dimension)] : [] + dimension = params.key?(:GROUP_BY) ? [params.delete(:GROUP_BY)] : [] if dimension.empty? + conditions = {} FIELDS.each do |f| v = params[f] @@ -548,3 +550,69 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def single_count(stats) + fail_count = 0 + pass_count = 0 + single_nr_fail = 0 + single_nr_pass = 0 + stats.each do |stat, value| + fail_count += 1 if stat.match(/\.fail$/i) + pass_count += 1 if stat.match(/\.pass$/i) + single_nr_fail = value if stat.match(/\.nr_fail$/i) + single_nr_pass = value if stat.match(/\.nr_pass$/i) + end + fail_count = single_nr_fail.zero? ? fail_count : single_nr_fail + pass_count = single_nr_pass.zero? ? pass_count : single_nr_pass + [fail_count, pass_count, fail_count + pass_count] +end + +def count_stats(job_list) + nr_fail = 0 + nr_pass = 0 + nr_all = 0 + job_list.each do |job| + next unless job['_source']['stats'] + + fail_count, pass_count, all_count = single_count(job['_source']['stats']) + nr_fail += fail_count + nr_pass += pass_count + nr_all += all_count + end + { 'nr_fail' => nr_fail, 'nr_pass' => nr_pass, 'nr_all' => nr_all } +end + +def get_jobs_stats_count(dimension, must, size, from) + dimension_list = get_dimension_list(dimension) + stats_count = {} + dimension_list.each do |dim| + job_list = query_dimension(dimension[0], dim, must, size, from) + stats_count[dim] = count_stats(job_list) + end + stats_count.to_json +end + +def get_stats_by_dimension(conditions, dimension, must, size, from) + must += build_multi_field_subquery_body(conditions) + count_query = { query: { bool: { must: must } } } + total = es_count(count_query) + return {} if total < 1 + + get_jobs_stats_count(dimension, must, size, from) +end + +def get_jobs_stats(params) + dimension, conditions = get_dimension_conditions(params) + must = get_es_must(params) + get_stats_by_dimension(conditions, dimension, must, 1000, 0) +end + +def group_jobs_stats(params) + begin + body = get_jobs_stats(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH compass-ci] scheduler/elasticsearch_client.cr: fix search error
by Wu Zhende 19 Mar '21

19 Mar '21

Unhandled exception: Missing hash key: "hits" (KeyError) from /usr/lib/crystal/core/hash.cr:0:9 in '[]' from /usr/lib/crystal/core/json/any.cr:102:7 in '[]' from /c/cci/src/scheduler/elasticsearch_client.cr:85:5 in 'search' from /c/cci/src/lib/lifecycle.cr:109:5 in 'get_active_machines' from /c/cci/src/lib/lifecycle.cr:60:5 in 'init_from_es' from /c/cci/src/lifecycle.cr:14:3 in '__crystal_main' from /usr/lib/crystal/core/crystal/main.cr:97:5 in 'main_user_code' from /usr/lib/crystal/core/crystal/main.cr:86:7 in 'main' from /usr/lib/crystal/core/crystal/main.cr:106:3 in 'main' from ??? Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/elasticsearch_client.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index f6ff618..219cf3d 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -82,7 +82,10 @@ class Elasticsearch::Client results = @client.search({:index => index, :body => query}) raise results unless results.is_a?(JSON::Any) - results["hits"]["hits"].as_a + return results["hits"]["hits"].as_a unless results.as_h.has_key?("error") + + puts results + Array(JSON::Any).new end def update_account(account_content : JSON::Any, my_email : String) -- 2.23.0

1 0

[PATCH v3 compass-ci] container/web-backend: add API /get_table_error
by Lu Weitao 19 Mar '21

19 Mar '21

[Example] curl -X GET 'localhost:30000/get_job_error?group_id=wcl_ansible-openeuler-03-10' return: { "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, "table_fields":["job_id","error_id","error_message","link to result"], "data_table":[ [ "crystal.1354921", "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-xxx.fail", "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene… ate/crystal.1354921" ], [ "crystal.1354929", ... ] ... ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 19 +++++++++ src/lib/web_backend.rb | 68 +++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..284e7a3 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,22 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_job_error?suite=virttest&tbox_group=vm-2p8g +# Response +# { +# "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, +# "table_fields":["job_id","error_id","error_message","link to result"], +# "data_table":[ +# [ +# "crystal.1354921", +# "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-failed.fail", +# "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.} +# "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…" +# ], +# ... +# ] +# } +get '/get_job_error' do + get_job_error(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..48b7ef2 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/params_group.rb" require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -548,3 +549,70 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + + +# ------------------------------------------------------------------------------------------- +# job error table like: +# job_id error_id error_message link to result +# ------------------------------------------------------------------------------------- +# crystal.630608 "stderr.xxx" "messag:xxxx" https://$host:$port/$result_root +# ... +# ------------------------------------------------------------------------------------------- + +def get_job_error(params) + begin + body = job_error_body(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get error table error'] + end + + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def job_error_body(params) + error_table = get_error_table(params) + { + filter: params, + table_fields: ["job_id", "error_id", "error_message", "link to result"], + data_table: error_table, + }.to_json +end + +def get_error_table(filter_items) + error_table = [] + + job_list = get_job_list(filter_items) + job_list.each do |job| + error_table += get_error_from_job(job) + end + + error_table +end + +def get_job_list(items) + es = ESQuery.new(ES_HOST, ES_PORT) + query_results = es.multi_field_query(items) + + extract_jobs_list(query_results['hits']['hits']) +end + +# get all error_id from one job +def get_error_from_job(job) + job_error = [] + job['stats'].each do |metric, value| + next unless metric.end_with?('.message') + + result_host = ENV['SRV_HTTP_RESULT_HOST'] || SRV_HTTP_RESULT_HOST + result_port = ENV['SRV_HTTP_RESULT_PORT'] || SRV_HTTP_RESULT_PORT + error_id = metric.sub('.message', '.fail') + job_error << [ + job['id'], + error_id, + value, + "http://#{result_host}:#{result_port}#{job['result_root']}" + ] + end + + job_error +end -- 2.23.0

2 4

[PATCH lkp-tests] lib/monitor: add mirror_result action judgement
by Li Ping 19 Mar '21

19 Mar '21

[why] submit -m would trigger mirror_result, even if i have not use the option "-r" Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/monitor.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/monitor.rb b/lib/monitor.rb index 24191fa8d..f6dcef7fd 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -96,6 +96,8 @@ class Monitor end def mirror_result + return unless @action['mirror_result'] + @result_roots.each do |res| res.to_s.delete_prefix!('/srv') srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/web-backend: add api for group jobs stats count
by Li Yuanchao 19 Mar '21

19 Mar '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..2971719 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,28 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end + +# GET /get_jobs_summary?suite=iperf&dimension=my_name +# must: +# - query_conditions +# - suite / group_id / ... +# - dimension +# - group_id / my_email / my_name +# +# Response like: +# - { +# "kezhiming" : { +# "all" : $all, +# "nr_pass" : $nr_pass, +# "nr_fail" : $nr_fali +# }, +# "chenqun" : { +# "all" : $all, +# "nr_pass" : $nr_pass, +# "nr_fail" : $nr_fali +# }, +# ... +# } +get '/get_jobs_summary' do + group_jobs_stats(params) +end -- 2.23.0

2 2