- Compass-ci - mailweb.openeuler.org

[PATCH v4 compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..f96972e 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,6 +34,22 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=20002} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MONITOR_HOST:=$server_ip} +: ${MONITOR_PORT:=20001} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -53,6 +69,8 @@ EOF mkdir -p /etc/compass-ci/service cat > /etc/compass-ci/service/service-env.yaml <<EOF +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT ES_HOST: $ES_HOST ES_PORT: $ES_PORT SCHED_HOST: $sched_host @@ -88,6 +106,19 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MONITOR_HOST: $MONITOR_HOST +MONITOR_PORT: $MONITOR_PORT +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor.rb: update SRV_HTTP_RESULT_PORT
by Liu Yinsi 25 Mar '21

25 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/email.sh | 2 +- lib/monitor.rb | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/email.sh b/lib/email.sh index 63d5a51e2..bca4ab435 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -8,7 +8,7 @@ Subject: [NOTIFY Compass-CI] Test job failed: $id Dear $my_name: Sorry to inform you that your test job is failed, you can click the following link to view details. - http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT:-11300}$result_root + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… Regards Compass-CI diff --git a/lib/monitor.rb b/lib/monitor.rb index f14e131e3..bf7ef69ba 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -100,9 +100,9 @@ class Monitor @result_roots.each do |res| res.to_s.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' - srv_http_port = job['SRV_HTTP_PORT'] || '11300' - url = "http://#{srv_http_host}:#{srv_http_port}#{res}" + srv_http_result_host = job['SRV_HTTP_RESULT_HOST'] || 'api.compass-ci.openeuler.org' + srv_http_result_port = job['SRV_HTTP_RESULT_PORT'] || '20007' + url = "http://#{srv_http_result_host}:#{srv_http_result_port}#{res}" system "lftp -c mirror #{url} >/dev/null 2>&1" end end -- 2.23.0

1 0

help star
by Wu Fengguang 25 Mar '21

25 Mar '21

Yu Chuan, Wu Zhende is our help star again, thanks Yu Chuan, Wu Zhende! wfg@crystal ~/todo/bin% ./help-star ../people/*/credit/2021-03-25.md 15.2 Yu Chuan 15.1 Wu Zhende 11.2 Cao Xueliang 11.0 Wei Jihui 8.8 Liu Yinsi 7.4 Luan Shengde 7.0 Wang Yong 6.4 Lu Weitao 5.6 Wu Fengguang 5.4 Wang Chenglong 5.1 Zhang Dewan 5.0 Li Yuanchao 4.2 Liu Shaofei 4.1 Lin Jiaxin 2.9 Li Ping 1.8 Du Kaitian 1.6 Xu Xijian 1.2 Bai Jing 0.9 Cui Yili 0.0 Zhang Yale

1 0

git count
by Wu Fengguang 25 Mar '21

25 Mar '21

Wei Jihui is our code star, thanks Wei Jihui! wfg@crystal ~/todo/bin% ./git-count 6 Wei Jihui 5 Li Yuanchao 4 Wu Zhende 4 Yu Chuan 4 Luan Shengde 4 Lin Jiaxin 3 Cao Xueliang 3 Liu Shaofei 3 Liu Yinsi 3 Li Ping 2 Bai Jing 2 Xu Xijian 2 Lu Weitao 2 Cui Yili 2 Wang Chenglong 2 Wang Yong 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale 0 Zhang Dewan

1 0

review star
by Wu Fengguang 25 Mar '21

25 Mar '21

Luan Shengde is our review star, thanks Luan Shengde! wfg@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 9 4 19 9 Wu Fengguang 19 9 18 7 Luan Shengde 49 6 6 6 Wei Jihui 21 5 5 6 Liu Shaofei 39 2 2 5 Cao Xueliang 14 4 4 3 Lin Jiaxin 22 3 5 2 Li Yuanchao 3 0 0 2 Xu Xijian 8 3 5 2 Bai Jing 20 2 2 1 Wang Chenglong 17 3 5 1 Wang Yong 9 1 0 1 Lu Weitao 7 2 3 1 Cui Yili 3 0 0 0 Zhang Dewan 4 1 1 0 Yu Chuan 14 0 0 0 Wu Zhende 11 1 1 0 Li Ping 9 0 0 0 Liu Yinsi

1 0

[PATCH v5 compass-ci] container/web-backend: add API /get_table_error
by Lu Weitao 25 Mar '21

25 Mar '21

[Example] curl -X GET 'localhost:10002/get_job_error?group_id=wcl_ansible-openeuler-03-10' return: { "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, "attributes":["job_id","error_id","error_message","link_to_result"], "objects":[ { "job_id": "crystal.1354921", "error_id": "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-xxx.fail", "error_message": "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", "link_to_result": "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…", ate/crystal.1354921" }, { "job_id": "crystal.1354929", ... }, ... ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 18 +++++++++ src/lib/web_backend.rb | 65 +++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index adb91e0..c7d1ec6 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -170,4 +170,22 @@ end # } get '/get_jobs_summary' do group_jobs_stats(params) + +# GET /get_job_error?suite=virttest&tbox_group=vm-2p8g +# Response +# { +# "filter": {"group_id":"wcl_ansible-openeuler-03-10"}, +# "attributes": ["job_id", "error_id", "error_message", "link_to_result"], +# "objects": [ +# { +# "job_id": "crystal.1354921", +# "error_id": "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-failed.fail", +# "error_message": "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", +# "link_to_result": "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…" +# }, +# ... +# ] +# } +get '/get_job_error' do + get_job_error(params) end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 40471e0..7f26b9c 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/params_group.rb" require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -613,4 +614,68 @@ def group_jobs_stats(params) return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] + +# ------------------------------------------------------------------------------------------- +# job error table like: +# job_id error_id error_message link to result +# ------------------------------------------------------------------------------------- +# crystal.630608 "stderr.xxx" "messag:xxxx" https://$host:$port/$result_root +# ... +# ------------------------------------------------------------------------------------------- + +def get_job_error(params) + begin + body = job_error_body(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get error table error'] + end + + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def job_error_body(params) + error_objects = get_error_objects(params) + { + filter: params, + attributes: ['job_id', 'error_id', 'error_message', 'link_to_result'], + objects: error_objects, + }.to_json +end + +def get_error_objects(filter_items) + error_objs = [] + + job_list = get_job_list(filter_items) + job_list.each do |job| + error_obj = get_error_from_job(job) + error_objs << error_obj unless error_obj.empty? + end + + error_objs +end + +def get_job_list(items) + es = ESQuery.new(ES_HOST, ES_PORT) + query_results = es.multi_field_query(items) + + extract_jobs_list(query_results['hits']['hits']) +end + +# get all error_id from one job +def get_error_from_job(job) + job_error_obj = {} + job['stats'].each do |metric, value| + next unless metric.end_with?('.message') + + result_host = ENV['SRV_HTTP_RESULT_HOST'] || SRV_HTTP_RESULT_HOST + result_port = ENV['SRV_HTTP_RESULT_PORT'] || SRV_HTTP_RESULT_PORT + error_id = metric.sub('.message', '.fail') + job_error_obj['id'] = job['id'] + job_error_obj['error_id'] = error_id + job_error_obj['error_message'] = value + job_error_obj['link_to_result'] = value + end + + job_error_obj end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..b2c1814 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,12 +34,33 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=11306} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${LKP_CGI_PORT:=3000} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MAIL_HOST:=$server_ip} +: ${MAIL_PORT:=10001} +: ${MONITOR_HOST:=$server_ip} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} + mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port lab: $lab +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT EOF [[ -f /etc/compass-ci/defaults/upstream-config ]] || { @@ -88,6 +109,21 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +LKP_CGI_PORT: $LKP_CGI_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MAIL_HOST: $MAIL_HOST +MAIL_PORT: $MAIL_PORT +MONITOR_HOST: $MONITOR_HOST +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..005e29e 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,12 +34,32 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=11306} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${LKP_CGI_PORT:=3000} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MAIL_HOST:=$server_ip} +: ${MAIL_PORT:=10001} +: ${MONITOR_HOST:=$server_ip} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port lab: $lab +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT EOF [[ -f /etc/compass-ci/defaults/upstream-config ]] || { @@ -88,6 +108,21 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +LKP_CGI_PORT: $LKP_CGI_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MAIL_HOST: $MAIL_HOST +MAIL_PORT: $MAIL_PORT +MONITOR_HOST: $MONITOR_HOST +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

1 0

[PATCH v2 lkp-tests] stats/build-pkg: update filtering rule
by Lin Jiaxin 25 Mar '21

25 Mar '21

error_ids that are filtered out: /tmp/lkp/clementine-git/src/Clementine/3rdparty/qsqlite/qsql_sqlite.cpp: In function ‘QSqlError qMakeError(sqlite3*, const QString&, QSqlError::ErrorType, int)’: make: *** [makefile.linux:68: makefile.dependencies] Error 1 go: finding github.com/pingcap/errors v0.11.5-0.20201126102027-b0a155152ca3 Cyclomatic Complexity 8 ./include/trace/events/error_report.h:trace_error_report_end etc. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- stats/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/build-pkg b/stats/build-pkg index 90f27656..764594b7 100755 --- a/stats/build-pkg +++ b/stats/build-pkg @@ -20,7 +20,7 @@ while (line = STDIN.gets) seqno = '' end next unless in_stderr - next unless line.downcase =~ /error|warning/ + next unless line.downcase =~ /(error|warning):[^:]/ error_ids << build_pkg_error_id(line) end -- 2.23.0

3 2

[PATCH lkp-tests] stats/ansible_test: add module name in error id
by Wang Chenglong 25 Mar '21

25 Mar '21

before: error.Failed-to-install-some-of-the-specified-packages: 1 after: error.[msg].Failed-to-install-some-of-the-specified-packages: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index d6acad1ab..3d3b0b0cb 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -2,18 +2,18 @@ require 'json' -def output_error(error_msg) +def output_error(error_msg, error_key) return if error_msg.nil? if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.#{error_id}: 1" + puts "error.[#{error_key}].#{error_id}: 1" puts "error.#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.#{error_id}: 1" + puts "error.[#{error_key}].#{error_id}: 1" puts "error.#{error_id}.message: #{@ansible_failed_info}" end end @@ -33,8 +33,8 @@ while (line = STDIN.gets) ansible_failed_json = JSON.parse @ansible_failed_info - output_error ansible_failed_json['msg'] - output_error ansible_failed_json['message'] - output_error ansible_failed_json['cmd'] - output_error ansible_failed_json['failures'] + output_error ansible_failed_json['msg'],'msg' + output_error ansible_failed_json['message'],'message' + output_error ansible_failed_json['cmd'],'cmd' + output_error ansible_failed_json['failures'],'failures' end -- 2.23.0

1 0

2025

2024

2023

2022

2021

2020

Compass-ci