March 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] doc: fix some spell errors
by Li Yuanchao 26 Mar '21

26 Mar '21

blacklight => backlight Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- doc/manual/test-oss-project.en.md | 2 +- doc/manual/test-oss-project.zh.md | 4 ++-- .../{fork_blacklight.png => fork_backlight.png} | Bin 3 files changed, 3 insertions(+), 3 deletions(-) rename doc/pictures/{fork_blacklight.png => fork_backlight.png} (100%) diff --git a/doc/manual/test-oss-project.en.md b/doc/manual/test-oss-project.en.md index a3c0781..6b6d9ca 100644 --- a/doc/manual/test-oss-project.en.md +++ b/doc/manual/test-oss-project.en.md @@ -8,7 +8,7 @@ Perform the following steps to add the information of the code repository to be 1. Fork the upstream-repos repository and clone it to the local host. This document uses the **backlight** repository (https://github.com/baskerville/backlight) as an example. -![](./../pictures/fork_blacklight.png) +![](./../pictures/fork_backlight.png) 2. Run the following command to create a file path named with the repository name and its initial letter: diff --git a/doc/manual/test-oss-project.zh.md b/doc/manual/test-oss-project.zh.md index b455c32..413066d 100644 --- a/doc/manual/test-oss-project.zh.md +++ b/doc/manual/test-oss-project.zh.md @@ -6,9 +6,9 @@ 执行以下步骤，将想要测试的代码仓信息以 yaml 文件的方式添加到 upstream-repos 仓库（https://gitee.com/wu_fengguang/upstream-repos ）。 -1. Fork upstream-repos 仓库并 git clone 到本地，本文以 blacklight 仓库（https://github.com/baskerville/backlight ）为例说明。 +1. Fork upstream-repos 仓库并 git clone 到本地，本文以 backlight 仓库（https://github.com/baskerville/backlight ）为例说明。 -![](./../pictures/fork_blacklight.png) +![](./../pictures/fork_backlight.png) 2. 执行以下命令，以首字母和仓库名创建文件路径。 diff --git a/doc/pictures/fork_blacklight.png b/doc/pictures/fork_backlight.png similarity index 100% rename from doc/pictures/fork_blacklight.png rename to doc/pictures/fork_backlight.png -- 2.23.0

1 0

[PATCH lkp-tests] lib/email.sh: rename $MAIL_HOST
by Liu Yinsi 25 Mar '21

25 Mar '21

MAIL_HOST => SEND_MAIL_HOST [why] because MAIL_HOST is duplicate variable name, $MAIL_HOST is $SEND_MAIL_HOST. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/email.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/email.sh b/lib/email.sh index bca4ab435..4f1055033 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -87,5 +87,5 @@ send_email() ${subject}_content - curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-10001}/send_mail_text" -d "$email_content" + curl -XPOST "${SEND_MAIL_HOST:-$LKP_SERVER}:${SEND_MAIL_PORT:-10001}/send_mail_text" -d "$email_content" } -- 2.23.0

1 0

[PATCH v3 compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..d61f11e 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,6 +34,21 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=20002} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MONITOR_HOST:=$server_ip} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -53,6 +68,8 @@ EOF mkdir -p /etc/compass-ci/service cat > /etc/compass-ci/service/service-env.yaml <<EOF +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT ES_HOST: $ES_HOST ES_PORT: $ES_PORT SCHED_HOST: $sched_host @@ -88,6 +105,18 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MONITOR_HOST: $MONITOR_HOST +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

2 2

[PATCH v4 compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..f96972e 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,6 +34,22 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=20002} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MONITOR_HOST:=$server_ip} +: ${MONITOR_PORT:=20001} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -53,6 +69,8 @@ EOF mkdir -p /etc/compass-ci/service cat > /etc/compass-ci/service/service-env.yaml <<EOF +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT ES_HOST: $ES_HOST ES_PORT: $ES_PORT SCHED_HOST: $sched_host @@ -88,6 +106,19 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MONITOR_HOST: $MONITOR_HOST +MONITOR_PORT: $MONITOR_PORT +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor.rb: update SRV_HTTP_RESULT_PORT
by Liu Yinsi 25 Mar '21

25 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/email.sh | 2 +- lib/monitor.rb | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/lib/email.sh b/lib/email.sh index 63d5a51e2..bca4ab435 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -8,7 +8,7 @@ Subject: [NOTIFY Compass-CI] Test job failed: $id Dear $my_name: Sorry to inform you that your test job is failed, you can click the following link to view details. - http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT:-11300}$result_root + http://api.compass-ci.openeuler.org:${SRV_HTTP_RESULT_PORT:-20007}$result_r… Regards Compass-CI diff --git a/lib/monitor.rb b/lib/monitor.rb index f14e131e3..bf7ef69ba 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -100,9 +100,9 @@ class Monitor @result_roots.each do |res| res.to_s.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' - srv_http_port = job['SRV_HTTP_PORT'] || '11300' - url = "http://#{srv_http_host}:#{srv_http_port}#{res}" + srv_http_result_host = job['SRV_HTTP_RESULT_HOST'] || 'api.compass-ci.openeuler.org' + srv_http_result_port = job['SRV_HTTP_RESULT_PORT'] || '20007' + url = "http://#{srv_http_result_host}:#{srv_http_result_port}#{res}" system "lftp -c mirror #{url} >/dev/null 2>&1" end end -- 2.23.0

1 0

help star
by Wu Fengguang 25 Mar '21

25 Mar '21

Yu Chuan, Wu Zhende is our help star again, thanks Yu Chuan, Wu Zhende! wfg@crystal ~/todo/bin% ./help-star ../people/*/credit/2021-03-25.md 15.2 Yu Chuan 15.1 Wu Zhende 11.2 Cao Xueliang 11.0 Wei Jihui 8.8 Liu Yinsi 7.4 Luan Shengde 7.0 Wang Yong 6.4 Lu Weitao 5.6 Wu Fengguang 5.4 Wang Chenglong 5.1 Zhang Dewan 5.0 Li Yuanchao 4.2 Liu Shaofei 4.1 Lin Jiaxin 2.9 Li Ping 1.8 Du Kaitian 1.6 Xu Xijian 1.2 Bai Jing 0.9 Cui Yili 0.0 Zhang Yale

1 0

git count
by Wu Fengguang 25 Mar '21

25 Mar '21

Wei Jihui is our code star, thanks Wei Jihui! wfg@crystal ~/todo/bin% ./git-count 6 Wei Jihui 5 Li Yuanchao 4 Wu Zhende 4 Yu Chuan 4 Luan Shengde 4 Lin Jiaxin 3 Cao Xueliang 3 Liu Shaofei 3 Liu Yinsi 3 Li Ping 2 Bai Jing 2 Xu Xijian 2 Lu Weitao 2 Cui Yili 2 Wang Chenglong 2 Wang Yong 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale 0 Zhang Dewan

1 0

review star
by Wu Fengguang 25 Mar '21

25 Mar '21

Luan Shengde is our review star, thanks Luan Shengde! wfg@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 9 4 19 9 Wu Fengguang 19 9 18 7 Luan Shengde 49 6 6 6 Wei Jihui 21 5 5 6 Liu Shaofei 39 2 2 5 Cao Xueliang 14 4 4 3 Lin Jiaxin 22 3 5 2 Li Yuanchao 3 0 0 2 Xu Xijian 8 3 5 2 Bai Jing 20 2 2 1 Wang Chenglong 17 3 5 1 Wang Yong 9 1 0 1 Lu Weitao 7 2 3 1 Cui Yili 3 0 0 0 Zhang Dewan 4 1 1 0 Yu Chuan 14 0 0 0 Wu Zhende 11 1 1 0 Li Ping 9 0 0 0 Liu Yinsi

1 0

[PATCH v5 compass-ci] container/web-backend: add API /get_table_error
by Lu Weitao 25 Mar '21

25 Mar '21

[Example] curl -X GET 'localhost:10002/get_job_error?group_id=wcl_ansible-openeuler-03-10' return: { "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, "attributes":["job_id","error_id","error_message","link_to_result"], "objects":[ { "job_id": "crystal.1354921", "error_id": "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-xxx.fail", "error_message": "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", "link_to_result": "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…", ate/crystal.1354921" }, { "job_id": "crystal.1354929", ... }, ... ] } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/web-backend/web-backend | 18 +++++++++ src/lib/web_backend.rb | 65 +++++++++++++++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index adb91e0..c7d1ec6 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -170,4 +170,22 @@ end # } get '/get_jobs_summary' do group_jobs_stats(params) + +# GET /get_job_error?suite=virttest&tbox_group=vm-2p8g +# Response +# { +# "filter": {"group_id":"wcl_ansible-openeuler-03-10"}, +# "attributes": ["job_id", "error_id", "error_message", "link_to_result"], +# "objects": [ +# { +# "job_id": "crystal.1354921", +# "error_id": "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-failed.fail", +# "error_message": "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}", +# "link_to_result": "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/opene…" +# }, +# ... +# ] +# } +get '/get_job_error' do + get_job_error(params) end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 40471e0..7f26b9c 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/params_group.rb" require "#{CCI_SRC}/lib/compare_data_format.rb" UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -613,4 +614,68 @@ def group_jobs_stats(params) return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'group jobs table error'] end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] + +# ------------------------------------------------------------------------------------------- +# job error table like: +# job_id error_id error_message link to result +# ------------------------------------------------------------------------------------- +# crystal.630608 "stderr.xxx" "messag:xxxx" https://$host:$port/$result_root +# ... +# ------------------------------------------------------------------------------------------- + +def get_job_error(params) + begin + body = job_error_body(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get error table error'] + end + + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end + +def job_error_body(params) + error_objects = get_error_objects(params) + { + filter: params, + attributes: ['job_id', 'error_id', 'error_message', 'link_to_result'], + objects: error_objects, + }.to_json +end + +def get_error_objects(filter_items) + error_objs = [] + + job_list = get_job_list(filter_items) + job_list.each do |job| + error_obj = get_error_from_job(job) + error_objs << error_obj unless error_obj.empty? + end + + error_objs +end + +def get_job_list(items) + es = ESQuery.new(ES_HOST, ES_PORT) + query_results = es.multi_field_query(items) + + extract_jobs_list(query_results['hits']['hits']) +end + +# get all error_id from one job +def get_error_from_job(job) + job_error_obj = {} + job['stats'].each do |metric, value| + next unless metric.end_with?('.message') + + result_host = ENV['SRV_HTTP_RESULT_HOST'] || SRV_HTTP_RESULT_HOST + result_port = ENV['SRV_HTTP_RESULT_PORT'] || SRV_HTTP_RESULT_PORT + error_id = metric.sub('.message', '.fail') + job_error_obj['id'] = job['id'] + job_error_obj['error_id'] = error_id + job_error_obj['error_message'] = value + job_error_obj['link_to_result'] = value + end + + job_error_obj end -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code/dev-env: set vars defaults values
by Liu Yinsi 25 Mar '21

25 Mar '21

[why] add vars and defaults values is easy to management. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index f16cf80..b2c1814 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -34,12 +34,33 @@ server_name=$(hostname | cut -f1 -d.) : ${DOCKER_REGISTRY_PORT:=5001} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${GIT_SERVER:=$server_ip} +: ${SCHED_HTTPS_HOST:=$server_ip} +: ${SCHED_HTTPS_PORT:=11306} +: ${DOCKER_PROXY_HOST:=$server_ip} +: ${DOCKER_PROXY_PORT:=3128} +: ${LKP_CGI_PORT:=3000} +: ${JUMPER_HOST:=$server_ip} +: ${JUMPER_PORT:=10000} +: ${MAIL_HOST:=$server_ip} +: ${MAIL_PORT:=10001} +: ${MONITOR_HOST:=$server_ip} +: ${FETCH_MAIL_DIR:=/srv/cci/Maildir} +: ${FETCH_MAIL_BOX:=.compass-ci} +: ${GIT_DAEMON_HOST:=$server_ip} +: ${GIT_MIRROR_HOST:=$server_ip} +: ${REDIS_PORT:=6379} +: ${MAILDIR:=/srv/cci/Maildir/.compass-ci} +: ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} + mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port lab: $lab +SCHED_HTTPS_HOST: $SCHED_HTTPS_HOST +SCHED_HTTPS_PORT: $SCHED_HTTPS_PORT EOF [[ -f /etc/compass-ci/defaults/upstream-config ]] || { @@ -88,6 +109,21 @@ DOCKER_REGISTRY_PORT: $DOCKER_REGISTRY_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT MAILDIR: /srv/cci/Maildir/.compass-ci LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS +DOCKER_PROXY_HOST: $DOCKER_PROXY_HOST +DOCKER_PROXY_PORT: $DOCKER_PROXY_PORT +LKP_CGI_PORT: $LKP_CGI_PORT +JUMPER_HOST: $JUMPER_HOST +JUMPER_PORT: $JUMPER_PORT +MAIL_HOST: $MAIL_HOST +MAIL_PORT: $MAIL_PORT +MONITOR_HOST: $MONITOR_HOST +FETCH_MAIL_DIR: $FETCH_MAIL_DIR +FETCH_MAIL_BOX: $FETCH_MAIL_BOX +GIT_DAEMON_HOST: $GIT_DAEMON_HOST +GIT_MIRROR_HOST: $GIT_MIRROR_HOST +REDIS_PORT: $REDIS_PORT +MAILDIR: $MAILDIR +SENT_MAILDIR: $SENT_MAILDIR EOF mkdir -p /etc/compass-ci/scheduler -- 2.23.0

2 2