[Compass-ci] Re: [PATCH v3 compass-ci] container/web-backend: add API /get_table_error

On Wed, Mar 17, 2021 at 05:01:53PM +0800, Lu Weitao wrote:

[Example] curl -X GET 'localhost:30000/get_job_error?group_id=wcl_ansible-openeuler-03-10' return: { "filter":{"group_id":"wcl_ansible-openeuler-03-10"},

That's good to have.

"table_fields":["job_id","error_id","error_message","link to result"], "data_table":[

That 2 fields look plain mapping to table header and data lines. However may not be flexible enough for scripts (javascript, ruby clients) to handle.

Imagine the client script need embed the link into another field, what should it do? Hard code this?

data_line[3]

A better data exchange model could be

attributes: [job_id, error_id, error_message, link_to_result] objects: - { job_id: crystal.1354921, error_id: ..., error_message: ..., link_to_result: ... } - { job_id: crystal.1354929, error_id: ..., error_message: ..., link_to_result: ... }

It could be a bit more bloated, however allows the client to use

object['link_to_result']

More importantly, backend API in general should focus on passing k/v pairs (data), not table columns (final representation).

It'll allow the client to do more creative logic and evolve over time. For example, - some k/v pairs can be used as mouse hover tips - link_to_result could be embed into another data column

  [
    "crystal.1354921",
    "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-xxx.fail",
    "{\"changed\": false, \"msg\": \"Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.}",
    "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/openeuler-20.03-aarch64/yogeshraheja-ansible-httpdtempl
    ate/crystal.1354921"
  ],

Thanks, Fengguang

  [
    "crystal.1354929",
    ...
  ]
  ...
]

}

Signed-off-by: Lu Weitao luweitaobe@163.com

container/web-backend/web-backend | 19 +++++++++ src/lib/web_backend.rb | 68 +++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+)

diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 33a3fae..284e7a3 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -146,3 +146,22 @@ end get '/get_repo_statistics' do new_refs_statistics(params) end

+# GET /get_job_error?suite=virttest&tbox_group=vm-2p8g +# Response +# { +# "filter":{"group_id":"wcl_ansible-openeuler-03-10"}, +# "table_fields":["job_id","error_id","error_message","link to result"], +# "data_table":[ +# [ +# "crystal.1354921", +# "ansible_test.error.Unable-to-start-service-httpd-Job-for-httpdservice-failed.fail", +# "{"changed": false, "msg": "Unable to start service httpd: Job for httpd.service failed because a timeout was exceeded.} +# "http://172.17.0.1:11300/result/ansible_test/2021-03-10/vm-2p16g--wcl1/openeu..." +# ], +# ... +# ] +# } +get '/get_job_error' do

• get_job_error(params)

+end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 89c0f3c..48b7ef2 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -12,6 +12,7 @@ require "#{CCI_SRC}/lib/compare.rb" require "#{CCI_SRC}/lib/constants.rb" require "#{CCI_SRC}/lib/es_query.rb" require "#{CCI_SRC}/lib/matrix2.rb" +require "#{CCI_SRC}/lib/params_group.rb" require "#{CCI_SRC}/lib/compare_data_format.rb"

UPSTREAM_REPOS_PATH = ENV['UPSTREAM_REPOS_PATH'] || '/c/upstream-repos' @@ -548,3 +549,70 @@ def new_refs_statistics(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end

+# ------------------------------------------------------------------------------------------- +# job error table like: +# job_id error_id error_message link to result +# ------------------------------------------------------------------------------------- +# crystal.630608 "stderr.xxx" "messag:xxxx" https://$host:$port/$result_root +# ... +# -------------------------------------------------------------------------------------------

+def get_job_error(params)

• begin
• body = job_error_body(params)
• rescue StandardError => e
• warn e.message
• return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get error table error']
• end
• [200, headers.merge('Access-Control-Allow-Origin' => '*'), body]

+end

+def job_error_body(params)

• error_table = get_error_table(params)
• {
• filter: params,
• table_fields: ["job_id", "error_id", "error_message", "link to result"],
• data_table: error_table,
• }.to_json

+end

+def get_error_table(filter_items)

• error_table = []
• job_list = get_job_list(filter_items)
• job_list.each do |job|
• error_table += get_error_from_job(job)
• end
• error_table

+end

+def get_job_list(items)

• es = ESQuery.new(ES_HOST, ES_PORT)
• query_results = es.multi_field_query(items)
• extract_jobs_list(query_results['hits']['hits'])

+end

+# get all error_id from one job +def get_error_from_job(job)

• job_error = []
• job['stats'].each do |metric, value|
• next unless metric.end_with?('.message')
• result_host = ENV['SRV_HTTP_RESULT_HOST'] || SRV_HTTP_RESULT_HOST
• result_port = ENV['SRV_HTTP_RESULT_PORT'] || SRV_HTTP_RESULT_PORT
• error_id = metric.sub('.message', '.fail')
• job_error << [

•  job['id'],
  

•  error_id,
  

•  value,
  

•  "http://#{result_host}:#{result_port}#{job['result_root']}"
  

• ]
• end
• job_error

+end

2.23.0