December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] daemon/multi-docker: add default value of queues
by Hu Xuejiao 21 Dec '20

21 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- daemon/multi-docker | 2 ++ 1 file changed, 2 insertions(+) diff --git a/daemon/multi-docker b/daemon/multi-docker index 257645b8..7759fdf3 100755 --- a/daemon/multi-docker +++ b/daemon/multi-docker @@ -6,6 +6,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +: ${queues:="$tbox_group.$HOSTNAME,$tbox_group.$(arch)"} + multi_docker() { export CCI_SRC=${CCI_SRC:-"/c/compass-ci"} -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] lib: use suites_hash instead of suites_list to keep consistent
by Lu Kaiyi 21 Dec '20

21 Dec '20

[why] suite_list maybe not consistent with group key if sort group matrixes. [how] use suites_hash instead of suites_list to keep correct congruent relationship with same group_key. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 10 +++++----- lib/matrix2.rb | 6 +++--- 3 files changed, 10 insertions(+), 10 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index da761ef..9090c3b 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -62,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices, suites_list = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, suites_list, options) + groups_matrices, suites_hash = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_hash, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 0727797..6e4c2aa 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -387,25 +387,25 @@ end # input: group matrices # output: pre compare result of each group # the result with more comparison objects first -def compare_group_matrices(group_matrices, suites_list, options) +def compare_group_matrices(group_matrices, suites_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) group_matrices_array.each do |matrice_kv| - result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_list, options) + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) end result_str end -def get_matrix_str(matrice_key, matrice_value, suites_list, options) +def get_matrix_str(matrice_key, matrice_value, suite_list, options) m_list = [] m_titles = [] matrice_value.each do |dim, matrix| m_titles << dim m_list << matrix end - return compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) if options[:no_print] + return compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) if options[:no_print] - print compare_matrixes(m_list, suites_list.shift, m_titles, matrice_key, options: options) + print compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) return '' end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 4393a65..dfc606e 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -121,7 +121,7 @@ end # ... # } def combine_group_query_data(job_list, dims) - suites_list = [] + suites_hash = {} groups = auto_group(job_list, dims) groups.each do |group_key, value| if value.empty? @@ -133,10 +133,10 @@ def combine_group_query_data(job_list, dims) groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) end - suites_list << suite_list + suites_hash[group_key] = suite_list end - return groups, suites_list + return groups, suites_hash end # input: -- 2.23.0

1 0

[PATCH v4 compass-ci 2/3] lib/compare_matrixes.rb: don't remove the field that start with $suite
by Lu Kaiyi 21 Dec '20

21 Dec '20

if the metric field start with $suite, even if the field is unchanged, still keep it in compare result. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index b7ea5a4..0727797 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -176,11 +176,15 @@ def get_matrixes_values(matrixes_list, options) matrixes_values end -def remove_unchanged_field(matrixes_values) +def remove_unchanged_field(matrixes_values, suite_list) # remove unchanged field from matrixes values and remove :changed key # matrixes_values.each_key do |success| matrixes_values[success].delete_if do |field| + if suite_list.any? { |suite| field.start_with?(suite) } + matrixes_values[success][field][:changed] = true + end + !matrixes_values[success][field].delete(:changed) end end @@ -203,7 +207,7 @@ def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key options = { 'perf-profile': 5, theme: :none, no_print: false }.merge(options) matrixes_values = get_matrixes_values(matrixes_list, options) - remove_unchanged_field(matrixes_values) if matrixes_list.length > 1 + remove_unchanged_field(matrixes_values, suite_list) if matrixes_list.length > 1 no_print = options[:no_print] result_str = group_key ? "\n\n\n\n\n" + group_key : '' result_str += get_all_result_str( -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 21 Dec '20

21 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index bff4eee..4393a65 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -82,10 +82,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| next if key.include?('timestamp') -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: move the service logic to the Sched class
by Wu Zhende 21 Dec '20

21 Dec '20

Keep scheduler.cr neat and do not perform operations related to env. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 40 +++++++++++++++++++++++++++------- src/scheduler/find_job_boot.cr | 2 +- src/scheduler/scheduler.cr | 32 ++++----------------------- 3 files changed, 37 insertions(+), 37 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index a1ef28e..f7220f9 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -51,20 +51,44 @@ class Sched mac.gsub(":", "-") end - def set_host_mac(mac : String, hostname : String) - @redis.hash_set("sched/mac2host", normalize_mac(mac), hostname) + def set_host_mac + if (hostname = @env.params.query["hostname"]?) && (mac = @env.params.query["mac"]?) + @redis.hash_set("sched/mac2host", normalize_mac(mac), hostname) + + "Done" + else + "No yet!" + end end - def del_host_mac(mac : String) - @redis.hash_del("sched/mac2host", normalize_mac(mac)) + def del_host_mac + if mac = @env.params.query["mac"]? + @redis.hash_del("sched/mac2host", normalize_mac(mac)) + + "Done" + else + "No yet!" + end end - def set_host2queues(hostname : String, queues : String) - @redis.hash_set("sched/host2queues", hostname, queues) + def set_host2queues + if (queues = @env.params.query["queues"]?) && (hostname = @env.params.query["host"]?) + @redis.hash_set("sched/host2queues", hostname, queues) + + "Done" + else + "No yet!" + end end - def del_host2queues(hostname : String) - @redis.hash_del("sched/host2queues", hostname) + def del_host2queues + if hostname = @env.params.query["host"]? + @redis.hash_del("sched/host2queues", hostname) + + "Done" + else + "No yet!" + end end def update_tbox_wtmp diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 7565375..9b3db88 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -29,7 +29,7 @@ class Sched # then hostname will be "sut-00-01-02-03-04" !!! def submit_host_info_job(mac) host = "sut-#{mac}" - set_host_mac(mac, host) + @redis.hash_set("sched/mac2host", normalize_mac(mac), host) Jobfile::Operate.auto_submit_job( "#{ENV["LKP_SRC"]}/jobs/host-info.yaml", diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 7ec149c..2103f7c 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -79,46 +79,22 @@ module Scheduler # # curl -X PUT "http://localhost:3000/set_host_mac?hostname=wfg&mac=00-01-02-03-04-05" put "/set_host_mac" do |env| - if (client_hostname = env.params.query["hostname"]?) && (client_mac = env.params.query["mac"]?) - env.sched.set_host_mac(client_mac, client_hostname) - - "Done" - else - "No yet!" - end + env.sched.set_host_mac end # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" put "/set_host2queues" do |env| - if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) - env.sched.set_host2queues(client_host, client_queues) - - "Done" - else - "No yet" - end + env.sched.set_host2queues end # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| - if client_mac = env.params.query["mac"]? - env.sched.del_host_mac(client_mac) - - "Done" - else - "No yet!" - end + env.sched.del_host_mac end # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" put "/del_host2queues" do |env| - if client_host = env.params.query["host"]? - env.sched.del_host2queues(client_host) - - "Done" - else - "No yet!" - end + env.sched.del_host2queues end # client(runner) report job's status -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: check account info
by Wu Zhende 21 Dec '20

21 Dec '20

If submit yaml in /srv/result, will not check account info. Fix it Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 4d0b5df..e6f7e6a 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -51,17 +51,23 @@ class Job def initialize(job_content : JSON::Any, id) @hash = job_content.as_h + @es = Elasticsearch::Client.new + @account_info = Hash(String, JSON::Any).new # init job with "-1", or use the original job_content["id"] id = "-1" if "#{id}" == "" if initialized? - return if @hash["id"] == "#{id}" + if @hash["id"] == "#{id}" + return unless @hash.has_key?("my_uuid") || @hash.has_key?("my_token") + + check_account_info() + return + end end - @es = Elasticsearch::Client.new @hash["id"] = JSON::Any.new("#{id}") - @account_info = Hash(String, JSON::Any).new + check_required_keys() check_account_info() set_defaults() -- 2.23.0

1 0

[PATCH compass-ci 4/4] web_backend.rb: optimize the way of query data for compare
by Li Yuanchao 21 Dec '20

21 Dec '20

Before: When do compare with dimension, first query 100(default size) jobs by filters, then group these jobs by dimension. But if one kind of job is done too many, other kinds of jobs may can not be got. After: Fisrt query how many kinds of dimensions in es, take dimension = 'os' as an example, the result can be ["openeuler", "archlinux", "centos", "debian", "kylin", "archLinux"] Then query 100 jobs by each os, then group all these jobs. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/compare.rb | 2 +- lib/matrix2.rb | 3 +- src/lib/web_backend.rb | 64 ++++++++++++++++++++++++++++-------------- 3 files changed, 45 insertions(+), 24 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index db264f8..da761ef 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -69,7 +69,7 @@ end def create_groups_matrices_list(conditions, dims) es = ESQuery.new(ES_HOST, ES_PORT) query_results = es.multi_field_query(conditions) - combine_group_query_data(query_results, dims) + combine_group_query_data(query_results['hits']['hits'], dims) end # ------------------------------------------------------------------------------------------- diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2507b77..b6cfa79 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -109,9 +109,8 @@ end # group2_key => {...} # ... # } -def combine_group_query_data(query_data, dims) +def combine_group_query_data(job_list, dims) suites_list = [] - job_list = query_data['hits']['hits'] groups = auto_group(job_list, dims) groups.each do |group_key, value| if value.empty? diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index c4182a3..a680b8a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -34,7 +34,7 @@ ALL_FIELDS = FIELDS + NOT_SHOW_FIELDS NOT_NEED_EXIST_FIELDS = %w[error_ids upstream_repo].freeze PREFIX_SEARCH_FIELDS = ['tbox_group'].freeze ES_CLIENT = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") -COMPARE_RECORDS_NUMBER = 50 +COMPARE_RECORDS_NUMBER = 100 def es_query(query) ES_CLIENT.search index: 'jobs*', body: query @@ -148,30 +148,21 @@ def get_es_must(params) must end -def do_get_groups_matrices(query, dimension, total, size, from) - result = es_query(query) - matrices, suites_list = combine_group_query_data(result, dimension) - while matrices.empty? - from += size - break if from > total - - query[:from] = from - result = es_query(query) - matrices, suites_list = combine_group_query_data(result, dimension) - end - [matrices, suites_list] +def get_dimension_list(dimension) + query = { size: 0, aggs: { dims: { terms: { size: 10000, field: dimension[0] } } } } + buckets = es_query(query)['aggregations']['dims']['buckets'] + dimension_list = [] + buckets.each { |dims_agg| dimension_list << dims_agg['key'] } + return dimension_list end -def get_groups_matrices(conditions, dimension, must, size, from) - must += build_multi_field_subquery_body(conditions) - count_query = { query: { bool: { must: must } } } - total = es_count(count_query) - return {} if total < 1 - +def query_dimension(dim_field, dim_value, must, size, from) + must_dim = Array.new(must) + must_dim << { term: { dim_field => dim_value } } query = { query: { bool: { - must: must + must: must_dim } }, size: size, @@ -180,8 +171,39 @@ def get_groups_matrices(conditions, dimension, must, size, from) start_time: { order: 'desc' } }] } + es_query(query)['hits']['hits'] +end + +def get_dimension_job_list(dimension, must, size, from) + dimension_list = get_dimension_list(dimension) + job_list = [] + dimension_list.each do |dim| + job_list += query_dimension(dimension[0], dim, must, size, from) + end + job_list +end + +def do_get_groups_matrices(must, dimension, total, size, from) + job_list = get_dimension_job_list(dimension, must, size, from) + + matrices, suites_list = combine_group_query_data(job_list, dimension) + while matrices.empty? + from += size + break if from > total + + job_list = get_dimension_job_list(dimension, must, size, from) + matrices, suites_list = combine_group_query_data(job_list, dimension) + end + [matrices, suites_list] +end + +def get_groups_matrices(conditions, dimension, must, size, from) + must += build_multi_field_subquery_body(conditions) + count_query = { query: { bool: { must: must } } } + total = es_count(count_query) + return {} if total < 1 - do_get_groups_matrices(query, dimension, total, size, from) + do_get_groups_matrices(must, dimension, total, size, from) end def get_compare_body(params) -- 2.23.0

2 1

[PATCH v2 lkp-tests 2/4] lib/env.sh: move functions to env.sh
by Wei Jihui 21 Dec '20

21 Dec '20

move functions:write_shell_profile and write_host to lib/env.sh Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/env.sh | 30 ++++++++++++++++++++++++++++++ sbin/set-env.sh | 39 --------------------------------------- 2 files changed, 30 insertions(+), 39 deletions(-) delete mode 100755 sbin/set-env.sh diff --git a/lib/env.sh b/lib/env.sh index 8ac42a83..9f37d8cc 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -125,3 +125,33 @@ shell_profile() [ -f "$profile" ] || touch $profile echo $profile } + +write_shell_profile() +{ + shell_profile_file=$(shell_profile) + if [ $# -gt 0 ]; then + echo "$@" >> $shell_profile_file + fi + + source $shell_profile_file +} + +write_host() +{ + if is_system "Linux"; then + nr_cpu=$(nproc) + memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') + else + nr_cpu=$(sysctl -n hw.logicalcpu) + memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') + fi + memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') + memory=$(echo $memory_new | awk '{print int($0)+1}')G + + cat > hosts/$(hostname) <<-EOF + nr_cpu: $nr_cpu + memory: $memory + hdd_partitions: + ssd_partitions: + EOF +} diff --git a/sbin/set-env.sh b/sbin/set-env.sh deleted file mode 100755 index cc7f9050..00000000 --- a/sbin/set-env.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/bin/sh - -. $LKP_SRC/lib/env.sh - -write_shell_profile() -{ - shell_profile_file=$(shell_profile) - if [ $# -gt 0 ]; then - echo $@ >> $shell_profile_file - else - echo "export LKP_SRC=$PWD" >> $shell_profile_file - echo "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" >> $shell_profile_file - fi - - source $shell_profile_file -} - -write_host() -{ - if is_system "Linux"; then - nr_cpu=$(nproc) - memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') - else - nr_cpu=$(sysctl -n hw.logicalcpu) - memory_total=$(top -l 1 | grep MemRegions | awk '{print $2}') - fi - memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024}') - memory=$(echo $memory_new | awk '{print int($0)+1}')G - - cat > hosts/$(hostname) <<-EOF - nr_cpu: $nr_cpu - memory: $memory - hdd_partitions: - ssd_partitions: - EOF -} - -write_shell_profile -write_host -- 2.23.0

2 1

[PATCH compass-ci] container/delimiter: install util-linux, terminal-table, mail
by Cao Xueliang 21 Dec '20

21 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/Dockerfile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/delimiter/Dockerfile b/container/delimiter/Dockerfile index 3db5890..93c52c6 100644 --- a/container/delimiter/Dockerfile +++ b/container/delimiter/Dockerfile @@ -6,8 +6,8 @@ FROM alpine:3.11 RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ apk add --no-cache 'ruby-dev' \ - 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' RUN umask 002 && \ echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ - gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket + gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket terminal-table mail -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] lib/matrix2.rb: discard the no $suite related job
by Lu Kaiyi 21 Dec '20

21 Dec '20

if job has suite, but no $suite related stats metric, then discard this job when create matrix. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/matrix2.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 2769f7c..ef5d935 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -82,10 +82,15 @@ def create_matrix(job_list) matrix = {} suites = [] job_list.each do |job| - suites << job['suite'] if job['suite'] stats = job['stats'] next unless stats + if job['suite'] + next unless stats.keys.any? { |stat| stat.start_with?(job['suite']) } + + suites << job['suite'] + end + stats.each do |key, value| next if key.include?('timestamp') -- 2.23.0

2 1