- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 06 Jan '21

06 Jan '21

timing work: reset email rate limit queues everyday Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..099d89d --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday +# the email counts queues will be reset everyday +# to enable to send mail in the new day to user +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v3 compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 06 Jan '21

06 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH v3 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 06 Jan '21

06 Jan '21

[why] the service need to install redis to store email counts to redis queue the service need to install rufus-scheduler to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 0

[PATCH compass-ci 2/2] compare: remove not latest failed metrics
by Lu Weitao 06 Jan '21

06 Jan '21

[Why] User just need the failed metrics belong latest job in compare result [How] 1. query job_list and order by start_time desc 2. auto group job_list - the first job of each dimension is the latest job { group_key => { dim1 => [job1, ...] ... } } 3. get latest job_list for each group 4. in compare result, ignore the failed metric that not belongs the latest job_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 4 ++-- lib/compare_matrixes.rb | 32 ++++++++++++++++++++++++-------- lib/matrix2.rb | 24 +++++++++++++++++------- 3 files changed, 43 insertions(+), 17 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 6a2f515..21e24df 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -62,8 +62,8 @@ end def compare_group(argv, dimensions, options) conditions = parse_conditions(argv) dims = dimensions.split(' ') - groups_matrices, suites_hash = create_groups_matrices_list(conditions, dims) - compare_group_matrices(groups_matrices, suites_hash, options) + groups_matrices, suites_hash, latest_jobs_hash = create_groups_matrices_list(conditions, dims) + compare_group_matrices(groups_matrices, suites_hash, latest_jobs_hash, options) end def create_groups_matrices_list(conditions, dims) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 21bd5c9..7747ea1 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -159,7 +159,7 @@ def get_matrixes_fields(matrixes_list) matrixes_fields end -def get_matrixes_values(matrixes_list, options) +def get_matrixes_values(matrixes_list, latest_jobs, options) # get all matrixes all field values # matrixes_values = { false => {}, true => {} } @@ -168,6 +168,10 @@ def get_matrixes_values(matrixes_list, options) next if field == 'stats_source' success = successful?(field) + unless success + next unless latest_failure?(field, latest_jobs) + end + matrixes_values[success][field] = get_values_by_field( matrixes_list, field, matrixes_size, success, options @@ -176,6 +180,18 @@ def get_matrixes_values(matrixes_list, options) matrixes_values end +def latest_failure?(field, latest_jobs) + flag = false + latest_jobs.each do |job| + if job['stats'][field] + flag = true + break + end + end + + flag +end + def remove_unchanged_field(matrixes_values, suite_list) # remove unchanged field from matrixes values and remove :changed key # @@ -197,7 +213,7 @@ def matrixes_empty?(matrixes_list) return matrixes_list.any?(&:empty?) end -def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key = nil, options: {}) +def compare_matrixes(matrixes_list, suite_list, latest_jobs, matrixes_titles = nil, group_key = nil, options: {}) # compare matrix in matrixes_list and print info # @matrixes_list: list consisting of matrix # @matrixes_titles: number or dimension of matrix @@ -206,7 +222,7 @@ def compare_matrixes(matrixes_list, suite_list, matrixes_titles = nil, group_key return warn('Matrix cannot be empty!') || '' if matrixes_empty?(matrixes_list) options = { 'perf-profile': 5, theme: :none, no_print: false }.merge(options) - matrixes_values = get_matrixes_values(matrixes_list, options) + matrixes_values = get_matrixes_values(matrixes_list, latest_jobs, options) remove_unchanged_field(matrixes_values, suite_list) if matrixes_list.length > 1 no_print = options[:no_print] result_str = group_key ? "\n\n\n\n\n" + group_key : '' @@ -387,28 +403,28 @@ end # input: group matrices # output: pre compare result of each group # the result with more comparison objects first -def compare_group_matrices(group_matrices, suites_hash, options) +def compare_group_matrices(group_matrices, suites_hash, latest_jobs_hash, options) result_str = '' group_matrices_array = sort_by_matrix_size(group_matrices) have_multi_member = group_matrices_array[0][1].size > 1 group_matrices_array.each do |matrice_kv| next if have_multi_member && matrice_kv[1].size < 2 - result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], options) + result_str += get_matrix_str(matrice_kv[0], matrice_kv[1], suites_hash[matrice_kv[0]], latest_jobs_hash[matrice_kv[0]], options) end result_str end -def get_matrix_str(matrice_key, matrice_value, suite_list, options) +def get_matrix_str(matrice_key, matrice_value, suite_list, latest_jobs, options) m_list = [] m_titles = [] matrice_value.each do |dim, matrix| m_titles << dim m_list << matrix end - return compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) if options[:no_print] + return compare_matrixes(m_list, suite_list, latest_jobs, m_titles, matrice_key, options: options) if options[:no_print] - print compare_matrixes(m_list, suite_list, m_titles, matrice_key, options: options) + print compare_matrixes(m_list, suite_list, latest_jobs, m_titles, matrice_key, options: options) return '' end diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 09573ec..e3ae580 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -115,27 +115,37 @@ end # } def combine_group_query_data(job_list, dims) suites_hash = {} + latest_jobs_hash = {} groups = auto_group(job_list, dims) have_multi_member = multi_member?(groups) one_size_count = 0 groups.each do |group_key, value| if value.size < 2 one_size_count += 1 - if have_multi_member || one_size_count > 3 - groups.delete(group_key) - next - end + next if delete_group?(groups, group_key, have_multi_member, one_size_count) end - get_groups_matrix(groups, group_key, value, suites_hash) + get_groups_matrix(groups, group_key, value, suites_hash, latest_jobs_hash) end - return groups, suites_hash + return groups, suites_hash, latest_jobs_hash end -def get_groups_matrix(groups, group_key, value, suites_hash) +def delete_group?(groups, group_key, have_multi_member, one_size_count) + was_deleted = false + if have_multi_member || one_size_count > 3 + groups.delete(group_key) + was_deleted = true + end + + was_deleted +end + +def get_groups_matrix(groups, group_key, value, suites_hash, latest_jobs_hash) suite_list = [] + latest_jobs_hash[group_key] = [] value.each do |dimension_key, jobs| groups[group_key][dimension_key], suites = create_matrix(jobs) suite_list.concat(suites) + latest_jobs_hash[group_key] << jobs[0] end suites_hash[group_key] = suite_list end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/compare.rb: order query result with desc
by Lu Weitao 06 Jan '21

06 Jan '21

[Why] 1. When query result include more than 10,000 jobs, we still need the latest jobs use to compare 2. Support we recognize the latest job that have failure messages from each group-dimension-job_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/compare.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/compare.rb b/lib/compare.rb index 9090c3b..6a2f515 100644 --- a/lib/compare.rb +++ b/lib/compare.rb @@ -40,7 +40,7 @@ def create_matrices_list(conditions) suite_list = [] es = ESQuery.new(ES_HOST, ES_PORT) conditions.each do |condition| - query_results = es.multi_field_query(condition) + query_results = es.multi_field_query(condition, desc_keyword: 'start_time') matrix, suites = combine_query_data(query_results) matrices_list << matrix suite_list.concat(suites) @@ -68,7 +68,7 @@ end def create_groups_matrices_list(conditions, dims) es = ESQuery.new(ES_HOST, ES_PORT) - query_results = es.multi_field_query(conditions) + query_results = es.multi_field_query(conditions, desc_keyword: 'start_time') combine_group_query_data(query_results['hits']['hits'], dims) end -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: git clone only one commit at first
by Li Yuanchao 05 Jan '21

05 Jan '21

Some repositories have many commits which make it big, and cause overtime when git clone. Clone only one commit at first, and pull all commits later, can help clone large repositories. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 156d5bb..7f265d9 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -42,7 +42,7 @@ class GitMirror ret = false url = get_url(Array(url)[0]) 10.times do - stderr = %x(git clone --mirror #{url} #{mirror_dir} 2>&1) + stderr = %x(git clone --mirror --depth 1 #{url} #{mirror_dir} 2>&1) ret = !stderr.include?('fatal') break if ret end @@ -51,6 +51,11 @@ class GitMirror end def git_fetch(mirror_dir) + if File.exist?("#{mirror_dir}/shallow") + %x(git -C #{mirror_dir} fetch --unshallow 2>&1) + return false + end + fetch_info = %x(git -C #{mirror_dir} fetch 2>&1) # Check whether mirror_dir is a good git repository by 2 conditions. If not, delete it. if fetch_info.include?(ERR_MESSAGE) && Dir.empty?(mirror_dir) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin/run_job: registering user when submit-job
by Hu Xuejiao 05 Jan '21

05 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/run_job b/sbin/run_job index c84df0d..325fa9f 100755 --- a/sbin/run_job +++ b/sbin/run_job @@ -1,5 +1,8 @@ #!/bin/bash +# register user information +ruby build-my-info.rb -e "$my_email" -n "$USER" -l "$lab" + cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file -- 2.23.0

1 0

[PATCH v4 compass-ci] docker-rootfs: add README.md
by Wang Chenglong 05 Jan '21

05 Jan '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..e7b5084 --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,18 @@ +# Use the tool to create a new nfsroot for os + +## Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs. + +## Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + # Please ensure $HOME/.config/compass-ci/rootfs-passwd exists. + +## Some configuration items: + ./packages-to-install + # If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + # Set the password for the image into this file. -- 2.23.0

1 0

[PATCH v2 compass-ci] docker-rootfs/run: fix the incorrect image names
by Wang Chenglong 05 Jan '21

05 Jan '21

[Why] When docker's image name contains multilevel addresses, it needs to intercept again. image name: docker.io/openeuler20.03-LTS package name: openeuler20.03-LTS.img Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run index cf91337..ce51ba3 100755 --- a/container/docker-rootfs/run +++ b/container/docker-rootfs/run @@ -11,7 +11,8 @@ check_cmd_input "$@" check_passwd_file "$root_pwd_file" load_docker_img "$1" echo $docker_name_tag -IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) +IMAGE_NAME=${docker_name_tag%%:*} +IMAGE_PACK=${IMAGE_NAME##*/}.cgz start=( docker run -- 2.23.0

1 0

[PATCH compass-ci] docker-rootfs/run: fix the incorrect image names
by Wang Chenglong 05 Jan '21

05 Jan '21

[Why] When docker's image name contains multilevel addresses, it needs to intercept again. image name: docker.io/openeuler20.03-LTS package name: openeuler20.03-LTS.img Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/run | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/docker-rootfs/run b/container/docker-rootfs/run index cf91337..8a7b315 100755 --- a/container/docker-rootfs/run +++ b/container/docker-rootfs/run @@ -11,7 +11,8 @@ check_cmd_input "$@" check_passwd_file "$root_pwd_file" load_docker_img "$1" echo $docker_name_tag -IMAGE_PACK=$(echo ${docker_name_tag%%:*}.cgz) +IMAGE_NAME=${docker_name_tag%%:*} +IMAGE_PACK=$(echo ${IMAGE_NAME##*/}.cgz) start=( docker run -- 2.23.0

2 2