December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/mail_bisect_result: auto_tool => assign_account_client
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5e296a6..1d4ed69 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -4,7 +4,7 @@ require 'json' require 'base64' require_relative 'git' require_relative 'mail_client' -require_relative 'auto_tool' +require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult -- 2.23.0

1 0

[PATCH compass-ci] web_backend.rb: optimize var name
by Li Yuanchao 22 Dec '20

22 Dec '20

As the implementation of method combine_group_query_data changes, var name suit_list is ok, cause the method return a hash in fact. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 2405107..0ca71e1 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -186,15 +186,15 @@ end def do_get_groups_matrices(must, dimension, total, size, from) job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_list = combine_group_query_data(job_list, dimension) + matrices, suites_hash = combine_group_query_data(job_list, dimension) while matrices.empty? from += size break if from > total job_list = get_dimension_job_list(dimension, must, size, from) - matrices, suites_list = combine_group_query_data(job_list, dimension) + matrices, suites_hash = combine_group_query_data(job_list, dimension) end - [matrices, suites_list] + [matrices, suites_hash] end def get_groups_matrices(conditions, dimension, must, size, from) @@ -209,11 +209,11 @@ end def get_compare_body(params) dimension, conditions = get_dimension_conditions(params) must = get_es_must(params) - groups_matrices, suites_list = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) + groups_matrices, suites_hash = get_groups_matrices(conditions, dimension, must, COMPARE_RECORDS_NUMBER, 0) if !groups_matrices || groups_matrices.empty? body = 'No Data.' else - body = compare_group_matrices(groups_matrices, suites_list, { no_print: true }) + body = compare_group_matrices(groups_matrices, suites_hash, { no_print: true }) body = 'No Difference.' if !body || body.empty? end return body -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies: adjuest file format
by Wei Jihui 22 Dec '20

22 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/install-dependencies.sh | 39 ++++++++++++++++++------------------ 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index 3a206134..6263dc91 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -6,28 +6,27 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh . $PROJECT_DIR/sbin/set-env.sh -# choose install function base on -# DISTRIBUTION +# choose install function base on DISTRIBUTION linux_dep() { -source /etc/os-release -case $ID in -ubuntu|debian) - export DEBIAN_FRONTEND=noninteractive - apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client - ;; -openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - else - yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git - fi - sudo gem install -f git activesupport rest-client - ;; -*) - echo "$ID not support! please install dependencies manually." - ;; -esac + source /etc/os-release + case $ID in + ubuntu|debian) + export DEBIAN_FRONTEND=noninteractive + apt-get install -yqm ruby-git ruby-activesupport ruby-rest-client + ;; + openEuler|fedora|rhel|centos) + if type dnf > /dev/null 2>&1; then + dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + else + yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git + fi + sudo gem install -f git activesupport rest-client + ;; + *) + echo "$ID not support! please install dependencies manually." + ;; + esac } -- 2.23.0

1 0

[PATCH lkp-tests] lib/env.sh: fix the check result of is_virt
by Zhang Yu 22 Dec '20

22 Dec '20

Hypervisor is a software that simulates hardware resources and scheduling on a physical machine. so if grep success, return 1 means it's a physical machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/env.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/lib/env.sh b/lib/env.sh index 8ac42a836..a1d5e445b 100755 --- a/lib/env.sh +++ b/lib/env.sh @@ -69,7 +69,8 @@ is_virt() # run as root [ -n "$(virt-what)" ] else - grep -q -w hypervisor /proc/cpuinfo + grep -q -w hypervisor /proc/cpuinfo && return 1 + return 0 fi } -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: add repository version pointing
by Hu Xuejiao 22 Dec '20

22 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 494107e..2159ddc 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -13,11 +13,18 @@ cd /c || exit # modify and manual run for now: [ -d "compass-ci/.git" ] || { git clone https://gitee.com/wu_fengguang/compass-ci.git - ln -s compass-ci cci + + cd /c/compass-ci || exit + [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + + ln -s compass-ci /c/cci } [ -d "lkp-tests/.git" ] || { git clone https://gitee.com/wu_fengguang/lkp-tests.git + + cd /c/lkp-tests || exit + [ -n "$lkp_commitid" ] && git reset --hard "$lkp_commitid" } [ -d "upstream-repos/.git" ] || { -- 2.23.0

1 0

[PATCH lkp-tests] tests/deploy-cci: specifying the repository version
by Hu Xuejiao 22 Dec '20

22 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 3 +++ 1 file changed, 3 insertions(+) diff --git a/tests/deploy-cci b/tests/deploy-cci index e296568f..55835544 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -19,6 +19,9 @@ mkdir -p /c cd /c || exit git clone https://gitee.com/wu_fengguang/compass-ci.git +cd /c/compass-ci || exit +[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error # "exportfs: /exports/result does not support NFS export" mount -t tmpfs none /srv -- 2.23.0

1 0

[PATCH compass-ci] lib/git_bisect: return all errors to bisect mail.
by Cao Xueliang 22 Dec '20

22 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index bc1e035..f789a71 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -82,7 +82,6 @@ class GitBisect result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} #{(a)error_id.inspect} #{@work_dir}` end Utils.create_bisect_log(@work_dir) - FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) puts "\nbisect result: #{result}" analyse_result(result) end @@ -92,10 +91,16 @@ class GitBisect return nil unless temp[0].include?('is the first bad commit') || temp[-1].include?('bisect run success') first_bad_commit = Utils.parse_first_bad_commit(result) - error_messages = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) + bisect_error = ErrorMessages.new(@build_pkg_dir).obtain_error_messages_by_errorid(@error_id) + all_errors = Utils.obt_errors(@work_dir, first_bad_commit) + first_bad_commit_result_root = Utils.obt_result_root_by_commit(first_bad_commit) + FileUtils.rm_r(@work_dir) if Dir.exist?(@work_dir) - return Hash['repo' => @upstream_repo, 'commit' => first_bad_commit, - 'error_messages' => error_messages] + return Hash['repo' => @upstream_repo, + 'commit' => first_bad_commit, + 'bisect_error' => bisect_error, + 'all_errors' => all_errors, + 'first_bad_commit_result_root' => first_bad_commit_result_root] end # first search the good commit in db -- 2.23.0

1 0

[PATCH compass-ci] delimiter/utils: add the get all errors function
by Cao Xueliang 22 Dec '20

22 Dec '20

1. fix some rubocop warnings. 2. add get errors by commits - get the pre_commit by find_parent_commit - get cur_id and pre_id by obt_id_by_commit - get errors by get_compare_result 3. add get result root by commit Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 48 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 41803f3..372e821 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -4,10 +4,12 @@ require 'set' require 'json' +require 'yaml' require 'fileutils' require_relative './constants' require_relative '../../lib/sched_client' +require_relative '../../lib/compare_error_messages' require_relative "#{ENV['LKP_SRC']}/lib/monitor" # a utils module for delimiter service @@ -83,9 +85,20 @@ module Utils es = ESQuery.new new_job = es.query_by_id(new_job_id) - return 'bad' if new_job['stats'].key?(error_id) - return 'good' + status = new_job['stats'].key?(error_id) ? 'bad' : 'good' + puts "new_job_id: #{new_job_id}" + puts "upstream_commit: #{job['upstream_commit']}" + record_jobs(new_job_id, job['upstream_commit']) + + return status + end + + def record_jobs(job_id, job_commit) + FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT + commit_jobs = File.join(TMP_RESULT_ROOT, 'commit_jobs') + content = "#{job_commit}: #{job_id}" + File.open(commit_jobs, 'a+') { |f| f.puts content } end def get_account_info @@ -100,6 +113,8 @@ module Utils account_info = get_account_info raise "query #{DELIMITER_ACCONUT} account info failed!" unless account_info + record_jobs(job['id'], job['upstream_commit']) + job['suite'] = 'bisect' job['my_name'] = account_info['my_name'] job['my_email'] = account_info['my_email'] @@ -133,5 +148,34 @@ module Utils log_content.each { |line| f.puts(line) } end end + + def find_parent_commit(git_dir, commit) + response = %x(git -C #{git_dir} rev-parse #{commit}~1) + return response.chomp + end + + def obt_id_by_commit(commit) + puts "#{TMP_RESULT_ROOT}/commit_jobs" + content = YAML.load(File.open("#{TMP_RESULT_ROOT}/commit_jobs")) + puts content + content[commit] + end + + def obt_errors(git_dir, commit) + pre_commit = find_parent_commit(git_dir, commit) + obt_errors_by_commits(commit, pre_commit) + end + + def obt_errors_by_commits(cur_commit, pre_commit) + cur_id = obt_id_by_commit(cur_commit) + pre_id = obt_id_by_commit(pre_commit) + _, errors = get_compare_result(pre_id, cur_id) + return errors + end + + def obt_result_root_by_commit(commit) + id = obt_id_by_commit(commit) + ESQuery.new.query_by_id(id)['result_root'] + end end end -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_client: add send_mail_encode function
by Cao Xueliang 22 Dec '20

22 Dec '20

If some special characters the payload of post, we use base64 encode to pass the payload. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_client.rb | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/lib/mail_client.rb b/lib/mail_client.rb index 1d0938a..b13492f 100644 --- a/lib/mail_client.rb +++ b/lib/mail_client.rb @@ -2,6 +2,7 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'base64' require 'rest-client' require_relative 'constants.rb' @@ -18,4 +19,9 @@ class MailClient resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_yaml") resource.post(mail_json) end + + def send_mail_encode(mail_data) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_mail_encode") + resource.post(Base64.encode64(mail_data)) + end end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job.rb: delete input job key which is path
by Zhang Yu 22 Dec '20

22 Dec '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- lib/job.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/job.rb b/lib/job.rb index 2ed06edc6..7553312ed 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -254,6 +254,7 @@ class Job @overrides.merge!(hash['override']){ |_key, a, _b| a} hash.delete('override') end + hash.delete_if { |key, _| key.start_with?('#!') } hash.merge!(@overrides) @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end -- 2.23.0

1 0