- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] scheduler: fix: don't return queues individually
by Ren Wen 29 Dec '20

29 Dec '20

- avoid to auto submit idle job if finding no job from sub-queues, because idle-queues are not found yet. - sub-queues should be shuffled. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 21396c4..a1e3497 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -72,14 +72,14 @@ class Sched end end - return sub_queues unless sub_queues.empty? - idle_queues = [] of String default_queues.each do |queue| idle_queues << "#{queue}/idle" end - return idle_queues + all_queues = rand_queues(sub_queues) + idle_queues + + return all_queues end def get_job_from_queues(queues, testbox) -- 2.23.0

1 0

[PATCH compass-ci 2/2] scheduler: fix confused comments and add comments
by Ren Wen 29 Dec '20

29 Dec '20

fix: - use 'is joined with' instead of 'is link with'. - fix format. add: - add a TODO comment. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index f60c0e5..a7b4bd8 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -25,10 +25,11 @@ class Sched @log.warn(e) end - # auto submit a job to collect the host information - # grub hostname is link with ":", like "00:01:02:03:04:05" - # remind: if like with "-", last "-05" is treated as host number - # then hostname will be "sut-00-01-02-03-04" !!! + # auto submit a job to collect the host information. + # + # grub hostname is joined with ":", like "00:01:02:03:04:05". + # remind: if joined with "-", last "-05" is treated as host number + # then hostname will be "sut-00-01-02-03-04" !!! def submit_host_info_job(mac) host = "sut-#{mac}" @redis.hash_set("sched/mac2host", normalize_mac(mac), host) @@ -63,6 +64,7 @@ class Sched sub_queues = [] of String default_queues.each do |queue| + # TODO: this could be high cost and should be improved in future : keys(pattern). matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") next if matched_queues.empty? -- 2.23.0

1 0

[PATCH compass-ci 1/2] scheduler: find_job_boot: remove useless functions
by Ren Wen 29 Dec '20

29 Dec '20

- find_job : used to find job according to testbox. Now use sub-queues. - get_idle_job : used to find and submit idle jobs. Now use idle-queues. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 38 ---------------------------------- 1 file changed, 38 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 21396c4..f60c0e5 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -183,32 +183,6 @@ class Sched end end - private def find_job(testbox : String, count = 1) - tbox_group = JobHelper.match_tbox_group(testbox) - tbox = tbox_group.partition("--")[0] - - queue_list = query_consumable_keys(tbox) - - boxes = ["sched/" + testbox, - "sched/" + tbox_group, - "sched/" + tbox, - "sched/" + tbox_group + "/idle"] - boxes.each do |box| - next if queue_list.select(box).size == 0 - count.times do - job = prepare_job(box, testbox) - return job if job - - sleep(1) unless count == 1 - end - end - - # when find no job, auto submit idle job at background - spawn { auto_submit_idle_job(tbox_group) } - - return nil - end - private def prepare_job(queue_name, testbox) response = @task_queue.consume_task(queue_name) job_id = JSON.parse(response[1].to_json)["id"] if response[0] == 200 @@ -232,18 +206,6 @@ class Sched return job end - private def get_idle_job(tbox_group, testbox) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - - # if there has no idle job, auto submit and get 1 - if job.nil? - auto_submit_idle_job(tbox_group) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - end - - return job - end - private def auto_submit_idle_job(testbox) full_path_patterns = "#{CCI_REPOS}/#{LAB_REPO}/allot/idle/#{testbox}/*.yaml" fields = ["testbox=#{testbox}", "subqueue=idle"] -- 2.23.0

1 0

[PATCH v3 compass-ci] lib/params_group.rb optimize filter_groups()
by Lu Weitao 29 Dec '20

29 Dec '20

[Why] 1. We had for-each key-value in hash, needn't query value by key once more. 2. When remove empty value of Hash(String, Hash(String, Array(job)), remove the first level key-value at first. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 7bb7f5b..9be2cbe 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -72,10 +72,11 @@ end def filter_groups(groups) groups.each do |group_key, value| - value.each_key do |dim_key| - value.delete(dim_key) if value[dim_key].empty? + if value.empty? + groups.delete(group_key) + next end - groups.delete(group_key) if groups[group_key].empty? + value.delete_if { |_dim_key, job_list| job_list.empty? } end end -- 2.23.0

1 0

[PATCH compass-ci] sbin/auto_submit: take the pkgbuild_repo if it is linux
by Lin Jiaxin 29 Dec '20

29 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index f96be8c..51ecc6a 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -20,7 +20,7 @@ class AutoSubmit def get_pkgbuild_repo(repo_array) pkgbuild_repo = "archlinux/#{repo_array[0]}" repo_array.each do |repo| - next unless repo =~ /-git$/ + next unless repo =~ /(-git|linux)$/ pkgbuild_repo = "archlinux/#{repo}" break -- 2.23.0

2 1

[PATCH lkp-tests] auto_submit: add build-linux.yaml for linux
by Lin Jiaxin 29 Dec '20

29 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/build-linux.yaml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 jobs/build-linux.yaml diff --git a/jobs/build-linux.yaml b/jobs/build-linux.yaml new file mode 100644 index 00000000..080a84b3 --- /dev/null +++ b/jobs/build-linux.yaml @@ -0,0 +1,4 @@ +suite: build-pkg +category: functional +build-pkg: +config: <%= 'aarch64-randconfig-'+SecureRandom.random_number(1000).to_s %> -- 2.23.0

2 1

[PATCH compass-ci] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 29 Dec '20

29 Dec '20

[why] get multi-field aggregation count for overview of report page. [example] es = ESQuery.new query_items = { 'os' => 'openeuler' } field1 = 'suite' field2 = 'job_state' es.query_fields(field1, field2, query_items) [input]: field1, field2, query_items (optional for query_items, default no scope limitation) [output]: {"build-pkg"=> {"failed"=>3804, "finished"=>800, "incomplete"=>196, "submit"=>136, "OOM"=>11}, "cci-depends"=>{"finished"=>580, "failed"=>218, "incomplete"=>1}, "iperf"=>{"finished"=>136, "failed"=>57}, "spinlock"=>{"finished"=>52, "failed"=>1}, ... } Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 59 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..d881f6b 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,52 @@ class ESQuery result end + + # select field1, field2, count(*) from index where query_items group by field1, field2 + # example: + # es = ESQuery.new + # query_items = { + # 'os' => 'openeuler' + # } + # field1 = 'suite' + # field2 = 'job_state' + # es.query_fields(field1, field2, query_items) + # input: + # field1, field2, query_items + # (optional for query_items, default no scope limitation) + # output: + # {"build-pkg"=> + # {"failed"=>3804, "finished"=>800, "incomplete"=>196, "submit"=>136, "OOM"=>11}, + # "cci-depends"=>{"finished"=>580, "failed"=>218, "incomplete"=>1}, + # "iperf"=>{"finished"=>136, "failed"=>57}, + # "spinlock"=>{"finished"=>52, "failed"=>1}, + # ... + # } + def query_fields(field1, field2, query_items = {}) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: { + "all_#{field1}" => { + terms: { field: field1, size: 1000 }, + aggs: { + "all_#{field2}" => { + terms: { field: field2, size: 1000 } + } + } + } + }, + size: 0 + } + es_result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + es_result.sort_by! { |h| h['doc_count'] } + es_result.reverse! + + parse_fields(es_result, field2) + end end # Range Query Example: @@ -146,3 +192,16 @@ def assign_desc_body(keyword) }] } end + +def parse_fields(es_result, field) + result_hash = {} + es_result.each do |result| + key = result['key'] + all_field = result["all_#{field}"]['buckets'] + field_count = {} + all_field.map { |item| field_count[item['key']] = item['doc_count'] } + result_hash[key] = field_count unless field_count.empty? + end + + result_hash +end -- 2.23.0

2 1

[PATCH compass-ci] scheduler: fix: only use nodes job specified
by Ren Wen 29 Dec '20

29 Dec '20

[Before]: Every node in cluster config file will be distributed a job when submitting cluster jobs. [Now]: Job will be distributed according to job content. For example: If job has: if role server: if role client: then only nodes which roles contains 'server' and 'client' will use. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c09d145..bcb265f 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -38,6 +38,7 @@ class Sched job_messages = Array(Hash(String, String)).new lab = job.lab subqueue = job.subqueue + roles = get_roles(job) # collect all job ids job_ids = [] of String @@ -52,6 +53,9 @@ class Sched # steps for each host cluster_config["nodes"].as_h.each do |host, config| + # continue if role in cluster config matches role in job + next if (config["roles"].as_a.map(&.to_s) & roles).empty? + queue = host.to_s job_id = add_task("#{queue}/#{subqueue}", lab) @@ -157,6 +161,11 @@ class Sched end end + def get_roles(job) + roles = job.hash.keys.map { |key| $1 if key =~ /^if role (.*)/ } + roles.compact.map(&.strip) + end + def get_commit_date(job) if (job["upstream_repo"] != "") && (job["upstream_commit"] != "") data = JSON.parse(%({"git_repo": "#{job["upstream_repo"]}.git", -- 2.23.0

1 0

[PATCH lkp-tests] optimize the deploy-cci script
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 53 ++++++++++++++++++++++++++++-------------------- 1 file changed, 31 insertions(+), 22 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 55835544..21308ecf 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -2,30 +2,39 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# redirect remote-repository address to "/srv/git/gitee.com" -if [ -n "$GIT_SERVER" ]; then - cat >> /etc/gitconfig <<-EOF - [url "git://$GIT_SERVER/gitee.com"] - insteadOf=https://gitee.com - EOF -fi +env() +{ + export initrd_http_host=$INITRD_HTTP_HOST + export initrd_http_port=$INITRD_HTTP_PORT + export gitcache_host=$GITCACHE_HOST + export gitcache_port=$GITCACHE_PORT +} -export initrd_http_host=$INITRD_HTTP_HOST -export initrd_http_port=$INITRD_HTTP_PORT -export gitcache_host=$GITCACHE_HOST -export gitcache_port=$GITCACHE_PORT +deploy() +{ + # redirect remote-repository address to "/srv/git/gitee.com" + if [ -n "$GIT_SERVER" ]; then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF + fi -mkdir -p /c -cd /c || exit -git clone https://gitee.com/wu_fengguang/compass-ci.git + mkdir -p /c + cd /c || exit + git clone https://gitee.com/wu_fengguang/compass-ci.git -cd /c/compass-ci || exit -[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + cd /c/compass-ci || exit + [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" -# Need mount tmpfs to fix compass-ci/container/os-nfs/start error -# "exportfs: /exports/result does not support NFS export" -mount -t tmpfs none /srv + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error + # "exportfs: /exports/result does not support NFS export" + mount -t tmpfs none /srv -# test the script for one-click deployment -cd /c/compass-ci/sparrow || exit -./install-tiny + # test the script for one-click deployment + cd /c/compass-ci/sparrow || exit + ./install-tiny +} + +env +deploy -- 2.23.0

2 1

[PATCH v1 compass-ci 2/2] lib/bisect: add PKGBUILD info in bisect report
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 7 +++++++ lib/mail_bisect_result.rb | 8 ++++++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index f789a71..2043730 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -22,6 +22,7 @@ class GitBisect set_bad_job set_build_pkg_dir set_upstream + set_pkgbuild_repo set_work_dir set_bad_commit set_good_commit @@ -61,6 +62,11 @@ class GitBisect raise "checkout repo: #{@upstream_repo} to commit: #{@upstream_commit} failed!" unless @work_dir end + def set_pkgbuild_repo + @pkgbuild_repo = @bad_job['pkgbuild_repo'] + raise 'pkgbuild_repo is null' unless @pkgbuild_repo + end + def set_bad_commit @bad_commit = @upstream_commit end @@ -100,6 +106,7 @@ class GitBisect 'commit' => first_bad_commit, 'bisect_error' => bisect_error, 'all_errors' => all_errors, + 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 1d4ed69..a4a604b 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -9,10 +9,11 @@ require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @bisect_error = bisect_info['bisect_error'] - @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @all_errors = bisect_info['all_errors'] + @bisect_error = bisect_info['bisect_error'] + @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) # now send mail to review @@ -28,6 +29,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY @@ -43,6 +45,8 @@ class MailBisectResult all errors/warnings: #{@all_errors} + reference information: + #{pkgbuild_repo_url} #{bisect_job_url} #{first_bad_commit_job_url} Regards, -- 2.23.0

1 1